Experimental Design Project Overview
For this project, you are to design, carry out, analyze, and report on the results of an experiment. You will work in groups of three on this project.
Guidelines for the experiment
The best projects are often inspired by students’ everyday lives. Successful past projects have included finding the best places for cellphone reception and finding the best position for holding chopsticks for different tasks. Students sometimes propose experiments that try to vary some kind well-known experiment from psychology (e.g. the Stroop effect). This is allowable if you adapt the experiment rather than just rerun it.
Avoid experiments that are “testing” things to which you would be reasonably expected to know the answer. Bad examples: Is a water balloon more likely to pop thrown from a height of 3 feet or 6 feet? Does oatmeal heat up faster if there is only a little in the pan? Basic physical phenomena are often bad candidates unless you add a personal element to them that makes them have a unique behavior for which the answer cannot be found on Wikipedia. Also do not propose to test things that a reasonable person wouldn’t expect to affect the response, unless you really have personal knowledge to suggest otherwise. Bad example: Does the color of a balloon affect how easy it is to pop?
Make sure that you are proposing something feasible in the time limits you have. A single observation from the experiment should not take more than a few minutes to observe and measure, in general. Do not propose projects that are dangerous, that involve the violation of laws or University policies, or that could be expected to cause harm to people or property.
Stages of the Project
Stage 1: Initial Proposal
Write a brief proposal for an experiment where you describe the question you are trying to address, the specific response, conditions, and units you are plan to use. At this point, we will not have covered the specific designs you are to pick from, and so you will not know for certain how many conditions the relevant designs need. You should give at least three conditions you wish to consider.
The lab session during the final week of classes will be required for all students and will be devoted to running experiments for those experiments that need subjects (i.e. humans) as its unit. However, if this is the case for your experiment, you must have an experiment that can be run in the confines of the classroom and in the time limits such that you can get through the necessary subjects during the section. If you plan to use subjects that are not from the sections, you must clearly explain where these subjects will come from, and have volunteers already agreed and a date set for the experiment.
In preparation for the design stage, you should also run and record several replications of your experiment with a single set of conditions so as to get an idea of the measurement error/replicability of your experiment and how long it takes. This will be essential for power calculations in the next stage. You should not be actually running your full experiment.
The point of the initial proposal is to consider carefully the feasibility of your design. We will evaluate each proposal, and if we think that the proposal is either too ambitious (or not ambitious enough – think about the balloon examples above), we will send you back to the drawing board. You should think of this proposal as your opportunity to convince us to allow your experiment to go forward.
After writing the initial proposal, if you do not think your group will work together well this is the best time to come to us because we can reassign people and form new groups. After this point, it will be difficult to change the project members.
Stage 2: Design Proposal
In this proposal, you should now give specific description of the design you will use to carry out your experiment. The experiment should be one of the following designs that we will learn about during the semester:
- A replicated Latin Squares design (only one factor of interest necessary, but at least 4 levels for the factor): repeated LS[1]
- A Split Plot / Repeated Measures design (only one factor of interest necessary for a block, and one for subunits within block): SP/RM[1,1]
- A Complete Block design or Generalized Complete Block Design with at least two factors of interest: (CB[2])
- A basic factorial design with at least 3 factors of interest, with at least one factor with more than levels (>2) and relevant contrasts between the levels beyond all pairwise.
For this proposal, you should specify the important components of your design: what are your specific treatments, what type of design will you use, how many units do you need for this design, how you will assign your units to treatments. You need to justify the design decisions you make. You should imagine you were asking me for money to fund your experiment – what would I want to know before I gave you money?
We will again evaluate each proposal and may require you to redo the proposal, if, for example, we think another design more naturally fits your data or if you seem to be misunderstanding the components of the design.
Stage 3: Protocol
Write up a protocol for your experiment. This should include a detailed set of instructions for how to carry out the experiment, including an explicit assignment of treatments to the units which you should report. Also describe possible sources of variation and how they have been controlled and be prepared for what to do if there was a non-response or missing run.
Conduct a dry run of the experiment based on this protocol, and revise your protocol based on your findings. Describe what happened in your dry run and how you chose to address the issues that arose.
Stage 4: Experiment
Conduct your experiment. Record your results carefully, and note any unusual occurrences in the experimental process. For example, if your experiment is conducted outside over several days, you might want to record the temperature and whether or not it was rainy that day, even if those were not factors in your original design structure. Or, if a run was interrupted and continued later or abandoned, record what happened. Your should record this information on a spreadsheet or a piece of paper, with a table that looks something like this:
| Run | Block Level | Factor Level | Response | Notes |
|---|---|---|---|---|
| 1 | 1 | 2 | 15.5 | interrupted |
These are what I refer to as your lab notes that you must include in your writeup.
Stage 5: Final Report
Enter your data into R and analyze it using graphical techniques as well as the appropriate inferential methods. Write a report as a format technical document. If you have done well on the previous components, you should be able to reuse much of the material written for the previous stages.
Stage 6: Self-evaluation
Each member of the group must individually submit a description of how the work was distributed throughout the project. You should describe specifically the organization strategy you took in the project (see suggestions below). These do not need to be formal, but should be about 2-3 paragraphs.
If there is a problem with one of the group members, this is a time you can describe it, though it is better to bring it up with me before this point.
Suggestions for Organizing Your Project
If you are having problems working with someone in the group, please come and talk to us as soon as possible.
Students in the past have found Google Docs to be a good tool for people to be able to work on a document jointly, minimizing the need to pass around a document by email.
There might be a temptation to delegate each component of the project (design, analysis, writing) to a different group member. We strongly advise against this as it often leads to disjointed projects. The writing of the final report is a task that can be well-divided among group members but the design and analysis should be done jointly.
A possible way to avoid all of the organizational duties falling to one person: at the beginning, assign someone for each of the stages (1-5) to be the organizer for that stage. This person will send out emails reminding everyone of duties, pressure everyone to get their part done, do a final check that the required writeup is not missing any requested components, etc. The final writeup is probably more work to organize, so for a group of 3, I would suggest that the organizer for stages 1-4 alternate between two people, and stage 5 be the third person.