DESIGNING EXPERIMENTS
DESIGNING EXPERIMENTS
Hamish Macleod
Uses
University of Edinburgh. Every educational innovation is an experiment in some sense of the word; you change something about the students' experience, predicting that better learning will take place.
A controlled experiment is a way of teasing out the details of just which aspects of your innovation are influencing the outcomes you are considering and bringing about the changes you observe. The experimental method is a way of thinking about the evaluation process such that all the possible sources of influence are kept in mind.
Process
Dont tinker with too many
1. Describe the Innovation
aspects of your course at one time, or it will be impossible to
Construct a clear description of the innovation that you are seeking to monitor.
identify just which of these Exactly what will be different in the students' experience after the change you propose as compared to the current changes you have made has situation? The ideal experiment manipulates only one factor at a time, thus enabling very direct causal links to be explored. caused the improvement (or
In practice, a number of changes may have to take place at once for reasons of expedience. disruption) which you observe.
As far as possible, compare
2. Decide the parameters of your experimental design
like with like, trying to exclude the possibility that any
What are you going to compare with what? Will it involve a comparison of what happened last year (before the initiative) differences you observe can with the experience of the current year (in which the initiative is in place)? Perhaps only part of the class will experience
be explained in terms of the new learning situation, and their performance (or their judgement of their enjoyment of the course, or whatever) will differences between your
be compared with that of their immediate colleagues who have not experienced the change. Or perhaps you plan to student groups rather than in continue with your normal practice and compare the learning outcomes of your students with those of an equivalent class
terms of your educational taught by a colleague at another institution where some sort of innovation has been put in place. innovation.
Be specific in your choice of
3. Define success
expected or desired outcome, as this will help you to decide
Decide what outcome would be needed for you to consider your experiment to be a success. Perhaps the objective is what you are wanting to to address some issue of recruitment onto a subsequent level course. Are changes going to be reflected in the measure.
students' academic performance? Will they be expected to enjoy their learning experience more, or to express more confidence and satisfaction with their achievements?
Be conscious too that
4. Decide how to measure successfulness
measurements can intrude, Decide how your predicted outcome can best be measured. Other sections in this guide have more to say about the distorting the very outcome in
different ways in which you can obtain qualitative or quantitative estimates of some dimension which tells you something which you are interested.
about the outcomes of interest.
While you wish your outcome measures to be rigorous, they
Be aware that what you measure, and what you are interested in, may be subtly or even profoundly different. Some things should not be so detailed and
may be easily measured (like the scores in a multiple-choice examination) while others (like the depth of understanding of extensive that they become a some concept) may be considerably more difficult to measure; and the temptation is always to take the simple course. On
burden on your participants. the other hand, good, simple proxy measures can often be found for the outcome of interest. It is not necessary that the measurement you collect be direct, but only that it is strongly correlated with what you need to know about.
moderate low to
moderate low to low low
The more spread there is in
5. Analyse your data.
the scores within the groups, Analysis of data gathered through an experimental approach will most likely focus on deciding whether your innovation the more the groups must be has had the predicted effect. Is there a difference to be seen in the outcome measure(s) gathered between your original separated in order that we be (control) and post-intervention (experimental) situation? Is the difference in the direction which was predicted? And is convinced that the differences the difference greater than a change which might be expected by random chance alone; that is, is it statistically significant?
we see are unlikely to have
been the result of random Do not think about statistical significance as being an all or nothing thing but as an expression of your confidence in
fluctuations in our coming to a particular conclusion or making a particular claim. measurements. In short, the Always begin the analysis with a general exploration or your data. Consider using confidence intervals first, as a good
more confident we can be general comparison between datasets. If it appears that differences do exist, then proceed to some test of statistical that the difference we see is significance. "real". Descriptive statistics (like an arithmetic mean) can be calculated, or some graphical technique (such as the plotting of a
histogram) can be employed to display differences between your baseline (pre-intervention) and novel (post-intervention) measurements. Inferential procedures enable the exploration of the statistical significance of such differences. Basically, these latter procedures enable you to express the size of the differences between two (or more) groups in relation to the spread of the individual measurements within the groups.
Remember that differences in average value are not the only possible interesting outcomes. Difference in the spread of scores may be equally revealing. For example, if a topic is "well taught" (whatever that might mean) could very well result in a decrease in the spread of performance across the student group. Statistical techniques exist to explore changes of this sort as well.
Other Relevant Pages Recipes Pre and post testing
Focus groups Confidence logs Interviews Questionnaires Resource questionnaires Ethnography Trials
Information Pages Statistics questions
Student sample
Beware of testing too many hypotheses, the more you torture
Isolation or integration Pedagogic toolkit
the data, the more likely they are to confess but confession
obtained under duress may not be admissible in the court of
Pre and post testing
Working with groups
scientific opinion
Evaluation Report Outline Stigler (1987) References