E. Evaluation

The evaluation is the final part of the paper. In this section of the report you are expected to:
  • Discuss your results in comparison to the results of the original study. If they differed from the original study, why do you think that this happened?
  • Discuss your findings with reference to the theory outlined in the introduction.
  • State and explain the strengths and limitations of your design, sample and procedure.
  • Suggest modifications to address the limitations of your own investigation.
  • A final statement of conclusion with regard to your hypothesis.

In this section of the report it is important that you evaluate the strengths and limitations of your methodology. You should also explain how the limitations may have affected the outcome of your study. The strongest reports will identify possible extraneous variables that may have influenced the study, and will not rely on a simplistic evaluation, such as “the experimental study should have used a larger sample”.  Since it is always true that we could have more participants, this is not a very rigorous analysis.

In addition, you should not include examples of “experimenter error.”  For example, if for one group you forget to read the directions from the briefing notes, this is not a valid limitation.  This is a mistake.  If you make a mistake, the expectation is that you redo the experiment. Discussion of such errors will not earn any marks. 

Finally, it is not appropriate to discuss the assumed behaviour of the participants. For example, do not assume that the participants did not follow directions.  Use your debriefing to gather evidence to support this so that you can make informed comments about the experiment.

Suggestions for relevant modifications for future replications must also be included. These should only address the limitations you have identified in your own investigation. 

Sample evaluation

As can be seen by the results stated above, we were able to support the findings of Tversky & Kahnemann (1974) and the prediction that an anchor influences a person’s estimate of a value. They found a much lower difference between the two groups than we did.  The difference between the two medians in the original study was 1738; in our study, it was a difference of 9202.50. This could be because our participants were younger and may not have as much conceptual understanding of mathematics. It could also have to do with participant variability; the standard deviation and the semi-interquartile range were both very high, indicating that the skill level in the two groups was quite different.

One of the strengths of our study was that we had the participants practice estimation first so that we could rule out that they didn’t understand instructions.  In addition, we had them answer the key question on the back of the paper so that they did not see the previous numbers on the other side of the paper.  We did this so that those numbers did not influence their estimate.  However, we probably should have carried out a distractor task - such as reciting the alphabet backwards – between tasks.  We cannot rule out that the memory of the previous estimates may have influenced their final response.

A limitation of the study was that we used an independent measures design. Our differences may be due to participant variability. We do not know how many of the participants have experience with factorials or with estimating values. It may be more appropriate to first test the students for their ability to estimate values of equations or their knowledge of factorials and then eliminate them from the sample if necessary.

There were also variables that could not be well controlled because of the nature of our sample. It is not possible to know to what extent the participants were trying to quickly calculate the response or whether they were really estimating.  We also do not know if they read the equation from left to right or right to left. During the debriefing, the participants did not show that they knew the goal of the study, but some indicated that they found it uninteresting.  This may have resulted in the high variation of the responses.  Finally, it is difficult to generalize our findings as the sample was made up of only grade 9 maths students. Older participants or participants with lower maths abilities may not demonstrate the same results.

For future research it would be interesting to put this experiment in a more realistic context.  Rather than simply giving them a math problem, several studies give suggested prices for goods. The participants then have to determine what a fair price would be. If anchoring works, then when the given price should have an effect on the concept of a “fair price.”

From our study, we are able to conclude that in a sample of grade 9 mathematics students, the value of the anchor in a math equation affects the estimate of the answer to the equation.

NB: The entire sample in this chapter is only 1660 words in length.  The maximum word count is 2200.


Comments on the sample

Paragraph 1: This paragraph compares the findings of the study to the original study.  There is an attempt to explain any differences in the data or results.

Paragraph 2: Strengths of the study are outlined; there is a suggestion for improvement.

Paragraphs 3 & 4: Limitations of the study are explained and then suggestions are made for improvement.

Paragraph 5: A suggestion is made for how to study this topic in a different manner - furthering our understanding of the role of anchors in decision making.

Paragraph 6:  The report ends with a statement of conclusion that reflects the hypothesis.

How you are assessed 

The following table is the assessment rubric used to award marks for your analysis.

MarksLevel descriptor
0
  • Does not meet the standard below.
1 - 2
  • The findings of the investigation are described without reference to the background theory or model.
  • Strengths and limitations of the design, sample and/or procedure are stated but are not directly relevant to the hypothesis.
  • One or more modifications are stated.
3 - 4
  • The findings of the investigation are described with reference to the background theory or model.
  • Strengths and limitations of the design, sample and procedure are stated and relevant, but not explained.
  • Modifications are explained, but not explicitly linked to the limitations of the investigation.
5 - 6
  • The findings of the investigation are discussed with reference to the background theory or model.
  • Strengths and limitations of the design, sample and procedure are stated, relevant and explained.
  • Modifications are explained and explicitly linked to the limitations of the investigation.

Next: Overview of report format

All materials on this website are for the exclusive use of teachers and students at subscribing schools for the period of their subscription. Any unauthorised copying or posting of materials on other websites is an infringement of our copyright and could result in your account being blocked and legal action being taken against you.

Comments 2

Korey Haugland 23 April 2018 - 10:13

Hello, John! I noticed in this page that you have written,

"NB: The entire sample in this chapter is only 1660 words in length. The maximum word count is 2000."

Should this say that the maximum word count is 2,200?

Thanks,
-Korey

John Crane 23 April 2018 - 20:19

Thanks, Korey. That was changed after I wrote this page. I will fix it.