Replication, Replication, Replication

I couldn’t resist the allusion to my earlier, lame little joke about participation. Anyway, everybody’s been having so much trouble wrapping their heads around the replication homework assignments that I thought I’d try to provide a bit more background as to why replication is such a core part of doing science. If you’re at all familiar with Milgram’s obedience study (discussed in our textbook and every other textbook that ever talks about research methods, or research ethics, or, well, obedience), you’ll know that his findings were surprising, even to Milgram. When you get surprising results, the most sensible thing to do is to stage a replication — a ‘do-over’, as Janet Ruane (2005) dubs it — in order to establish that the results the first time weren’t just a fluke. If  you repeat your study and get much the same results, you’ve now got grounds for more confidence in the findings. If other people replicate your study and get much the same results (after all, what if you really, really wanted to be surprised? what if you therefore saw what you wanted to see, and not what was actually there?), then you’ve got even more confidence in the findings.

Even after you’ve established some confidence in the findings, you might wonder how much they can be generalised to other settings, populations, and circumstances. And so you might seek to replicate the study but add some twists, in order to test the scope of the study’s external validity. All of these options — repeating the study step-for-step, having other researchers try to replicate the findings, tweaking the study’s setup — are alluded to in the replication HW assignments. The point is to digest the methods and findings of a previous study, and then to ‘reverse-engineer’ it in order to come up with your own version, which can be more or less faithful to the original (you should always give reasons for the changes you make, of course).

Why would anyone want to do that? you ask. Well, whenever someone ends up with surprising or controversial findings, we have good reasons to seek to replicate them. But often the most surprising ones or the ones that are only marginally significant, statistically speaking, are the ones that can’t be replicated: ‘Suppose you have two well-designed, carefully run studies, A and B, that investigate the same phenomenon. They perform what appear to be identical experiments, and yet they reach opposite conclusions. Study A produces the predicted phenomenon, whereas Study B does not’ (Barrett 2015). What we have here is a failure to replicate — but it’s a telling failure.

And thus we eliminate tempting but misleading bits of falsehood from the fund of human knowledge. Science can get things wrong, but this doesn’t mean we can do without science; in such cases, as one researcher argues in a recent, highly informative interview, our best hope for a corrective is … more science.

Further Reading/Notes Toward a Personal Canon

Patten, Mildred L. 2014. Proposing Empirical Research: A Guide to the Fundamentals, 5th Ed. Pyrczak Publishing. For advice on how to ‘reverse-engineer’ when you think you’ve got reason to introduce some changes in the original design, see Patten (2014:19, Topic A) on modified replication.

 

REFERENCES

Barrett, Elizabeth Feldman. 2015. ‘Psychology is not in crisis’. New York Times, 1 September, p. A23.

Ruane, Janet. 2005. Essentials of Research Methods: A Guide to Social Science Research. Blackwell.

Advertisements

Tags: , ,

2 Responses to “Replication, Replication, Replication”

  1. Presentations, Presentations, Presentations | Preyer 207 Says:

    […] know, I know: that joke just isn’t funny anymore. In any event, here’s another brief post that I’ll […]

  2. For Friday, 2/5 (Week 2) | Preyer 207 Says:

    […] Reverse Replication Outline (Content Analysis). In this assignment, you’ll be designing a replication for the […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s


%d bloggers like this: