Mech Turk and Written Recall

Ah, we’ve just got back the first result of two studies that used a written recall paradigm via Mechanical Turk to test a couple of predictions of Uniform Information Density. You can see an example template of a written recall procedure here (JavaScript required). Each study took about 1 day for the equivalent of 20 participants (balanced across 4 lists) at $.02 per trial plus some boni (see below).

The next step is to implement a spoken recall paradigm. If anyone out there has already done that, let me know.

We also tested progressive payment as a way to elicit more balanced data sets. Whereas normal MechTurk data sets exhibit Zipf distributions with regard to the trials per participant, a simple progressive scheme ($.20 for at least 20 trials, $.50 for at least 40 trials, etc.) worked quite well to drastically increase the percentage of data that comes from participants who’ve done the entire experiment.

Furthermore, HLP lab manager Andrew Watts has written a little script that makes sure that each item gets only seen in one condition by each participant and that conditions are counterbalanced across participants (worker IDs). We’re still working on some details, but once it’s ready for prime time, we’ll share it here.