Using pyjamas to program external Mechanical Turk experiments

Posted on Updated on


I recently set up my first external Mechanical Turk study. My greatest friend and foe in this process was pyjamas, a Python-to-Javascript compiler and Widget Set API. THE great advantage of using pyjamas: you can program your entire experiment in python, and pyjamas will create the browser-dependent javascript code. If you already know javascript, writing your experiment in python without having to worry about browser-dependent issues will save you time. And if you don’t, you don’t have to go through the frustrating process of learning javascript. On the downside, the documentation for pyjamas is currently not very good, so figuring out how to get things to work can take a while.

That’s why I’m providing the (commented) code that I generated to create my MechTurk experiment. A short demo version of the experiment can be found here.

If you find this code helpful, please consider acknowledging it via the following URL in your paper/presentation to spread the word:
http://hlplab.wordpress.com/2011/12/25/using-pyjamas-to-program-external-mechanical-turk-experiments/

A screenshot of the experiment. Participants were asked to rate on a 7-point scale how natural the statement they heard was as a description of the scene.

The structure of the experiment, which combines timed image presentation, playing audio via HTML5, recording naturalness ratings via radio buttons, and recording feedback via a text box:

  1. Participants first see an instructions page with a CONTINUE button. Upon clicking CONTINUE,
  2. they continue to a sound test. This is to make sure participants can actually hear sound via their speakers/headphones. They click PLAY and are asked to enter the first word of the phrase they hear. If the word is entered correctly,
  3. the main body of the experiment begins. Each trial consists of
    • displaying an image of a full gumball machine, which after 1.5 seconds changes such that
    • a second image is displayed where a certain number of gumballs has moved to the lower chamber. An audio file is played (statements of the form “You got X of the gumballs”). Next to the gumball machine, a 7-point scale of radio buttons is displayed on which participants are asked to rate the naturalness of the statements they heard as a description of the scene. Below the scale, there is a FALSE button that participants are asked to click if they think the statement was false. When a button is clicked, information about which button it was is recorded and the next trial begins.
  4. On the last trial, a feedback box is displayed alongside a SUBMIT button. Upon clicking SUBMIT, all the information about the experiment (worker ID, trial information, participants’ responses, etc) is submitted to Mechanical Turk. Note that in the demo version I’ve made available, the information is printed to a window that pops up when you click the SUBMIT button.

The tgz file on my homepage (scroll to the bottom of the page) contains all the files necessary for building the pyjamas project that generates the experiment. You need to have installed pyjamas for the build to work. The structure of the experiment is reflected in the file natural.py, which contains the python code that pyjamas compiles to javascript. Read the README to get started.

Good luck. Questions & comments welcome!

About these ads

4 thoughts on “Using pyjamas to program external Mechanical Turk experiments

    Fatemeh said:
    February 4, 2012 at 3:41 pm

    Very useful post, reading this I decided to design my self-paced reading experiment by pyjamas instead of wrting javascript. But I have problem using keyboard events, looks like not all libraries work very well. Have you ever tried handling events (keypress or mouse clicks)?

    Like this

      jdegen responded:
      February 5, 2012 at 11:59 am

      Hi Fatemeh – So far I haven’t done any complicated event handling, in particular I haven’t done anything with keyboard events because I haven’t been interested in timing issues the way I assume you are if you’re doing self-paced reading. There’s a tutorial here about event handling, though unfortunately it looks like they haven’t gotten round to explaining the handling of keyboard events yet (only mouse events): http://pyjs.org/controls_tutorial02.html
      If you can’t find a solution after googling for a while, my experience has been that posting to the pyjamas mailing list (which you can sign up for here: http://pyjs.org/#Getting%20Help) helps. You usually get a response within a few hours. Sorry I can’t be of more help. Good luck!

      Like this

      tiflo said:
      November 9, 2012 at 3:24 pm

      Hi Fatemeh,

      Are you aware of IBex, the javascript based SPR tool? Also, Hal Tily has a Flash-based SPR applet that works very nicely. Finally, if you insist ;) on using pyjamas, perhaps contact Esteban Buz (listed on our lab website), who has been using pyjamas to elicit reaction times online (in a picture forced choice task).

      Like this

    [...] If you’re interested in pyjamas contact me … if you’re interested in the python-based package for python-to-javascript conversation, which comes with a variety of widgets that are useful for creating web-based experiments, make sure to read this recent HLP Lab post by Judith Degen on the pyjamas packages. [...]

    Like this

Questions? Thoughts?

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s