Using pyjamas to program external Mechanical Turk experiments

I recently set up my first external Mechanical Turk study. My greatest friend and foe in this process was pyjamas, a Python-to-Javascript compiler and Widget Set API. THE great advantage of using pyjamas: you can program your entire experiment in python, and pyjamas will create the browser-dependent javascript code. If you already know javascript, writing your experiment in python without having to worry about browser-dependent issues will save you time. And if you don’t, you don’t have to go through the frustrating process of learning javascript. On the downside, the documentation for pyjamas is currently not very good, so figuring out how to get things to work can take a while.

That’s why I’m providing the (commented) code that I generated to create my MechTurk experiment. A short demo version of the experiment can be found here.

A screenshot of the experiment. Participants were asked to rate on a 7-point scale how natural the statement they heard was as a description of the scene.

The NSF in 2020: The future of the Social, Behavioral,and Economic Sciences

The NSF/SBE released its executive summary of 252 short white papers on the future of the social, behavioral, and economic sciences. Among other things, the report identifies four focus areas (population change; sources of disparities; communication, language, and linguistics; and technology, new media, and social network) and three properties of future research (data-intensive, multidisciplinary, and collaborative). But read for yourself. The report summarizes what the community (authors that submitted white papers) had to say about what works well and what needs to be improved in terms of the processes that are currently employed by the NSF to distribute its funding. On p. 24 an onward, you can read a summary of the many many linguistic white papers that seem to have been submitted (see p. 39 for a summary of which disciplines the white papers came from). On p.29 an onward the report lays out possible scenarios as to how the NSF might change in order to get to the outlined vision.

Google scholar now provides detailed citation report

This might be of interest to some of you: Google Scholar now allows you to correct links or citations to your work. It also provides a complete summary of all your citations, by article, by year, etc. It’s a functionality similar to academia.edu, but it let’s you remove wrong links to your work (e.g. to old prepublished manuscripts).

The interface is rather convenient since it allows you to import all references from scholar, which is almost 95% correct. Overall, it’s actually much more convenient than academia.edu (though I’d say it serves a slightly different purpose). It also generates a list of all your co-authors and other schnick-schnack . Check it out. Sweet.

Lot’s of zeros? Be careful with your chi-square (exact or not) and alike

If you’re running chi-squares to analyze categorical data and you have lots of very low count (or even 0 cells), be careful in how to interpret the result. There’s a nice article by Andrew Gelman on this topic, where he shows that the problem is that all the low counts can make it harder to detect the signal (and hence a significant deviation from the expected values for a part of the table). Put differently, you might have a significant pattern, but not detect. I don’t think it’s so much a problem for most of the tests we conduct since contingency tables in psycholinguistic and linguistic research are usually rather small. I can’t recall the last time that I saw anything larger than a 3×4 or alike. From what I understand from the Gelman’s post, it would seem that the problem he points out becomes more serious the larger the table is.

some (relatively) new funding mechanisms through NSF

This might be of interest to folks, in case you haven’t seen it. First, there’s RAPID and EAGER. RAPID is a mechanism for research that requires fast funding decisions (e.g. b/c the first language with only one phoneme was just discovered but its last speaker is just about to enter into a vow of silence). EAGERs are “Early-concept Grants for Exploratory Research” for exploratory work – i.e. high risk research with a high potential for high pay-off. One important property of both mechanisms is that submissions do not have to be sent out for external review, which should substantially shorten the time until you hear back from NSF.

Second, there is now a new type of proposal that is specifically aimed at interdisciplinary work that would not usually be funded by any of the existing NSF panels alone – CREATIV: Creative Research Awards for Transformative Interdisciplinary Ventures.

Note that all three of these funding types allow no re-submission.

The serial founder hypothesis and word order universals

Check out this article in ScienceNews summarizing commentaries on two recent language studies in Science (Atkinson, 2011: ) and Nature (Dunn et al., 2011). Each of the studies has received a lot of attention and they are the subject of two special issues in press for Linguistic Typology, to which HLP Lab contributed on three articles. I will add a link to the special issue(s) once it comes out.

More papers relevant to questions about information density

And while I am at it, let me post three more papers that are interesting for anyone interested in uniform information density and, more generally, theories of communicatively efficient language production (though most of you may already know these papers):

• They call it speech information rate, but it’s essentially the same: Pellegrine, F., Coupe, C., and Marsico, E. 2011. A cross-linguistic perspective on speech information rate. Language 87(3), 539-558.
• Maurits, L., Perfors, A., and Navarro, D. 2010. Why are some word orders more common than others. A uniform information density account. NIPS.
• S.T. Piantadosi, H. Tily, and E. Gibson. 2011. Word lengths are optimized for efficient communication.Proceedings of the National Academy of Sciences, 108(9):3526.

Lots of food for thought.

UID and text generation

Ah, just when I thought it couldn’t get any better: Uniform Information Density has been applied to text generation . Have a look at this paper (thanks, Raja, for forwarding it):

• Rajakrishnan Rajkumar and Michael White. 2011. Linguistically Motivated Complementizer Choice in Surface Realization. In Proc. of the EMNLP-11 Workshop on Using Corpora in NLG. (bib)

According to Raja (the first author), more on this issue is in progress (e.g. an extension beyond complementizers) and future updates on this work  will be posted on Michael White’s lab at Ohio State.

Belated congratulations to Dave Kleinschmidt

Better late than never: Congratulations to Dave Kleinschmidt for winning the “Student Talk Prize” at the 2011 meeting of Architecture and Mechanisms of Language Processing in Paris, France. If you want to learn more about’s Dave’s work on A Bayesian belief updating model of phonetic recalibration and selective adaptation either have a look at this AMLaP abstract or read Dave’s short ACL paper on some the findings presented at the 2011 Cognitive Modeling and Computational Linguistics workshop in Portland, Oregon (here’s a link to the full proceedings).

If you’re interested in this line of work, you might also enjoy reading Morgan Sonderegger and Alan Yu’s 2010 CogSci paper on A rational account of perceptual compensation for coarticulation, which we learned about recently.

New R resource for ordinary and multilevel regression modeling

Here’ s what I received from the Center of Multilevel Modeling at Bristol (I haven’t checked it out yet; registration seems to be free but required):

The Centre for Multilevel Modelling is very pleased to announce the addition of
R practicals to our free on-line multilevel modelling course. These give
detailed instructions of how to carry out a range of analyses in R, starting
from multiple regression and progressing through to multilevel modelling of
continuous and binary data using the lmer and glmer functions.

MLwiN and Stata versions of these practicals are already available.
You will need to log on or register onto the course to view these
practicals.

Read More...
http://www.cmm.bris.ac.uk/lemma/course/view.php?id=13

LSA 2011 class on Computational Psycholinguistics

Due to popular demand – you can find the Computational Psycholinguistics class Roger Levy and I are currently teaching at the LSA 2011 institute at Boulder mirrored here.

R code for Jaeger, Graff, Croft and Pontillo (2011): Mixed effect models for genetic and areal dependencies in linguistic typology: Commentary on Atkinson

Below I am sharing the R code for our paper on the serial founder effect:

• Jaeger, Graff, Croft, and Pontillo. 2011. Mixed effect models for genetic and areal dependencies in linguistic typology: Commentary on Atkinson. Linguistic Typology 15(2), 281–319. [if you're not subscribed to Linguistic Typology, check out this pre-final draft or contact me for an offprint].

This paper is a commentary on Atkinson’s 2011 Science article on the serial founder model (see also this interview with ScienceNews, in which parts of our comment in Linguistic Typology and follow-up work are summarized). In the commentary, we provide an introduction to linear mixed effect models for typological research. We discuss how to fit and to evaluate these models, using Atkinson’s data as an example.We illustrate the use of crossed random effects to control for genetic and areal relations between languages. We also introduce a (novel?) way to model areal dependencies based on an exponential decay function over migration distances between languages.

Finally, we discuss limits to the statistical analysis due to data sparseness. In particular, we show that the data available to Atkinson did not contain enough language families with sufficiently many languages to test whether the observed effect holds once random by-family slopes (for the effect) are included in the model. We also present simulations that show that the Type I error rate (false rejections) of the approach taken in Atkinson is many times higher than conventionally accepted (i.e. above .2 when .05 is the conventionally accepted rate of Type errors).

The scripts presented below are not intended to allow full replication of our analyses (they lack annotation and we are not allowed to share the WALS data employed by Atkinson on this site anyway). However, there are many plots and tests in the paper that might be useful for typologists or other users of mixed models. For that reason, I am for now posting the raw code. Please comment below if you have questions and we will try to provide additional annotation for the scripts as needed and as time permits. If you find (parts of the) script(s) useful, please consider citing our article in Linguistic Typology.

More on random slopes and what it means if your effect is not longer significant after the inclusion of random slopes

I thought the following snippet from a somewhat edited email I recently wrote in reply to a question about random slopes and what it means that an effect becomes insignificant might be helpful to some. I also took it as an opportunity to updated the procedure I described at http://hlplab.wordpress.com/2009/05/14/random-effect-structure/. As always, comments are welcome. What I am writing below are just suggestions.

[...] an insignificant effect in an (1 + factor|subj) model means that, after controlling for random by-subject variation in the slope/effect of factor, you find no (by-convention-significant) evidence for the effect. Like you suggest, this is due to the fact that there is between-subject variability in the slope that is sufficiently large to let us call into question the hypothesis that the ‘overall’ slope is significantly different from zero.

[...] So, what’s the rule of thumb here? If you run any of the standard simple designs (2×2, 2×3, 2x2x2,etc.) and you have the psychologist’s luxury of plenty of data (24+item, 24+ subject [...]), the full random effect structure is something you should entertain as your starting point. That’s in Clark’s spirit. That’s what F1 and F2 were meant for. [...] All of these approaches do not just capture random intercept differences by subject and item. They also aim to capture random slope differences.

[...] here’s what I’d recommend during tutorials now because it often saves time for psycholinguistic data. I am only writing down the random effects but, of course, I am assuming there are fixed effects, too, and that your design factors will remain in the model. Let’s look at a 2×2 design:

Two interesting papers on mixed models

While searching for something else, I just came across two papers that should be of interest to folks working with mixed models.

• Schielzeth, H. and Forstmeier, W. 2009. Conclusions beyond support: overconfident estimates in mixed models. Behavioral Ecology Volume 20, Issue 2, 416-420.  I have seen the same point being made in several papers under review and at a recent CUNY (e.g. Doug Roland’s 2009? CUNY poster). On the one hand, it should be absolutely clear that random intercepts alone are often insufficient to account for violations of independence (this is a point, I make every time I am teaching a tutorial). On the other hand, I have reviewed quite a number of papers, where this mistake was made. So, here you go. Black on white. The moral is (once again) that no statistical procedure does what you think it should do if you don’t use it the way it was intended to.
• The second paper takes on a more advanced issue, but one that is becoming more and more relevant. How can we test whether a random effect is essentially non-necessary – i.e. that it has a variance of 0? Currently, most people conduct model comparison (following Baayen, Davidson and Bates, 2008).  But this approach is not recommended (and neither do Baayen et al recommend it) if we want to test whether all random effects can be completely removed from the model (cf. the very useful R FAQ list, which states “do not compare lmer models with the corresponding lm fits, or glmer/glm; the log-likelihoods [...] include different additive terms”). This issue is taken on in Scheipl, F., Grevena, S. and Küchenhoff, H. 2008. Size and power of tests for a zero random effect variance or polynomial regression in additive and linear mixed models. Computational Statistics & Data Analysis.Volume 52, Issue 7, 3283-3299. They present power comparisons of various tests.

Mixed model’s and Simpson’s paradox

For a paper I am currently working on, I started to think about Simpson’s paradox, which wikipedia succinctly defines as

“a paradox in which a correlation (trend) present in different groups is reversed when the groups are combined. This result is often encountered in social-science [...]“

The wikipedia page also gives a nice visual illustration. Here’s my own version of it. The plot shows 15 groups, each with 20 data points. The groups happen to order along the x-axis (“Pseudo distance from origin”) in a way that suggests a negative trend of the Pseudo distance from origin against the outcome (“Pseudo normalized phonological diversity”). However, this trend does not hold within groups. As a matter of fact, in this particular sample, most groups show the opposite of the global trend (10 out of 15 within-group slopes are clearly positive). If this data set is analyzed by an ordinary linear regression (which does not have access to the grouping structure), the result will be a significant negative slope for the Pseudo distance from origin. So, I got curious: what about linear mixed models?