Some time ago, I announced that some folks have been thinking about organizing a small workshop on common issues and standards in regression modeling (including multilevel models) in psycholinguistic research to be held the day before CUNY 2009 (i.e. 03/25 at UC Davis). Here’s an update on this “workshop” along with some thoughts for planning.

The target audience for this workshop are researchers that have already used regression models for their research.

The goal is a discussion of quality standards for the field (how should we fit models? when can we trust them? what should we report in papers?). We hope to gather interested researchers who are beginning to use regression and multilevel methods and “experts” who have already used them for some time. Undoubtedly, it will not be the case that there is only one correct statistical method for each problem, but hopefully the discussion will lead to some common ground and increased awareness of what experienced regression users consider “good”. It will be impossible to arrive at standards for all types of models, but the goal is to work towards some general minimum quality standards. Similarly, I don’t anticipate that we will solve all questions that have been sent to me, but maybe we make some progress.

The structure of the workshop will probably consist of some overview talks (see below for topics) and enough time for a forum/discussion and maybe an invited talk on future perspectives, conceptual background of multilevel models, and/or advanced implementation details of currently available multilevel fitting methods.

Invited speaker(s)? Given the spontaneous decision to have this workshop, it’s going to be hard to get invited speakers from outside of the CUNY sentence processing community. But we are trying.

Specific topics and questions:

• Standards of regression model fitting (using the right linking function, dealing with collinearity, guarding against overfitting) and what to report so that reviewers and readers can judge the adequacy of the model
  • standards for random effect in-/exclusions
  • standards for assessing significance of n-ary factor removal for n > 2 in mixed models that are not fit by ML.
  • for more detail see also the previous post.
• How to report, summarize, and visualize regression results (incl. interpretation of coefficients, coding of variables)
• Short refresher on conceptual background for multilevel models
  
• Other open issues:
  • Eye-tracking and other time series data
  • consequences of unbalanced data (incl. data resulting from balanced designs with high exclusion rates)
  • non-gaussian linking functions?
  • future developments, what’s coming up?

Feedback is, of course, very welcome.

And for all you stats nerds: fivethirtyeight.com was almost spot-on about the election outcome, for each state, as well as in terms of the popular vote prediction. For those of you who have followed, you probably noticed that the Gelman also contributed to the blog (from Gelman and Hill). Good to see that proper stats actually works.

If you need a reason, here are some:

and the precessor:

Well, first of all you should be using Firefox or Safari to look at this page, but, ah well, some IE 7 users have run into problem ;). This is seems to be specific to the inability of IE 7 (earlier versions do fine) to handle frames in a normal way. Some posts suggest that you need to change some of your options (see http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=824022&SiteID=; thanks Sara!). You may as well download Firefox, I mean, really …

Out of recent conversations with a whole bunch of folks (e.g. John Trueswell, Jennifer Arnold, Elsi Kaiser, Matt Traxler, Mike Tanenhaus, Jim Magnuson, and more), we came up with the idea to possible hold a workshop on “good practices in ordinary and multilevel regression models” [working title ;)] for researchers working on psycholinguistics/the psychology of language just a day before CUNY 2009 (to be held 03/26-28 at UC Davis), so 03/25 in Davis. This is just a baby of thought at this point, but if you’re interested, I’ve summarized some thoughts below and I’d appreciate your feedback (just leave a comment below and I will receive it).

Motivation

Regression techniques, including multilevel/mixed models, have received increasing attention in the literature (especially, the special issue in JML that’s about to come out) and more and more researchers in psycholinguistics seem to be gviving them a shot. however, there are not (yet) a standard part of the education in statistics for most people in our field. This leads to the problem that more and more work uses these analyses while at the same time there is a large degree of uncertainty among the users as to what constitutes good practice. This workshop is meant to provide a forum to discuss question like: what do you need to do to check whether you can trust your mode? What information do you need to provide in a paper so that readers can evaluate your model?

What the workshop is meant to provide a forum for:

• the meeting is thought of as an informal meeting with at least 50% discussion, question time for researchers that are beginning to work with these models. It’s not meant as an introduction to these models (i.e. not a tutorial on regression or multilevel models; we would be catering mostly to folks who are in the process of using these models, no? I also don’t imagine this to be an advertisement workshop for these models, but rather a forum for those of us who are interested in them to exchange ideas and what we know about best practices).
• we want to keep this meeting simple (it’s too late to organize a huge workshop), lasting between 3 - 6 hours in the afternoon on the day before CUNY?
  
• there would be lectures/introductions to the following issues
  • common issues in regression modeling
    • collinearity
    • overfitting
    • overly influential cases
    • overdispersion?
    • model quality (e.g. residuals for linear models)
    • building a model: adding/removing variables (also: interactions)
  • some solutions to these problems for common model types
    • outlier handling
    • centering
    • removing collinearity (e.g. PCA, residualization)
    • stratification (using subsets of data)
  • interpreting the model, making sure the model answers the question of interest:
    • testing significance (SE-based tests vs. model comparison)
    • interpration of model output, e.g. interpreation of coefficients
    • (also: coding of variables)
    • follow-up tests
  • differences between different models (e.g. ordinary vs. multilevel; linear vs. logit) in terms of available measures of fit; test of significance; etc.
  • what’s out there right now in terms of model types and what’s soon to come? how do these models relate to other models being using in other disciplines? — Harald suggested to invite Doug Bates (the developer of lmer and a first class statistician working on multilevel models; he has apparently attended other similar conferences where folks were interested in multilevel models), Matt mentioned Shelley Blozis (a quantitative psychologists at UC Davis working with various types of multilevel models). I think that is a GREAT idea, because quite frankly they know loads more about these models than we (at least I) do. I think especially for a look into the future and where the field may be heading, they could be very helpful. However, we probably would need some funding to invite Doug Bates, if we want to do that (on the odd chance that he might have time).
    • Any ideas as to how we could get a modest amount of funding, e.g. to invite Doug Bates and/or Shelly Blozis and/or anybody else you would think is a great person for what we have in mind?
    • Update 10/18/08: Several people have offered potential institutional support for this event:
      • Matt Traxler and Tamara Swaab (UC Davis)
      • Center of Language Science (University of Rochester)
      • Institute for Research in Cognitive Science (University of Pennsylvania)
        

That’s my five cents for now. Updates to follow. Please feel free to leave comments.

Florian

I recently visited the GAIN meeting in Cambridge, MA, where about 200-300 German academics met to get information on and discuss the recent reform(s) of the German university system, brain drain (away from Germany), how to reverse it, how to allow young researchers to conduct independent research, etc. Representatives from all major research and education funding institutions (DFG, Alexander von Humboldt, DAAD), members of the parliament, deans from several universities, representatives from independent research foundations (MPI, Frauenhoffer, etc.), and industry representatives gave presentations about recent improvements, persistence problems, and plans for the future. There were also many interesting workshops on, for example, how to start your own lab and how to apply for funding for it (”Nachwuchsgruppe”) or, more generally, on academic and non-academic careers in different disciplines.

If you’re interested in more information, have a look at the GAIN website, which contains the whole program, contact info, and so on. A summaries of the meeting is available, for example, at the site of the German Scholar Organization (GSO). The event also had quite some press echo (see this GAIN summary of the press echo, e.g.,  Stuttgarter Zeitung (in German) –which also “quotes” utterances of me out of context ;), so pls take that one with a grain of salt, make that a bucket full of salt– or Martin Fenner’s blog; I am sure there is more).

My one paragraph summary would be that the situation seems to have improved a lot and that most involved parties in Germany (DFG, the universities, politicians, …) are aware that these changes need to continue. I was good to hear from young researchers that have decided to start there careers in Germany that they are able to conduct their research independently. At the same time, there still are some oddities about the current state of affairs - for example, in some states, “Junior Professors” (young professors, roughly comparable to an associate professor) are not allowed to graduate students because they aren’t “habilitiert”. That is, their doctoral students must take the final examination (Promotion) with a senior colleague of the Junior Professor. Also, it seems that start-up funds are still not comparable with those at top US universities, but IMHO start-up funding is quite essential in that it allows young researchers to conduct their own research without in any way depending on their senior colleagues (well, obviously there is always gonna be influence from senior colleagues and that is good and fair, but financial independence is an important component).

There is much more to be said, but in any case, if you’re a German scholar in the US or a scholar interesting in working in Germany, I think you may be interested in reading up on the recent changes to the German university system and the GAIN website given above is a great starting point for that (check out the GAIN newsletter archive).

We just received word that our group (Florian Jaeger, BCS; Dan Gildea, CS; Lenhart Schubert, CS; Mike Tanenhaus; BCS) was among the six reciepients of the first Provost’s Multidisciplinary Award at the University of Rochester. This will fund a one-year project furthering interdisciplinary collaborations between the Brain and Cognitive Sciences and Computer Science on Computational Psycholinguistics: Integrating Computational and Behavioral Methods to Study Human Language Processing. Thanks to Benjamin Van Durme (CS), Austin Frank (BCS), Celeste Kidd (BCS), and Matt Post (CS) for help with conceiving this project. Let’s get crackin’.

This summer, Austin Frank and I organized a six 3h-session tutorial on regression and mixed models. It is posted on our HLP lab wiki and consists out of reading suggestions and commented R scripts that we went through in class. Among the topics (also listed for each session on the wiki) are:

• linear & logistics regression
• linear & logit mixed/multilevel/hierarchical models
• model evaluation (residuals, outliers, distributions)
• collinearity tests and dealing with collinearity
• coding of variables (contrasts)
• visualization

We used both Baayen’s 2008 textbook Analyzing Linguistic Data: A Practical Introduction to Statistics using R (available online) and Gelman and Hill’s 2007 book on Data Analysis using Regression and Multilevel/Hierarchical Models, both of which we can recommend (they also complement each other nicely). If you have questions about this class or you have suggestions for improvement, please send us an email or leave a comment to this page (we’ll get notified).

Here’s a quick summary of an article about phrasal ordering preferences by H. Yamashita and F. Chang, published 2001 in Cognition. The title of the paper is “Long before short” preference in the production of a head-final language. Enjoy.

Summary: The authors conduct a set of production experiments confirming results from corpus studies which show a preference for long-before-short phrasal orderings in Japanese (Hawkins 1994). This is the opposite of the preference in English heavy NP shift constructions for short-before-long orders (Arnold et al 2000), posing a potential problem for the accessibility-based accounts of phrasal ordering which have been used to explain the findings in English.

Experiment 1: Subjects were shown three boxes of text containing a subject NP, object NP, and a transitive verb in different corners of the screen and were asked to create a sentence using those constituents. The weight of the NPs was manipulated by adding a modifier to the subject NP, the object NP, or to neither. The dependent variable was the percentage of trials in each condition in which subjects produced the shifted DO-Sub-V order. Both orders (i.e. canonical sub-NP obj-NP V and shifted obj-NP sub-NP V) were possible in all trials. While the canonical order was always preferred to the shifted order, object-fronting was more likely when objects were long than when short.

Experiment 2: Same basic procedure as in Exp. 1, but here subjects were shown a direct object-NP, indirect object-NP, and ditransitive V. Weight of the NPs was manipulated by adding a relative clause to the DO-NP, the IO-NP, or neither. In each of these trials, four orderings are possible:

1. Canonical: Sub-NP + IO-NP + DO-NP + Verb
2. IO-shift: IO-NP + Sub-NP + DO-NP + Verb
3. DO-shift: DO-NP + Sub-NP + IO-NP + Verb
4. Sentence-internal DO-shift: Sub-NP + DO-NP + IO-NP + Verb

Both IO-shift and DO-shift were more common when the IO-NPs and DO-NPs are long, respectively. Sentence-internal DO-shift was also more common when DO-NPs were long than when DO- and IO-NPs were short. Finally, when DO-NPs were long, sentence-internal shifting was more common than sentence-initial shifting.

Discussion: The long-before-short preference in Japanese is not consistent with accessibility-based accounts of phrasal ordering, which predict that shorter phrases—having fewer words to retrieve and therefore fewer orderings to select from—are more accessible to speakers, and therefore are likely to be produced earlier in an incremental production system.

To account for the findings in Japanese, the authors appeal to the theory of grammatical encoding (Bock and Levelt 1994; Garrett 1980), and propose that the contradictory patterns in English and Japanese are a function of different levels of the production system, or different levels of constraints within the production system. Specifically, they propose that short-before-long orderings in English are a result of form-based decisions, whereas the long-before-short preference in Japanese is a result of decisions which are more conceptually driven. Long phrases are semantically richer than short phrases, making them more salient and therefore more conceptually accessible than short phrases. On the other hand, long phrases are more costly for retrieval, so short phrases are likely to be available before long ones. Phrase ordering decisions then are subject to two possibly competitive sets of constraints, which may be weighed relative to each other in different ways, depending on the language or grammatical construction under consideration. There is evidence that form-based considerations are more influential post-verbally (Stallings et al. 1998), which could tip the scales in favor of conceptually driven as opposed to form-based decisions in verb-final languages such as Japanese.

–Alex Fine

Elisabeth Norcliffe and I are currently working on a short summary paper on cross-linguistic psycholinguistic work on sentence production (experiments and corpus studies w/ a sufficiently large sample and quantitative investigation). We would like to focus this effort on research on:

• accessibility effects on word order, voice, and other morphosyntactic variations
• radical vs moderate incrementality during sentence formulation
• syntactic persistence (within and across languages)
• agreement
• disfluencies (but not speech errors)

If you know of work in any of these areas that we should consider, we would appreciate your feedback. Just leave a comment at the bottom of this page (it’s a moderated forum, so it will not appear immediately, but usually it does not take us longer than a day to accept your posts). You can also send an email to Florian. Other topics within sentence production would also be of interest. The selection of references for English (below) mostly serves to exemplify the type of reference we are looking for and is, of course, absolutely incomplete ;). We are interested in references to work on languages other than English and we are especially interested in work on understudied languages.

The list of references we have so far is the result of feedback from many folks on the CUNY list (thanks to all of you). I will update this list periodically.

The references below are available in form of an EndNote library at http://www.hlp.rochester.edu/resources/bibs/, starting 08/26/08. [last updated 08/23/08]

• English: (Arnold, Wasow, Losongco, & Ginstrom, 2000; Bates & Devescovi, 1989; Birner, 1994; Bock, 1982, 1986, 1987; Bock et al., 2006; Bock, Eberhard, Cutting, Meyer, & Schriefers, 2001; Bock & Irwin, 1980; Bock & Warren, 1985; Bresnan, Cueni, Nikitina, & Baayen, 2007; Bresnan, Dingare, & Manning, 2001; Bresnan & Hay, 2007; Brown-Schmidt & Konopka, in press; Byrne & Davidson, 1985; Chang, Lieven, & Tomasello, 2008; Clark & Wasow, 1998; F. Ferreira, 1994; F. Ferreira & Henderson, 1998; V. S. Ferreira, 1996, 2003; Gennari, Mirkovic, & MacDonald, 2005; Gleitman, January, Nappa, & Trueswell, 2007; Gomez Gallo, Jaeger, & Smyth, 2008; Griffin & Spieler, 2006; Harbusch & Kempen, 2002; Hartsuiker, Pickering, & Veltkamp, 2004; Hawkins, 1999; Hernandez, Bates, & Avila, 1996; Jaeger, 2005, 2006; Jaeger & Wasow, 2007; Levelt & Maassen, 1981; Loebell & Bock, 2003; Lohse, Hawkins, & Wasow, 2004; MacWhinney & Bates, 1978; McDonald, Bock, & Kelly, 1993; Meijer & Fox Tree, 2003; Myachyko, 2007; Osgood & Bock, 1977; Prat-Sala & Branigan, 2000; Race & MacDonald, 2003; Rosenbach, 2003, 2008; Salamoura & Williams, 2006; Salamoura & Williams, 2007; Sridhar, 1988; Stallings, MacDonald, & O’Seaghdha, 1998; Wasow, 1997)
• British vs. American English: (Bock et al., 2006)
• American vs. New Zealand English: (Bresnan & Hay, 2007)
• Welsh: (Chang et al., 2008)
• German: (Bader & Häussler, in press; Behaghel, 1909/10, 1930; Bresnan & Hay, 2007; Chang et al., 2008; Harbusch & Kempen, 2002; Hartsuiker, Schriefers, Bock, & Kikstra, 2003; Kempen & Harbusch, 2003, 2004a, 2004b, 2005, 2008; Loebell & Bock, 2003; Scheepers, 2003; van Nice & Dietrich, 2003)
• Platt Deutsch: (Strunk, 2005)
• Dutch: (Bock et al., 2001; Brysbaert, Fias, & Noel, 1998; Harbusch & Kempen, 2002; Hartsuiker, Kolk, & Huinck, 1999; Hartsuiker & Kolk, 1998; Hartsuiker, Kolk, & Huiskamp, 1999; Hartsuiker et al., 2003; Hartsuiker & Westenberg, 2000; Levelt, 1981, 1982; Meyer & Bock, 1999; Salamoura & Williams, 2006; van der Beek, 2005)
• Norwegian: (Ovrelid, 2004)
• Italian: (Bates & Devescovi, 1989; MacWhinney & Bates, 1978; Vigliocco, Butterworth, & Semenza, 1995; Vigliocco & Franck, 1999)
• French: (Brysbaert et al., 1998; Chang et al., 2008; Vigliocco & Franck, 1999, 2002)
• Spanish: (Anton-Mendez, Nicol, & Garrett, 2002; Brown-Schmidt & Konopka, in press; Gennari et al., 2005; Hartsuiker et al., 2004; Hernandez et al., 1996; Igoa, 1996; Meijer & Fox Tree, 2003; Prat-Sala & Branigan, 2000; Sridhar, 1988; Vigliocco, Butterworth, & Garrett, 1996)
• Hungarian: (Chang et al., 2008; MacWhinney & Bates, 1978; Sridhar, 1988)
• Finish: (Myachyko, 2007; Sridhar, 1988)
• Russian: (Lorimor, Bock, Zalkind, Sheyman, & Beard, 2008; Myachykov, 2007)
• Estonian: (Chang et al., 2008)
• Slovenian: (Sridhar, 1988)
• Slovak : (Badecker & Kuminiak, 2007)
• Serbo-Croation: (Chang et al., 2008; Gennari et al., 2005)
• Greek: (Branigan & Feleki, 1999; Branigan, Pickering, & Tanaka, 2008; Salamoura & Williams, 2007)
• Turkish: (Kita & Özüyrek, 2003; Kita et al., 2007; Sridhar, 1988)
• Hebrew: (Chang et al., 2008; Sridhar, 1988)
• Kanada: (Sridhar, 1988)
• Tamil: (Chang et al., 2008)
• Japanese: (Branigan et al., 2008; Chang, Kondo, & Yamashita, 2000; Chang et al., 2008; V. S. Ferreira & Yoshita, 2003; Fry, 2003; Kita & Özüyrek, 2003; Kita et al., 2007; Sridhar, 1988; Tanaka, Branigan, & Pickering, 2005; Yamashita & Chang, 2001, 2006)
• Korean: (Choi, 2007; Dennison, 2008; Lee, 2006)
• Chinese (Cantonese): (Chang et al., 2008; Matthews & Yeung, 2001; Peck, 2008; Sridhar, 1988)
• Sesotho : (Chang et al., 2008)
• Fiji: (Byrne & Davidson, 1985)
• Lummi: (Bresnan et al., 2001)
• Odawa: (Christianson & Ferreira, 2005)
• Mayan : (Norcliffe, 2008, In prep.)

• Direct cross-linguistic comparison of more than two languages: (Chang et al., 2008; Clark & Fox Tree, 2002; Gennari et al., 2005; MacWhinney & Bates, 1978; Prat-Sala & Branigan, 2000; Sridhar, 1988)
• Discussion of value of cross-linguistic investigation: (Bates & Devescovi, 1989; Hawkins, 2007; Vigliocco & Kita)

There are many more papers in sociolinguistics on different dialects of English and on other languages, as well as work on language change. However, most of this work does not include psycholinguistic controls. Work on language change often has to be conducted on small data sets.

References

1. Anton-Mendez, I., Nicol, J. L., & Garrett, M. E. (2002). The relation between gender and number agreement processing. Syntax, 5, 1-25.
2. Arnold, J. E., Wasow, T., Losongco, T., & Ginstrom, R. (2000). Heaviness vs. Newness: The effects of structural complexity and discourse status on constituent ordering. Language, 76(1), 28-55.
3. Badecker, W., & Kuminiak, F. (2007). Morphology, agreement, and working memory retrieval in sentence production: Evidence from gender and case in Slovak. Journal of Memory and Language, 56, 65-85.
4. Bader, M., & Häussler, J. (in press). A Corpus Study of the Determinants of Word Order in German. In. University of Konstanz.
5. Bates, E., & Devescovi, A. (1989). Crosslinguistic studies of sentence production. In B. MacWhinney & E. A. Bates (Eds.), The crosslinguistics study of sentence processing (pp. 225-253). Cambridge: Cambridge University Press.
6. Behaghel, O. (1909/10). Beziehungen zwischen Umfang und Reihenfolge von Satzgliedem. Indogermanische Forschungen, 25, 110-142.
7. Behaghel, O. (1930). Von deutscher Wortstellung. Zeitschrift für Deutschkunde, 44, 81-89.
8. Birner, B. J. (1994). Information status and word order: An analysis of English inversion. Language, 70, 233-259.
9. Bock, J. K. (1982). Toward a cognitive psychology of syntax: Information processing contributions to sentence formulation. Psychological Review, 89, 1-47.
10. Bock, J. K. (1986). Exploring levels of processing in sentence production. In G. Kempen (Ed.), Natural language generation: New results in artificial intelligence, psychology, and linguistics (pp. 351-364). Dordrecht, Netherlands: Martinus Nijhoff Publishers.
11. Bock, J. K. (1987). An effect of the accessibility of word forms on sentence structures. Journal of Memory and Language, 26(2), 119-137.
12. Bock, J. K., Butterfield, S., Cutler, A., Cutting, J. C., Eberhard, K. M., & Humphreys, K. R. (2006). Number agreement in British and American English: Disagreeing to agree collectively. Language, 82, 64-113.
13. Bock, J. K., Eberhard, K. M., Cutting, J. C., Meyer, A. S., & Schriefers, H. (2001). Some attractions of verb agreement. Cognitive Psychology, 43, 83-128.
14. Bock, J. K., & Irwin, D. E. (1980). Syntactic effects of information availability in sentence production. Journal of Verbal Learning and Verbal Behavior, 19, 467-484.
15. Bock, J. K., & Warren, R. (1985). Conceptual accessibility and syntactic structure in sentence formulation. Cognition, 21.
16. Branigan, H. P., & Feleki, E. (1999). Conceptual accessibility and serial order in Greek language production. In Proceedings of the 21st Conference of the Cognitive Science Society, Vancouver.
17. Branigan, H. P., Pickering, M. J., & Tanaka, M. (2008). Contributions of animacy to grammatical function assignment and word order during production. Lingua, 118(172-189).
18. Bresnan, J., Cueni, A., Nikitina, T., & Baayen, H. (2007). Predicting the Dative Alternation. In G. Boume, I. Kraemer & J. Zwarts (Eds.), Cognitive Foundations of Interpretation (pp. 69-94). Amsterdam: Royal Netherlands Academy of Science.
19. Bresnan, J., Dingare, S., & Manning, C. D. (2001). Soft constraints mirror hard constraints: Voice and Person in English and Lummi. Paper presented at the LFG01.
20. Bresnan, J., & Hay, J. (2007). Gradient Grammar: An Effect of Animacy of the Syntax of give in New Zealand and American English. Lingua(Special Issue on Animacy).
21. Brown-Schmidt, S., & Konopka, A. E. (in press). Little houses and casas pequeñas: message formulation and syntactic form in unscripted speech with speakers of English and Spanish. Cognition.
22. Brysbaert, M., Fias, W., & Noel, M.-P. (1998). The Whorfian hypothesis and numerical cognition: Is “twentyfour” processed in the same way as “four-and-twenty”? Cognition, 66, 51-77.
23. Byrne, B., & Davidson, E. (1985). On putting the horse before the cart: Exploring conceptual bases of word order via acquisition of a miniature artificial language. Journal of Memory and Language, 24(4), 377-389.
24. Chang, F., Kondo, T., & Yamashita, H. (2000). Conceptual accessibilility influences scrambling in Japanese. Paper presented at the 13th CUNY Conference on Sentence Processing.
25. Chang, F., Lieven, E., & Tomasello, M. (2008). Automatic evaluation of syntactic learners in typologically different languages. Cognitive Systems Research, 9(3), 198-213.
26. Choi, H.-W. (2007). Length and Order: A Corpus Study of Korean Dative-Accusative Construction. Discourse and Cognition, 14, 207-227.
27. Christianson, K., & Ferreira, F. (2005). Planning in sentence production: Evidence from a free word-order language (Odawa). Cognition, 98, 105-135.
28. Clark, H. H., & Fox Tree, J. E. (2002). Using uh and um in spontaneous speaking. Cognition, 84(1), 73-111.
29. Clark, H. H., & Wasow, T. (1998). Repeating words in spontaneous speech. Cognitive Psychology, 37, 201-242.
30. Dennison, H. Y. (2008). Universal versus language-specific conceptual effects on shifted word-order production in Korean: Evidence from bilinguals. Working Papers in Linguistics: University of Hawaii at Manoa, 39(2), 1-16.
31. Ferreira, F. (1994). Choice of passive voice is affected by verb type animacy. Journal of Memory and Language, 33, 715-736.
32. Ferreira, F., & Henderson, J. M. (1998). Linearization strategies during language production. Memory & Cognition, 26, 88-96.
33. Ferreira, V. S. (1996). Is it better to give than to donate? Syntactic flexibility in language production. Journal of Memory and Language, 35, 724-755.
34. Ferreira, V. S. (2003). The persistence of optional complementizer production: Why saying “that” is not saying “that” at all. Journal of Memory and Language, 48, 379-398.
35. Ferreira, V. S., & Yoshita, H. (2003). Given-New Ordering Effects on the Production of Scrambled Sentences in Japanese. Journal of Psycholinguistic Research, 32(6), 669-692.
36. Fry, J. (2003). Ellipsis and Wa-Marking in Japanese Conversation: Routledge.
37. Gennari, S., Mirkovic, J., & MacDonald, M. (2005). The role of animacy in relative clause production. Paper presented at the 18th Annual CUNY Conference on Human Sentence Processing.
38. Gleitman, L. R., January, D., Nappa, R., & Trueswell, J. C. (2007). On the give and take between event apprehension and utterance formulation. Journal of Memory and Language, 57(4), 544-569.
39. Gomez Gallo, C., Jaeger, T. F., & Smyth, R. (2008). Incremental Syntactic Planning across Clauses. In Proceedings of the 30th Annual Meeting of the Cognitive Science Society (CogSci08) (pp. 845-850).
40. Griffin, Z. M., & Spieler, D. H. (2006). Observing the what and when of language production for different age groups by monitoring speakers’ eye movements. Brain and Language, 99, 272-288.
41. Harbusch, K., & Kempen, G. (2002). A quantitative model of word order and movement in English, Dutch and German complement constructions. In Proceedings of the 19th International Conference on Computational Linguistics (COLING-2002), Taipei (Taiwan) (pp. 328-334). San Francisco: Morgan Kaufmann.
42. Hartsuiker, R. J., Kolk, H. H., & Huinck, W. J. (1999). Agrammatic production of subject-verb agreement: The effect of conceptual number. Brain and Language, 69, 119-160.
43. Hartsuiker, R. J., & Kolk, H. H. J. (1998). Syntactic Persistence in Dutch. Language and Speech, 41(2), 143-184.
44. Hartsuiker, R. J., Kolk, H. H. J., & Huiskamp, P. (1999). Priming word order in sentence production. The Quarterly Journal of Experimental Psychology Section A, 52(1), 129-147.
45. Hartsuiker, R. J., Pickering, M. J., & Veltkamp, E. (2004). Is Syntax Separate or Shared Between Languages? Psychological Science, 15(6), 409-414.
46. Hartsuiker, R. J., Schriefers, H. J., Bock, J. K., & Kikstra, G. M. (2003). Morphophonological influences on the construction of subject-verb agreement. Memory & Cognition, 31, 1316-1326.
47. Hartsuiker, R. J., & Westenberg, C. (2000). Word order priming in written and spoken sentence production. Cognition, 75(2), 27-39.
48. Hawkins, J. A. (1999). The relative ordering of prepositional phrases in English: Going beyond manner-place-time. Language Variation and Change, 11, 231-266.
49. Hawkins, J. A. (2007). Processing typology and why psychologists need to know about it. New Ideas in Psychology, 25, 87-107.
50. Hernandez, A. E., Bates, E. A., & Avila, L. X. (1996). Processing across the language boundary: A cross-modal priming study of Spanish-English bilinguals. Journal of Experimental Psychology: Learning, Memory, and Cognition, 22(4), 846-864.
51. Igoa, J. M. (1996). The Relationship between Conceptualization and Formulation Processes in Sentence Production: Some Evidence from Spanish. In M. Carreiras, J. E. Garcia-Albea & N. Sebastián-Gallés (Eds.), Language Processing in Spanish (pp. 305-351). Mahwah, NJ: Lawrence Erlbaum Associates.
52. Jaeger, T. F. (2005). Optional “that” indicates production difficulty: Evidence from disfluencies. In Proceedings of Disfluency in Spontaneous Speech, Aix-en-Provence, September 2005 (pp. 103-109).
53. Jaeger, T. F. (2006). Redundancy and Syntactic Reduction in Spontaneous Speech. Stanford University.
54. Jaeger, T. F., & Wasow, T. (2007). Processing as a Source of Accessibility Effects on Variation. Paper presented at the Proceedings of the 31st Annual Meeting of the Berkeley Linguistic Society.
55. Kempen, G., & Harbusch, K. (2003). Word order scrambling as a consequence of incremental sentence production. In H. Härtl & H. Tappe (Eds.), Mediating between concepts and grammar (pp. 141-164). Berlin: Mouton De Gruyter.
56. Kempen, G., & Harbusch, K. (2004a). A corpus study into word order variation in German subordinate clauses: Animacy effects linearization independently of grammatical function assignment. In T. Pechmann & C. Habel (Eds.), Multidisciplinary approaches to language production (pp. 173-181). Berlin, Germany: Mouton De Gruyter.
57. Kempen, G., & Harbusch, K. (2004b). Generating natural word orders in a semi-free word order language: Treebank-based linearization preferences for German. In A. Gelbukh (Ed.), Computational Linguistics and Intelligent Text Processing (pp. 350-354). Berlin: Springer Verlag.
58. Kempen, G., & Harbusch, K. (2005). The relationship between grammaticality ratings and corpus frequencies: A case study into word order variability in the midfield of German clauses. In S. Kepser & M. Reis (Eds.), Linguistic Evidence - Empirical, Theoretical, and Computational Perspectives (pp. 329-349). Berlin: Mouton De Gruyter.
59. Kempen, G., & Harbusch, K. (2008). Comparing linguistic judgments and corpus frequencies as windows on grammatical competence: A study of argument linearization in German clauses. In A. Steube (Ed.), The Discourse Potential of Underspecified Structures (pp. 179-192). Berlin: DeGruyter.
60. Kita, S., & Özüyrek, A. (2003). What does cross-linguistic variation in semantic coordination of speech and gesture reveal? Evidence for an interface representation of spatial thinking and speaking. Journal of Memory and Language, 48, 16-32.
61. Kita, S., Özüyrek, A., Allen, S., Brown, A., Furman, R., & Ishizuka, T. (2007). Relations between syntactic encoding and co-speech gestures: Implications for a model of speech and gesture production. Language and Cognitive Processes, 22, 1212-1236.
62. Lee, H. (2006). Parallel optimization in case systems: Evidence from case ellipsis in Korean. Journal of East Asian Linguistics, 15, 69-96.
63. Levelt, W. J. M. (1981). The speaker’s linearization problem. Philosophical Transactions of the Royal Society of London, B 295(1077), 305-315.
64. Levelt, W. J. M. (1982). Linearization in describing spatial networks. In S. Peters & E. Saarinen (Eds.), Processes, Beliefs, and questions. Dordrecht: D. Reidel.
65. Levelt, W. J. M., & Maassen, B. (1981). Lexical search and order of mention in sentence production. In W. Klein & W. Levelt (Eds.), Crossing the boundaries in linguistics (pp. 221-252). Dordrecht: Reidel.
66. Loebell, H., & Bock, J. K. (2003). Structural priming across languages. Linguistics, 41, 791-824.
67. Lohse, B., Hawkins, J., & Wasow, T. (2004). Processing domains in English verb-particle construction. Language, 80(2), 238-261.
68. Lorimor, H., Bock, J. K., Zalkind, E., Sheyman, A., & Beard, R. (2008). Agreement and attraction in Russian. Language and Cognitive Processes, 23(6), 769-799.
69. MacWhinney, B., & Bates, E. A. (1978). Sentential Devices for Conveying Givenness and Newness: A Cross-Cultural Developmental Study. Journal of Verbal Learning and Verbal Behavior, 17, 539-558.
70. Matthews, S., & Yeung, L. Y. Y. (2001). Processing motivations for topicalization in Cantonese. In K. Horie & S. Sato (Eds.), Cognitive-functional linguistics in an East Asian context (pp. 81-102). Tokyo: Kurosio Publishers.
71. McDonald, J. L., Bock, K., & Kelly, M. H. (1993). Word and word order: Semantic, phonological, and metrical determinants of serial position. Cognitive Psychology, 25(2), 188-230.
72. Meijer, P. J., & Fox Tree, J. E. (2003). Building syntactic structures in speaking: a bilingual exploration. Experimental Psychology, 50(3), 184-195.
73. Meyer, A. S., & Bock, J. K. (1999). Representations and processes in the production of pronouns: Some perspectives from Dutch. Journal of Memory and Language, 41, 281-301.
74. Myachykov, A. (2007). Integrating Perceptual, Semantic and Syntactic Information in Sentence Production. University of Glasgow.
75. Norcliffe, E. (2008). Variation and categorical constraints in Yucatec Maya relative clause constructions. Paper presented at the 82nd Annual Meeting of the Linguistic Society of America, Chicago.
76. Norcliffe, E. (In prep.). Syntactic variation in cross-linguistic perspective: A view from Yucatec Maya. Unpublished Doctoral dissertation, Stanford University.
77. Osgood, C. E., & Bock, J. K. (1977). Salience and sentencing: Some production principles. Sentence Production - Develepments in Research and Theory, 89-140.
78. Ovrelid, L. (2004). Disambiguation of syntactic functions in Norwegian: modeling variation in word order interpretations conditioned by animacy and definiteness. In Proceedings of the 20th Scandinavian Conference of Linguistics.
79. Peck, J. (2008). The positional variations of prepositional phrases in Chinese: Synchronic and diachronic perspectives. Stanford University, Stanford.
80. Prat-Sala, M., & Branigan, H. P. (2000). Discourse constraints on syntactic processing in language production: A corss-linguistic study in English and Spanish. Journal of Memory and Language, 42, 168-182.
81. Race, D. S., & MacDonald, M. C. (2003). The use of “that” in the production and comprehension of object relative clauses. Proceedings of the 25th Annual Meeting of the Cognitive Science Society.
82. Rosenbach, A. (2003). Aspects of iconicity and economy in the choice between the s-genitive and of-genitive in English. In G. Rohdenburg & B. Mondorf (Eds.), Determinants of Grammatical Variation in English (pp. 379-412). Berlin, New York: Mouton de Gruyter.
83. Rosenbach, A. (2008). Animacy and grammatical variation-Findings from English genitive variation. Lingua, 118, 151-171.
84. Salamoura, A., & Williams, J. N. (2006). Lexical activation of cross-language syntactic priming. Bilingualism: Language and Cognition, 9, 299-307.
85. Salamoura, A., & Williams, J. N. (2007). Processing verb argument structure across languages: Evidence for shared representations in the bilingual mental lexicon. Applied Psycholinguistics, 28, 627-660.
86. Scheepers, C. (2003). Syntactic priming of relative clause attachments: Persistence of structural configuration in sentence production. Cognition, 89, 179-205.
87. Sridhar, S. N. (1988). Cognition and Sentence Production: A Cross-linguistic study. New York: Springer Verlag.
88. Stallings, L. M., MacDonald, M. C., & O’Seaghdha, P. G. (1998). Phrasal ordering constraints in sentence production: Phrase length and verb disposition in heavy-NP shift. Journal of Memory and Language, 39, 392-417.
89. Strunk, J. (2005). The role of animacy in the nominal possessive constructions of modern low Saxon. Paper presented at the Pionier Workshop on ‘Animacy’.
90. Tanaka, M. N., Branigan, H. P., & Pickering, M. J. (2005). The role of animacy in Japanese sentence production. Paper presented at the CUNY Sentence Processing Conference.
91. van der Beek, L. J. (2005). Topics in Corpus-Based Dutch Syntax. Rijksuniversiteit Groningen.
92. van Nice, K. Y., & Dietrich, R. (2003). Task-sensitivity of animacy effects: Evidence from German picture descriptions. Linguistics, 5, 825-849.
93. Vigliocco, G., Butterworth, B., & Garrett, M. E. (1996). Subject-verb agreement in Spanish and English: differences in the role of conceptual constraints. Cognition, 61, 261-298.
94. Vigliocco, G., Butterworth, B., & Semenza, C. (1995). Constructing subject-verb agreement in speech: The role of semantic and morphological factors. Journal of Memory and Language, 34, 186-215.
95. Vigliocco, G., & Franck, J. (1999). When sex and syntax go hand in hand: Gender agreement in language production. Journal of Memory and Language, 40, 455-478.
96. Vigliocco, G., & Franck, J. (2002). Subject-verb agreement errors in French and English: The role of syntactic hierarchy. Language and Cognitive Processes, 17, 371-404.
97. Vigliocco, G., & Kita, S. Language-specific effects of meaning, sound and syntax: Implications for models of lexical retrieval in production. Language and Cognitive Processes.
98. Wasow, T. (1997). End-weight from the speaker’s perspective. Journal of Psycholinguistics Research, 26, 347-362.
99. Yamashita, H., & Chang, F. (2001). Long before short preference in the production of a head-final language. Cognition, 81(2), B45-B55.
100. Yamashita, H., & Chang, F. (2006). Sentence production in Japanese. In M. Nakayama, R. Mazuka & Y. Shirai (Eds.), Handbook of East Asian Psycholinguistics (Volume 2, Japanese). Cambridge, UK: Cambridge University Press.