Some thoughts on Healey et al (2014) failure to find syntactic priming in conversational speech

In a recent PLoS one article, Healey, Purver, and Howes (2014) investigate syntactic priming in conversational speech, both within speakers and across speakers. Healey and colleagues follow Reitter et al (2006) in taking a broad-coverage approach to the corpus-based study of priming. Rather than to focus on one or a few specific structures, Healey and colleagues assess lexical and structural similarity within and across speakers. The paper concludes with the interesting claim that there is no evidence for syntactic priming within speaker and that alignment across speakers is actually less than expected by chance once lexical overlap is controlled for. Given more than 30 years of research on syntactic priming, this is a rather interesting claim. As some folks have Twitter-bugged me (much appreciated!), I wanted to summarize some quick thoughts here. Apologies in advance for the somewhat HLP-lab centric view. If you know of additional studies that seem relevant, please join the discussion and post. Of course, Healey and colleagues are more than welcome to respond and correct me, too.

First, the claim by Healey and colleagues that “previous work has not tested for general syntactic repetition effects in ordinary conversation independently of lexical repetition” (Healey et al 2014, abstract) isn’t quite accurate.

The authors discuss a few studies that speak to this issue without quite doing what the authors would like to do (e.g., Reitter et al., 2006, Szmrecsanyi, 2004, 2005). Crucially, there are also some studies that have done what the authors claim hasn’t been done. For example (and I suspect there might be more):

1. Jaeger and Snider (2013-Cognition) and Recchia (2007) investigate syntactic priming in ditransitives in conversational speech, while controlling for verb overlap. Unlike in Healey et al, significant priming is observed, independent of lexical overlap. As a matter of fact, lexical overlap did not affect prime strength much (Study 1).
2. Snider (2009-Thesis) investigate syntactic priming in ditransitives in conversational speech, while controlling for lexical overlap, including both verbs and other parts of the sentence. Some of these analyses are presented in Jaeger and Snider (2007) and Jaeger and Snider (2008), along with other relevant investigations of that-omission priming in conversational speech within and across speakers (based on my 2006 CUNY presentation).

None of these studies are cited by the authors, so I suspect they weren’t no aware of them (which has happened to me – so no blame). How do these studies compare to Healey et al (2014)? Unlike Healey et al, the studies I listed above focus on a specific structure. In my view some advantages of this are:

• Meaning differences between the alternative structures being held more constant
• A better understanding of what one is studying
• Comparability with psycholinguistic experiments, all of which have focused on specific structures (e.g., in order to avoid confounds due to differences in meaning)

But there are also disadvantages to this focus on a specific structure. Put differently, there are advantages of the approach taken by Healey et al (2014; pioneered as far as I know by Reitter et al., 2006, though please correct me if I am wrong):

• Broad coverage ensures that one isn’t studying properties that only apply to some subset of the grammatical structures of a language

Of these four differences between the broad-coverage and the structure-specific approach, I’d consider the most crucial issue to be that the broad-coverage approach (as it has been applied so far, including by Healey and colleagues) does not control for (near) meaning-equivalence of the structural choices. Syntactic priming is about the choice of one structure over another under meaning equivalence or near-meaning equivalence. If a structure is chosen or not chosen because another meaning is conveyed, that wouldn’t standardly be considered syntactic priming (for good reasons). It is not clear to me how (near) meaning-equivalence is guaranteed in the broad-coverage approach employed in, for example, Healey et al (2014) and Reitter et al (2006). For example, if an NP node is expanded into NP –> DT N, it’s not clear that it could have been expanded into NP –> DT ADJ N or even NP –> N CP while still referring to the same entity without leading to semantic, pragmatic, and discourse structural violations. [this paragraph has been edited in response to a request for clarification by Patrick Healey]

This is critical because we already know that what people say changes throughout discourse. This is even reflected in the entropy profiles throughout discourse (Genzel and Charniak, 2002, 2003; Qian and Jaeger, 2012), including entropy profiles derived by parsers (Keller, 2004). This means, it is not expected that syntactic distributions do not change throughout conversations and, critically, one very plausible cause for this is that what changes throughout discourse is what messages (in the psycholinguistic sense) we encode. Since syntactic priming is about how (the same) message get encoded, this constitutes a rather glaring potential confound for the study presented by Healey and colleagues.

The syntactic priming studies cited above (Jaeger and Snider, 2007, 2008, 2013; Snider, 2009) also differ from Healey et al in that they used exclusively hand-annotated syntactic corpora. Having worked with automatically parsed corpora (like the BNC, employed in Healey et al’s study), I’m quite aware of the many pitfalls of relying on automatic parse (for discussion, see Jaeger, 2011-Chapter). I wish Healey et al would have discussed this issue in more depth and the steps they took to address problems such as false positives and negatives, which can be very substantial (for an example, see Jaeger, 2011). As far as I can tell, over 93% of the data analyzed by Healey et al comes from automatic parses of the BNC.

Summary

It seems critical to resolve these conflicting results. While there seems to be broad agreement between the different approaches that syntactic priming effects in conversational speech are weaker across than between speakers, the more controlled studies have found a) significant priming within speakers and b) no evidence for anti-priming across speakers, contrary to Healey et al (2014). As much as I like the broad-coverage approach taken by Healey and colleagues, there are considerably potential confounds (see above). Without further evidence I would this not be convinced that the results of Healey et al hold.

Let me conclude with a link to another line of work that is of interest in this context. Several studies have investigated how attitude and perceived group membership can affect alignment across speakers. Most of this research has focused on phonetic alignment (e.g., Babel, 2010, 2011, and references therein), but there is some work on socially mediated effects on syntactic alignment. In a recent web-based study, a joint collaboration between OSU and my lab (Weatherholtz et al, in press) found that syntactic alignment was socially mediated: the strength of syntactic priming depended on the speakers’ willingness to compromise as well as the perceived social distance to a previously encountered speaker. This goes rather well with some of the discussion in Healey et al (2014). Crucially, however, Weatherholtz and colleagues directly tested whether social mediated ever led to anti-priming, rather than just reduced priming. These two potential outcomes would argue for a rather different cognitive architecture (e.g., regarding the automaticity of the mechanisms underlying syntactic priming, for further discussion, see Weatherholtz et al., in press) At least in our study (which, however, did not involve direct communication with an interlocutor and thus won’t be the ultimate word on this issue), no anti-priming was observed (though social distance did cause speakers to not prime). I think further weaving these different lines of research together will be an interesting step for future work.

PS: One additional potential difference is that the syntactic priming studies by Neal Snider and me focus on on corpora that exclusively contain conversational speech. I couldn’t tell from the description in Healey et al (2014) whether all types of speech in the 10 million word spoken BNC were used or whether only unscripted conversations were extracted (the spoken BNC contains many monologues and scripted speech, such as parliamentary addressed). I suspect and hope it is the former, given the authors’ repeated focus on unscripted conversational speech throughout the paper. 

References

Babel, M. (2010). Dialect divergence and convergence in New Zealand English. Language in Society, 39, 437–456. doi:10.1017/S0047404510000400

Babel, M. (2011). Evidence for phonetic and social selectivity in spontaneous phonetic imitation. Journal of Phonetics, 40(1), 177–189.

Genzel, D., & Charniak, E. (2002). Entropy Rate Constancy in Text. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL) (pp. 199–206).

Genzel, D., & Charniak, E. (2003). Variation of entropy and parse trees of sentences as a function of the sentence number. Proceedings of the 2003 Conference on Empirical Methods in Natural Language Processing –, 10, 65–72. doi:10.3115/1119355.1119364

Jaeger, T. F. (2011). Corpus-based Research on Language Production: Information Density and Reducible Subject Relatives. In E. M. Bender & J. E. Arnold (Eds.), Language from a Cognitive Perspective: Grammar, Usage, and Processing. Studies in honor of Tom Wasow (pp. 161–197). Stanford: CSLI Publications.

Jaeger, T. F., & Snider, N. (2007). Implicit Learning and Syntactic Persistence: Surprisal and Cumulativity. University of Rochester Working Papers in the Language Sciences, 3(1), 26–44.

Jaeger, T. F., & Snider, N. (2008). Implicit learning and syntactic persistence : Surprisal and Cumulativity. In B. C. Love, K. McRae, & V. M. Sloutsky (Eds.), Proceedings of the 30th Annual Meeting of the Cognitive Science Society (CogSci08) (pp. 1061–1066). Austin, TX: Cognitive Science Society.

Jaeger, T. F., & Snider, N. E. (2013). Alignment as a consequence of expectation adaptation: Syntactic priming is affected by the prime’s prediction error given both prior and recent experience. Cognition, 127(1), 57–83. doi:10.1016/j.cognition.2012.10.013

Keller, F., & Eh, E. (2004). The Entropy Rate Principle as a Predictor of Processing Effort : An Evaluation against Eye-tracking Data. In F. Keller (Ed.), In Proceedings of the Conference on Empirical Methods in Natural Language Processing (pp. 1–8). Edinburgh, UK: University of Edinburgh.

Qian, T., & Jaeger, T. F. (2012). Cue effectiveness in communicatively efficient discourse production. Cognitive Science, 36(7), 1312–36. doi:10.1111/j.1551-6709.2012.01256.x

Reitter, D., Moore, J. D., & Keller, F. (2006). Priming of Syntactic Rules in Task-Oriented Dialogue and Spontaneous Conversation. In Proceedings of the 28th Annual Conference of the Cognitive Science Society (pp. 1–6).

Szmrecsanyi, B. (2004). Persistence phenomena in the grammar of spoken english. Albert-Ludwigs-Universität.

Szmrecsanyi, B. (2005). Language users as creatures of habit: A corpus-based analysis of persistence in spoken English. Corpus Linguistics and Linguistic Theory, 1(1), 113–150. doi:10.1515/cllt.2005.1.1.113

Weatherholtz, K., Campell-Kibler, K., and Jaeger, T. F. in press. Socially-mediated syntactic Alignment. Language Variation and Change.