Spatiotemporal coupling between speech and manual motor actions (Parrell, Goldstein, Lee & Byrd, 2014) – HIBAR

Another summary and HIBAR post on our most recent lab group reading.

Speech and motor activity co-ordination has long been observed and has been empirically demonstrated across the lifespan, from co-occurrence of babbling and rhythmic limb movement (e.g. Iverson, Hall, Nickel & Wozniak, 2007) to adult co-production of gestures with speech to a level of high temporal resolution. For example, gestures such as pointing tend to align exactly at the point of stressed syllable in a word (Rouchet-Capellan, Laboissiere, Galvan, & Schwarz, 2008).

The importance of motor coupling. Donald showing us how it's done.

The utility of speech and gesture coupling. Donald showing us how it’s done.

Parrell et al. (2014) aimed to probe a bit further the role of prosodic structure in the spatiotemporal coordination of speech and gesture.

Design

The experimenters had four participants tap their right finger on their left shoulder, while synchronously repeating a monosyllable. Midway through a 30s trial, they were required to impose a stress either on the finger tapping movement or in the production of the word, but maintain an unaffected performance in the other domain.  This was achieved by asking participants to watch a clock dial and impose the stress as it reached one of the quarter markers.

The kinematics being monitored were of lip aperture (LA) and fingertip movement (FT) and these were tracked by transducers attached to the articulators.  In order to address spatial effects, the magnitude of the emphasised repetition and mean magnitude of unemphasised trial repetitions were compared. Magnitude in LA and FT were characterised as the wideness/height (amplitude) of the aperture or tap.
Inter-response interval (IRI) was used to measure temporal effects of the emphasis and this was calculated as the time between the onsets of concurrent articulator repetitions.

Spatial Effects

While participants had been instructed to maintain unstressed movement in the uninstructed domain, three subjects showed increased magnitude in articulator repetitions concurrent with an emphasis in the other articulator.  There was also found to be a significant correlation between general movement magnitudes of both articulators (i.e. during unstressed repetitions).

Temporal Effects

A lengthening of IRIs were found near the stress boundary for both LA and TP regardless of stress domain, but a tendency was found for the lengthening in the unstressed domain to be delayed by one repetition.  Similarly to the spatial effects, a general correlation was found between temporal effects in both articulators.  Consistent with previous literature, they also found more robust effects of speech emphasis on finger tapping than vice versa, suggesting a closer coupling in this direction.

Theoretical Implications

The authors venture some possible interpretations of the data, including the π-gesture model of speech prosody. This proposes that all motor activity is controlled by a single internal clock recruited in accordance with prosodic structure. While comparable IRI-lengthening within and across modalities might indicate a common clock, a single driver does not account for the observed delay in the unstressed modality. In addition, the asymmetrical effects of stress across domain indicate that simple coupling the speech and finger articulators is an insufficient explanation.

The alternative hypothesis is that emphasis acted as a perturbation to the coordinative dynamics of the articulators. The delayed IRI in the unstressed domain would reflect a restoration of relative phase between articulators. Prosody is suggested to act as a means of grouping certain information and making it salient, therefore it is possible that this process recruits wider bodily resources than speech apparatus to achieve this. This hypothesis would also account for the asymmetry of stress domain effect, as the prosodic structure of language is stress in the speech domain. This then calls on a wider set of motor resources which form part of a larger prosodic architecture, where stress in the finger tapping domain will not do the reverse (1).

There are some points we would have raised, had we been reviewers of this paper:

1. We were unsure about why the sample size (four) for this study was so small, especially as there was a large amount of individual variation in performance and also that the observed effects were relatively small. This makes it harder to interpret “majority” (3/4) effects such as the augmented cross-modal effect for speech stress (see paper for a number of these instances)

2. The authors note that for each condition (stressed finger tap/syllable) they presented two blocks with the syllables /ma/ and /mop/. The reason cited was to investigate effects of syllable coda (the optional final part of a syllable) on the amplitude and timing of articulator movement. This was not elaborated on and it is unclear to us what justified including this factor, as no previous literature was cited (2).

3. While the authors aimed to reproduce a more natural prosodic context than previous studies (3), it could be argued that timing a stress according to an external stimulus (point on a clock dial) may simply be unnaturalistic in another manner. While the emphasis is ostensibly quasi-linguistic (i.e. a specific point in a speech string), it is possible that explicitly constraining the placement of the stress could have dynamical consequences on the preceding and following movements that would not be present in a natural language activity. The authors cite some previous literature for using this type of stress, but I currently don’t have access to these references. It is unclear whether participants in these cases were volitionally choosing which gesture to stress or whether they also had to time this to a clock

(1) General coupling principles do explain the domain-general effects in which a smaller amplitude change is seen in the unstressed domain and also the correlation between domains of spatial and temporal effects during unstressed repetitions
(2) N.B.The authors found no effect of presence of coda on spatial or temporal effects
(3) These typically employed alternating stressed-unstressed patterns which imposed a rhythm, something which the current authors were keen to avoid

References

Iverson, J. M., Hall, A. J., Nickel, L., & Wozniak, R. H. (2007). The relationship between reduplicated babble onset and laterality biases in infant rhythmic arm movements. Brain and language, 101(3), 198-207.

Parrell, B., Goldstein, L., Lee, S., & Byrd, D. (2014). Spatiotemporal coupling between speech and manual motor actions. Journal of phonetics, 42, 1-11.

Rochet-Capellan, A., Laboissière, R., Galván, A., & Schwartz, J. L. (2008). The speech focus position effect on jaw–finger coordination in a pointing task. Journal of Speech, Language, and Hearing Research, 51(6), 1507-1521.

 

Advertisements