Intersecting kinematic encoding and readout of intention in autism

Stimuli were selected from a dataset of 940 grasping actions obtained by recording 20 TD and 20 ASD children performing grasp-to-pour and grasp-to-place actions. For grasp-to-pour trials, a glass (height = 10 cm; diameter = 6.5 cm) was placed 19 cm from the bottle. Participants were instructed to reach for the bottle, lift it, and pour some water into the glass. A coexperimenter refilled the bottle on each trial. For grasp-to-place trials, a box (height = 6 cm; diameter = 10 cm) was placed 19 cm from the bottle. Participants were instructed to reach for the bottle, lift it, and place it in the box. Detailed procedures and the apparatus are described in ref. 13. Briefly, reach-to-grasp movements were tracked using a near-infrared camera motion capture system with six optical cameras (frame rate, 100 Hz; Vicon System) and simultaneously filmed from a lateral viewpoint using a video camera fully synchronized with the optical cameras (Vicon Vue; 100 frames/s, resolution 1,280 × 720). As previously described (13), the child’s right hand was outfitted with retroreflective hemispheric markers (6.5 mm in diameter) placed on the metacarpal joint and the tip of the index and the little finger, the trapezium bone and the tip of the thumb, the radial aspect of the wrist, and the hand dorsum. This marker set allowed us to finely track both the distal (hand shape) and proximal (transport) components of the motions. The kinematic data were run through a 6-Hz low-pass Butterworth filter. Based on previous studies investigating prospective action control in TD and ASD children (12, 13), we extracted 15 kinematic variables (SI Appendix, Fig. S2):

Custom software (MATLAB; MathWorks Inc.) was used to extract the selected variables. Each variable was calculated at intervals of 10% of the movement duration from reach onset to reach offset.

From the dataset of grasping actions, we selected, for each group, 50 grasping actions (grasp to place, n = 25; grasp to pour, n = 25) according to the following criteria: 1) minimized within-intention distance (using the metric reported in ref. 19) and 2) mean duration of movements not significantly different between intentions. Video clips that corresponded to the selected reach-to-grasp actions were used as stimuli for the intention discrimination task. Each video clip began with reach onset and ended with the contact between the hand and the bottle. To allow participants sufficient time to focus on the reach onset, static frames ranging in duration from 160 to 800 ms (in 160-ms increments) were randomly added to the beginning of each video.

Participants were seated in front of a 24-inch computer monitor (resolution 1,280 × 720; 100 Hz) at a viewing distance of 50 cm. The task structure conformed to a one-interval forced choice task with binary choice (to place vs. to pour). Each trial began with the presentation of a white central fixation cross for 1,000 ms. Then, a video clip showing the reach-to-grasp action was presented. After the video (followed by a waiting window of 80 ms), a screen prompted participants to indicate the action (to place or to pour) that would follow the observed grasp (5,000 ms). For half of the participants, the Italian word “mettere” (to place) on the left prompted a button press with the index finger of the left hand, and the word “versare” (to pour) on the right prompted a button press with the index finger of the right hand. The position of the two words was counterbalanced across participants. Participants completed two sessions in which they observed reach-to-grasp actions performed by TD children and ASD children in counterbalanced order. Each session consisted of 50 experimental trials performed in three blocks (10, 20, and 20 trials), with a 2-min break between each block. Participants received no feedback either during the experimental blocks or during the practice block.

The task was revised and administered by a clinically experienced experimenter. During the experiment, the experimenter positioned herself behind the child. The experimenter was the same for all participants. Participants were introduced to the stimuli and were given both written and verbal instructions. Practice trials (n = 20) were included before the experimental session to familiarize the child with the task and ensure that they had understood the task. Participants were instructed to respond as accurately and quickly as possible during the presentation of the prompt screen. If they did not comply with the instructions (e.g., they responded during video presentation), the experimenter would ask them to repeat practice trials. We verified that the number of participants who repeated the practice did not differ between groups (six in the ASD group and six in the TD group). The proportion of responses during video presentation or within the first 100 ms of the response window was generally low (TD observers, mean ± SEM = 0.021 ± 0.007; ASD observers, mean ± SEM = 0.015 ± 0.004) and did not differ between groups [t₍₆₈₎ = 0.752, P = 0.455]. Stimuli presentation, timing, and randomization were controlled using E-prime V2.0 software (Psychology Software Tools).

Gaze direction was measured with an infrared eye tracker (SMI RED500; SensoMotoric Instruments). The eye tracker suffered a fatal technical failure before testing was completed; moreover, calibration of the eye tracker was unsuccessful in some participants. Therefore, eye-tracking data are available for 35 TD participants and 23 ASD participants. For each observer and each trial, we extracted the sequence of spatial position coordinates (scan path). We computed the fraction of time in which the scan path was within the screen in each trial and then, averaged this value across trials for each observer. This fraction was overall high (TD observers, mean ± SEM = 0.91 ± 0.01; ASD observers, mean ± SEM = 0.88 ± 0.03) and did not differ between groups [t₍₅₆₎ = 0.860, P = 0.393].

Trials for which participants provided a response during the waiting window or within the first 100 ms of the response window were discarded from analyses (<2% of trials). We verified that the pattern of results and their significance remained similar even when all trials were included.

We used mixed effects models to assess the significance of differences in intention discrimination performance (Fig. 1D), response bias (SI Appendix, Fig. S1B), and encoding and readout model performance (Figs. 2C and 3B, respectively) compared with chance and across observer groups and observed actions. We used logistic regression with single-trial accuracy and response as the dependent variable to assess differences in intention discrimination performance and response bias and linear regression, with the fraction of correct predictions of each video across cross-validation repetitions as the dependent variable, to assess differences in encoding and readout model performance. The chance-level null hypothesis distribution for encoding and readout model performance was created by fitting the model after randomly permuting across trials the observer’s choice labels.

To determine the fixed and random effects to include in the model, we applied a model selection procedure that started from the model with the most complex structure to arrive at a model that included only the significant predictors. We first selected the random effects structure of the model by keeping the full fixed effects structure and using the Bayesian Information Criterion (BIC) (31). The BIC rewards model fit and penalizes model complexity. We then retained the optimal random effects structure and selected the best fixed effects structure by conducting likelihood ratio tests between models differing only by the presence or absence of one predictor (32). Model selection results are reported in SI Appendix, Table S1. CIs for model coefficients and statistical comparisons for the effects are reported in SI Appendix, Table S2. We performed model fitting using the R package lme4 (33). We performed comparisons across levels of the selected models using the glht command from the R package multcomp (34). The multcomp package estimates the value and SE of each effect, from which a z value (to calculate two-sided P values) is computed. The results are reported in SI Appendix, Table S2 along with CIs for the estimates of the regression coefficients and for the SD of random effects of the selected models, computed using the bootstrap option in the R function confint.

Because there was no significant response bias, we quantified individual performance on the intention discrimination task as the fraction of correct intention choices. For each participant, we assessed the significance of discrimination performance against chance using a binomial test separately for TD observed actions and ASD observed actions (SI Appendix, Table S5).

To quantify single-trial kinematics, the 15 kinematic variables of interest were averaged, for each grasping action, over 10 epochs (t), each spanning 10% of the normalized movement time (0 to 10, 10 to 20%, etc., of the movement duration from reach onset to reach offset). Next, for each trial and epoch, we created a 15-dimensional, single-trial time-dependent kinematic vector, $\overrightarrow{K}\left(t\right)$, whose entries, for each trial, were the 15 kinematic variables averaged over that time epoch. We used this kinematic vector for all logistic regressions (see below). We verified that increasing the number of kinematic features by considering the x, y, and z component of all the markers used to compute the kinematic variables of interest (3 × 6 retroreflective markers) did not improve the performance of the kinematic encoding model. This observation held true for both TD and ASD actions and even when using a finer time windowing (25 or 50 movement epochs rather 10 movement epochs, as in the analyses reported in the main text; P < 0.02 for all movement epochs number). These control analyses suggest that our kinematic encoding model provided adequate spatial and temporal resolution to capture intention-related variations in TD and ASD kinematics.

To determine the dependence of intention (kinematic encoding model) and intention choice (kinematic readout model) on kinematics over time, we used a logistic regression to estimate the single-trial cumulative probability $y(t)$ (i.e., the cumulative evidence) in favor of the intention to place as function of the time-dependent kinematic vector in that trial up to time t. Specifically, we modeled $y(t)$ as a sigmoid transformation of the sum of two terms: a linear transformation of the kinematic vector $\overrightarrow{K}\left(t\right)$, describing the evidence provided by the single-trial kinematic vector at the current time epoch (t), and a drift term, describing the contribution of the cumulated evidence $y\left(t-1\right)$ provided by the kinematic vectors up to the previous time epoch $(t-1)$. More precisely, the equation of the logistic model was as follows:where σ is the sigmoid function, $\overrightarrow{\beta }$ is the vector containing the values of the regression coefficients of each kinematic variable, $w$ is a coefficient weighting the accumulation of information over time, and β₀ is a kinematic-independent bias term. The value of $y(t)$ computed at reach offset provides the final probability of intention (kinematic encoding model) or intention choice (kinematic readout model) associated with the kinematics of the whole trial. In this model, a single regression coefficient is assigned to each variable, meaning that the contribution of each kinematic variable is weighted equally across all time epochs. More complex models with different regression weights assigned to each variable at different epochs as in ref. 6 yielded no better performance (P > 0.09 for all observer groups and observed actions), confirming that, despite its simplicity, our model fit well both intention encoding and readout.

Training and evaluation were performed in a similar manner for encoding and readout models. Each model was trained on a set of 50 trials. We z scored the single-trial kinematic vectors within each model to avoid penalizing predictors with larger value ranges. We trained the models by minimizing the negative binomial log likelihood with L² penalty via stochastic gradient descent with adaptive moment estimation (Adam) (35). The training was marginally improved by a data augmentation scheme based on small random deformations over the time dimension (SI Appendix, Data Augmentation Procedure for Training the Logistic Regressions has full details). The parameter λ, which controls the strength of the L² regularization term, was set to 0.05 for all models. A cross-validation approach for tuning this hyperparameter was also tested and yielded similar results. The kinematic encoding and readout models were implemented using Python/PyTorch (36).

The kinematic encoding model expressed the probability that a grasping action was performed with the intent to pour as a function of the kinematic vector of that action. We trained separate encoding models for TD and ASD actions. We used the encoding model to quantify the intention information encoded in movement kinematics (Fig. 2C) and to identify the kinematic variables that carry intention information in TD and ASD movement kinematics (Fig. 2D).

The kinematic readout model expressed the probability of intention choice in each trial as a function of the kinematic vector measured in that trial. We trained the readout model separately for each observer in each session.

We evaluated the performance of the encoding and readout models by repeated fivefold cross-validation (50 random splits) (37). We computed the most likely value of Y for each trial by taking the argmax over Y of $P\left(Y\text{|}\overrightarrow{\text{K}}\right)$ in the equation of the logistic model. Model performance was computed as the fraction of correct trials averaged over folds and random splits.

We used a binomial test to establish whether the number of readers and the fraction of good readers were statistically significant in each group. To assess the significance of differences between observer groups and observed actions in the proportion of readers and in the proportion of good readers, we used a nonparametric permutation test.

For all kinematic encoding and readout models, we obtained estimates and 95% CIs for the regression coefficients from a bootstrap distribution obtained by fitting the models to data randomly sampled with replacement from the original training data.

We assessed the informativeness of individual variables (Fig. 2D) by testing whether the corresponding encoding coefficients were significantly different from zero. We retained as informative those variables whose encoding coefficients (absolute value) were greater than the 95th percentile of the null hypothesis values obtained when training the kinematic encoding models with permuted trial labels. A similar procedure was used to determine the number of observers who read each variable during action observation (shown in Fig. 5B).

In Fig. 3C, we used the kinematic readout model to estimate the intention discrimination performance of individual participants. Using the equation of the logistic model, the intention choice predicted as most likely by the readout model was computed for each trial and compared with the actual intention choice. The individual intention discrimination performance was obtained by averaging the probability of correct choice across all trials for a given participant. For the analysis in Fig. 3D, we computed the confidence of the single-trial model prediction as the deviation of the estimated probability of to pour from chance (0.5).

We computed two indices of intersection between encoding and readout: overlap and alignment. The index quantifying the overlap in kinematic space between encoding and readout was computed by taking the elementwise absolute value of ${\beta }_{\mathit{enc}}$ and ${\beta }_{\mathit{read}}$ and computing the normalized scalar product of the resulting vectors:

The overlap index measures the amount of weight common to the two vectors, regardless of the sign of the coefficients.

The index quantifying the alignment of encoding and readout in kinematic space was computed as the normalized scalar product between the encoding and readout vectors:

Note that the absolute value of the alignment index is bounded from above by the value of overlap. Alignment values close to zero can be found either with low overlap values (when the variables with nonzero weights differ between encoding and readout) or with high overlap values (when the two models select the same variables with nonzero weights but the signs of the weights are inconsistent).

In Fig. 6 and in SI Appendix, Table S4, the statistics of the overlap index and the alignment index were computed over the set of observers who were classified as readers when observing either TD or ASD actions.

To assess the significance of the overlap and alignment indices, we compared them with a null hypothesis distribution obtained by recomputing their values after random permutation (n = 10⁵ random permutations) of the entries of the encoding vectors.

SI Appendix, Tables S1–S3 report details of logistic mixed effects models statistical tests, linear mixed effects models statistical tests, and nonparametric permutation tests. Reported P values are two sided and Holm–Bonferroni corrected. In the figures, *P < 0.05, **P < 0.01, ***P < 0.001, and ns indicates P > 0.05. Following standard notation, asterisks above bars indicate significance of difference from chance of an individual quantity, and asterisks above brackets indicate significance of difference between two quantities.

The significance of correlation values was assessed using the scipy.stats Python module, with two-sided parametric Student statistics for Pearson correlation and two-sided permutation distribution for Spearman correlation (38). We used the SciPy package (39). Significance values are shown in Fig. 3D.