Our research
Our research focuses on the cognitive processes and brain mechanisms underlying our perception of humans and human actions. How do we see and understand the actions of other people? What information is available from behaving humans that can be used to guide our social interactions and future responses? How is this information organised in our mind and brain?
We use a number of different scientific techniques including behavioural testing, computational modelling and functional neuroimaging. Please look below at some of our current and past projects to find out more about our work.
Cognitive and neural representation of human actions
The organising principal underlying our mental representations is that features of our external world are represented in different cognitive workspaces or ‘conceptual spaces’. These spaces capture the similarities and differences between items in a domain enabling classification, naming, and guide our responses to the information. In this ongoing research programme we aim to understand the conceptual space underlying our representation of human actions, referred to as ACTION SPACE. This programme involves modelling the structure of how we represent human actions in the mind, testing how this cognitive framework determines how we see and understand other peoples' actions, and measuring how brain structure and function underpins action space.
mopping.mp4catchingsmall.mp4celebratingalone.mp4dancingalone.mp4Example movies of actions
drinking.mp4accusing.mp4Our first steps were to collate a corpus of the most diverse range of human actions possible. We used a 32-channel motion capture system from NOITOM to capture 240 different actions, each performed by male and female actors. This motion-capture data was then used to animate an androgynous human avatar to convey the actions (posture and movement information from the body) in order to eliminate other potential confounding social information (e.g. from facial expressions, body shape, identity, clothing, age, gender race etc.)
All our actions can be obtained from the OSF: https://osf.io/4vew8/
The location of 240 actions in 3 dimensions of the 4D action space. X-axis formidableness, Y-axis friendliness, colour abduction, intentionality not represented here
These actions were rated on 23 different characteristics and Exploratory Factor Analysis (EFA) shows that our perceptual representation of human actions is best modelled by a 4-dimensional action space. The dimensions of 4D action space determines the fundamental action qualities on which we base our judgments of human actions. These are action formidableness, friendliness, intentionality and action abduction.
The location of an action within the 4D space determines the degree to which the action conveys the 4 different action qualities. Morphing between actions allows us to generate novel actions at any location within 4D action space. The figure on the left illustrates the principle of morphing between 4 actions within a 2D space to generate a continuum of actions that vary precisely in terms of their formidableness, but don't vary in friendliness. Here, the source actions are either formidable (bouncing basketball, stamping) or feeble (breaking bread, tearing object); or friendly (bouncing basketball, breaking bread) or unfriendly (tearing object, stamping). This technique allows us to parametrically vary actions on specific social qualities, and can then be used to measure how we see, and process this information in our mind and brain.
For example, we used stimuli from this method to examine how Autistic traits impact our ability to discriminate different types of social information from human actions:
We recently reconciled this model with the other competing models of action space and show that actions are represented within a refined 5D model of Action Space. This shows we evaluate actions on: Friendliness, Formidableness, Abduction, Locomotion and Environmental Interaction:
Our ongoing research projects in this area include:
1. Determining how we represent actions during natural viewing of human behavior
2. Measuring how action space interacts with dimensional spaces for other social human signals, e.g. information from the body (Hu et al. 2018) or the face (Sutherland et al. 2013).
3. Understanding how a 4D action space model can account for natural perception of photorealistic human actions performed in different contexts.
4. Understanding how action space is represented in the brain, and how action space varies between different individuals with different psychological characteristics.
Facial expression patterns across large groups
Humans naturally produce different facial expressions in response to their environment. During group events (concerts, cinema, talks, classes etc.) individuals experience various emotional states, and these emotions are reflected in specific changes in facial expressions. This is a rich source of information about the audience. Recently we and others have used non-invasive recording of facial expressions of large audiences to understand their emotion experiences during performances (e.g. Kayser et al. 2021). Our aim is to replace unreliable self reports of audience experiences and difficult, time-consuming and often unwelcome invasive physiological measures of audience experiences with methods that are simple, reliable, non-invasive, and massively scalable to provide information about audience engagement.
We record and analyse 8 different facial expressions in every individual present every 20 milliseconds during group events. The Figure on the left illustrates the time course of the expressions from one person during a ~30 minute performance.
Whilst expressions produced by each individual can tell us about the emotions that they are experiencing at any moment, the real power of this information is how the patterns of expressions change across the audience as a whole.
By analysing the synchrony of the different expressions between all members of an audience (see Figure on the left) we can determine when everyone is experiencing the same emotions. This information is distinct from how much of an emotion the audience is expressing, and is indicative of the commonality of the emotional experience.
The primary source of expression synchrony will be the shared experience of the audience, as they are all at the same event. Expression synchrony is dependent upon the degree to which the audience attends and follows the narrative and emotional content of a performance. In short the degree to which the audience is ‘engaged’ by the performance (Oakes et al. 2024).
Our ongoing research projects in this area include:
1. Developing methods to speed up the analysis of facial expression synchrony to allow us to measure audience engagement in 'real-time' to provide instantaneous feedback to users.
2. Using facial expression synchrony to measure the effectiveness of different interventions in difficult to reach user groups.
3. Using audience expression synchrony information to test the attractiveness of consumer products and the effectiveness of advertisements.
4. Investigating how the personal traits and characteristics of audience members influence their emotional synchrony within groups.
Action adaptation
Prolonged exposure to a visual stimulus typically results in a visual “aftereffect” where subsequent perception is biases for a short period afterwards. A good example of this phenomenon is the “waterfall illusion”; here, after watching the downwards movement of a waterfall for about 1 minute, will result in an illusionary upwards movement of the nearby stationary rock upon fixation. Visual adaptation to simple visual stimuli and resulting aftereffects have been well studied using psychophysics and neuroimaging techniques in humans as well as single unit recording in non-human primates. The link between psychophysical phenomena and underlying neurophysiological mechanisms is relatively well understood for these simple stimuli. As such, psychophysical adaptation paradigms have proved very powerful in accessing the brain mechanisms underlying the perception of the adapted stimuli.
Recently our lab has demonstrated that we adapt to the actions of other individuals (Barraclough et al. 2009, 2011, 2012, 2016, Keefe et al 2016). This adaptation is occurring at a high-level in the visual system where the actions themselves are coded. The results we find using psychophysical action adaptation paradigms in human observers show striking parallels with results obtained using single unit recording techniques in the temporal lobe of non-human primates. These results demonstrate that our psychophysical techniques can be used to access action coding mechanisms in the human brain allowing us to study them whilst potentially bypassing expensive neuroimaging techniques.
Here Bruce Keefe is viewing a test movie presented at 100Hz on the HIVE wall through LCD glasses. These LCD glasses ensure that each eye receives alternate and different images - the movie shown to each eye is therefore at 50Hz. This technique allows vivid stereoscopic presentation of photo-realistic movies such that the viewer appears to be within the scene with the actor.
Experiments are controlled by MATLAB and we have developed a 3D player (mex3Dplayer) that allows MATLAB running the psychophysics toolbox to call and display stereo .mp4 files containing experimental footage.
Adaptation to grasping and placing actions shows that these hand actions are coded together, rely on the presence of object in the hand, and neural mechanisms are broadly sensitive to the view from which hand actions are seen.
Adaptation to whole body actions indicate that we have separate populations of neurons in the brain that code forward walking and backward walking. Neural mechanisms that respond to walking appear to be view-independent and insensitive to identity.
Our perception of another agent at any one point in time is not just a representation of their behaviour, but also a product of our immediate perceptual history. Judgments we make about other individuals are therefore subject to short term biases in visual processing. During experiments, participants view full-scale, 3-dimensional (3D), true high-definition (HD), high frame rate (50Hz) movies of actors via the HIVE 5.3m x 2.4m projection system. This system allows us to perform highly controlled psychological testing of participants whilst they experience an immersive simulated real-life environment. This form of experimentation allows us to access perceptual mechanisms operating naturally that cannot be achieved using traditional psychophysical or any neuroimaging techniques.
Our ongoing research projects in this area include:
1. We are using adaptation methods to understand how our integrated perception of other humans is formed from the contributions from the separate cognitive systems for actions, faces and bodies.
2. Investigating how selective attention within our social environment influences the way adaptation can enhance perception of human behaviour.
3. Using adaptation as a tool to understand the fundamentals of the cognitive processing of human actions.
Past projects
During performances audience facial expression synchrony predicts their engagement. Expression synchrony increases with spatial proximity within the auditorium. Younger, female, and more empathetic individuals show greater expression synchrony. Videoing faces of large groups allows real-time non-invasive measures of engagement.
We show that the ability to discriminate the friendliness or formidableness of actors varies greatly between individuals - by over 1000%. Importantly, we found no evidence that autistic traits influenced perceptual discrimination of these action qualities. Our results confirm that sensory enhancements with autistic traits are limited to lower level stimuli, and suggest that the perceptual processing of these complex social signals are not affected by autistic traits.
We show that the cognitive space underlying our representation of actions is 4 dimensional, where we evaluate actions on the the fundamental qualities of Friendliness, Formidableness, Intentionality and Abduction. The first 2 qualities show a similarity to the principal dimensions underlying our representation of face traits and emotions, whilst the last 2 qualities appear unique to actions.
We show that knowledge of whether an individual's emotion is genuine modulates face expression aftereffects. We argue that this reflects that the neural substrate responsible for the perception of facial expressions of emotion incorporates the presumed felt emotion underpinning the expression.
We explored whether automated face analysis could detect facial expressions of emotion in a group of people in an ecologically valid listening context. We found that pieces of music that expressed sadness resulted in more facial expressions of sadness, whereas pieces that expressed happiness resulted in more facial expressions of happiness. Facial expressions predicted subjectively felt pleasantness and activation. Our results show that non-invasive measurements of audience facial expressions in a naturalistic concert setting are indicative of emotions expressed by the music, and the subjective experiences of the audience members themselves.
Attention to face or voice identity, while ignoring stimulus location, solely increases the gain of respectively face- or voice-sensitive cortex. In contrast, attention to voice while ignoring sound loudness increased neural selectivity. The combined results show that how attention affects adaptation depends on the level of feature-based competition, reconciling prior conflicting observations.
During scanning participants were either asked to judge the intent of actions, or judge the outcome of actions. We found that during mentalizing there was increased functional connectivity between the dmPFC and the mirror system in typically developing participants, but no increase in functional connectivity between these regions in ASC participants. Connectivity between the dmPFC and IFG was negatively correlated with autistic traits. Performance on the mentalizing task was also negatively correlated with autistic traits. Reduced connectivity between these systems may explain some of the behavioural difficulties experienced by adults with ASC.
We investigated Mirror System (MS) activity in adults with and without ASD when inferring others’ intentions using TMS-induced motor evoked potentials (MEPs) and mu suppression measured by EEG. Our data suggest ASD is associated with reduced right MS activity when mentalizing, TMS-induced MEPs and mu suppression measure different aspects of MS functioning and the MS is directly involved in inferring intentions.
We examined the mechanisms in central vision and far periphery (40 eccentricity) involved in the recognition of the actions of a life-size actor (target) and their sensitivity to the presence of a crowd surrounding the target. Our results suggest that the presence of a crowd carrying out actions similar to that of the target affects its recognition. We outline how these results can be understood in terms of high-level crowding effects that operate on action-sensitive perceptual channels.
We tested the abilities of 34 individuals (17 with ASD) to derive intentions from others’ actions during both explicit and implicit tasks and tracked their eye-movements. Adults with ASD displayed explicit but not implicit mentalizing deficits. Adults with ASD displayed typical fixation patterns during both implicit and explicit tasks. These results illustrate an explicit mentalizing deficit in adults with ASD, which cannot be attributed to differences in fixation patterns.
We show a novel crossmodal adaptation effect, where adaptation to a visual stimulus enhances subsequent auditory perception. Prior adaptation to visual, auditory, or audiovisual hand actions enhanced discrimination between 2 subsequently presented hand action sounds. Discrimination was most enhanced when the visual action "matched" the auditory action. These results together indicate that adaptation is a ubiquitous mechanism for optimizing perceptual processing of multisensory stimuli.
We used virtual reality (life-size photorealistic actors presented in stereoscopic three dimensions) to see how visual adaptation influences the perception of individuals in naturally unfolding social scenes at increasingly higher levels of action understanding. Aftereffects increased with the duration of the first actor’s behaviour, declined exponentially over time, and were independent of view direction. Adaptation is not acting at an action-independent abstract level but rather at an action-dependent level. Adaptation influences more complex inferences about belief states of individuals, this is likely to be a result of adaptation at an earlier action recognition stage rather than adaptation operating at a higher, more abstract level in mentalizing or simulation systems.