Artificial Neural Networks
I have been working on developing a digital performing agent using Artificial Neural Networks (ANN) as a means of allowing the digital agent to learn a shared vocabulary of movement from dancer, Steph Hutchison. The goal was to have an agent perfumer that could learn to dance with Steph and even co-create the live performance in a interactive manner. The main type of ANN I’ve used is a Self-Organising Map (SOM), an unsupervised form of ANN that is often used successfully to cluster and classify datasets. SOM are particularly good at handling high-dimensional data such as motion-capture data where each frame of data can have over 100 components. In this research the dataset was reduced to 79 input vectors equating to the position of the dancer and the rotation of 19 joints in the skeleton.
I’ve also been looking at how the relationship between the dancer and agent can be used to support the agent’s learning and performance by considering them as part of a system rather than developing the agent’s capability as a self-contained component.The framework of distributed cognition has been valuable in this regard as a way of conceptualising an inter-dependent relationship between the agent and dancer that can support the agent in the manner in which a more experienced dancer would mentor one less experienced.
I’ve been using various motion capture systems to record the dancer’s movement for both training and testing the agent. The system is also used in the live performance to both animate the dancer’s avatars and to allow the agent to “see” what the dancer is doing. The motion capture system essentially becomes the sensory apparatus of the agent. I’ve been using the motion capture systems at Motion.lab and at the Centre for Intelligent Systems Research (CISR), both research centres at Deakin University in Melbourne, Australia. The final performance Emergence, used the Motion Analysis motion capture system at Motion.lab.
Once the agent has gone through the learning process, the agent is able to use the learnt movements to form an internal representation of what the dancer is performing, allowing it to follow the dancer, generate movement sequences based on the dancer’s current movement and dance independently of the dancer using the shared movement vocabulary.
The choreographic structures used in the creation of the performances were quite simple and ones that any two dancers might use to get to know each other and explore movement creation together. The first (Grevillea Crystalis Incarnadine) was to have the agent simply follow the movements of the dancer as best it could. The dancer provided the agent with movement sequences that it learnt and then she performed and improvised around these movements. The agent responded with the closest movements it had learnt to what the dancer was performing. This simple following behaviour was a result of the agent’s neurons being stimulated by the dancer’s incoming movement data.
Self-Organising Synaptic Map
The next choreographic structure (Recognition) saw the agent and the dancer performing movement independently of each other. The dancer’s avatar was animated by the live motion capture data. The agent’s avatar was animated by the sequential firing of its neurons, which contain learnt information equating to movement postures. In the learning process, I used a second layer to go with the Self-Organisin Map. This layer contained a synaptic map generated by the connections made when one neurone fired followed by another.This was based on the concept of synaptic plasticity, whereby the synapses connecting neurons can strengthen according to firing frequency. The resultant structure I have called a Self-Organising Synaptic Map (SOSM).
For the next choreographic structure (Instrumental) the agent was able to see what the dancer was doing and use her movements as the starting point for its own movements. The dancer could also then take cues from the agent as to what she would do next or start another movement motif from which the agent could also start a new movement phrase. This combined the capabilities from the last two structures. The agent could follow what the dancer was doing and use that knowledge as the starting point for its own movement generation. There was no explicit programming to govern this behaviour. The ability of the agent to improvise with the dancer and create movement phrase based on the dancer’s own improvisation was emergent behaviour from the learning process.
Recognising Movement Gestures
By combining the SOSM with a Hidden Markov Model (HMM) the agent is able to recognize short full body movement phrases and respond when the dancer performs these phrases. The next choreographic structure (Verbose mode) had the dancer performing eight short movement gestures. When the agent thought it recognised a particular gesture it would respond by speaking aloud commentary associated with that particular gesture. The gestures were full body movements of some complexity. The agent was able to correctly recognise the dancer’s gestures around 90% of the time. However, when it was wrong it was not really a problem as the dancer was quite amused and would often try that gesture again to see how it coped a second time. The work drew on a particular aspect of distributed cognition concerned with shared memory and recollection. The agent and dancer engaged in an impromptu conversation where they cross-cued each other to arrive at a new rendition of past, shared experience; the movement they had both learnt and the texts the agent knew. The piece was playful and conversational in quality, like two old friends conversing.
Visualising the Agent – Dancer Relationship
In the live performance, Emergence, the works usually began with the agent and dancer’s avatars seen on screen performing together. I also developed different visualisations for each choreographic structure that used the particular choreographic relationship between the agent and dancer as the driving force of the visualisations. Following behaviour saw the closeness of the agent and dancer’s movements visualised as an ever more complex crystal flower. Dancing independently, the dancer and agent’s movement data was used by a third morphing creature to animate itself, alternating between the dancer and agent’s movement data streams. The agent using the dancer’s current movement as a seed for its own movement generated a trail of sculptured crystal beams that later became a musical instrument. And the gesture recognition task produced a combined dynamic particle field using the movement of the dancer and the voice of the agent. The visualisations were projected in stereoscopic 3D, requiring the audience to wear passive stereo glasses to view the dancer-encompassing effect.