Dance meets engineering in AR project

March 21, 2024 – DENTON – Imagine watching dance movement without seeing dancers. Visualizing the motion of arms and legs without seeing arms and legs.

Now imagine interacting with those dance movements, not on a stage or studio but in your home or office.

"You can see the world around you, but onto the world around you is also the dance," said Adesola Akinleye, assistant professor in Texas Woman's University's Division of Dance.

Through a grant from the National Endowment for the Arts, Akinleye is developing choreography and performance pieces for Space+Dance+Digital, or S+D2, an augmented reality system being created by Talis Reks at the Massachusetts Institute of Technology's Immersion Lab. The project began when Akinleye was doing an artistic residency at MIT in 2020-22.

"There's a lot of interaction," Akinleye said. "We talked about imagination. I'm limited by what I can imagine for the technology, and they're limited by what they can imagine for the movement and dance. So the advantage is for us to gently hold each other's hands in the unimaginable. We've been trying to trust each other to be in spaces that we couldn't have imagined."

You're probably familiar with virtual reality. Put on a headset connected to a computer or gaming console, your world goes away and you're thrust into a virtual world in which you can interact with phantom environments and creatures.

Augmented reality is similar but with a critical difference: you can still see your environment through the headset, but a computer overlays digital content. An AR example is Pokémon Go, in which players search real environments for animated characters that appear on a phone or tablet.

In this case, the headset projects the dance moves represented by swirling lines and glowing balls that circle the spectator. Just like with VR programs, seeing a video of S+D2 simply does not do it justice. To get the full impact, you have to see through the headset.

It's a new frontier for modern dance, which consistently challenges and breaks social barriers but is now taking on spatial barriers, moving dance away from site-specific locations and into the everyday. That, in turn, opens up the potential for dance as a gathering place for people from around the world.

"What I'm interested in is if the choreography itself could start to become a place of meeting," Akinleye said. "When someone's viewing the choreography through the AR, because AR is multidimensional they can kind of walk into it and look at it and have it moving around them. It starts to become almost like a sculptural thing that they can enter and be a part of. Using hand tracking, they can also see their own movements in response to the choreography. The choreography starts to become the meeting place. I'm interested in the idea of how we co-create something together. The choreography can become this kind of common space that people can connect with each other through the logic of the movement of the choreography, the way that physical space can connect people together through the logic of the design of the architecture or the buildings around."

The choreography and movements are digitized using motion capture, in which a performer is recorded while wearing reflective markers. Those markers allow computers to create animated characters in movies or video games, or in this case to capture a dancer's movement.

"It's more around catching the result of movement rather than catching movement precisely to analyze it," Akinleye said. "You see the movement of a dancer, you don't see the physical body of the dancer. You see the result of their movement, which is another nice thing about people being able to dance together. The sort of social constructs of what a body means are taken away a little bit, because it's only the result of the movement. People's movement allows you to get a sense of them, and that's where the artistry of the choreography that I've been doing is."

Akinleye's ambitions, however, go much deeper and higher than games or videos.

"I've been looking at choreography and architecture as creating kind of spatial syntax," Akinleye said. "The dance produces a new kind of meaning onto the space that you're in, because it exists in that space, you're seeing it in that space. And just like with site-specific dance, it gives different perspectives of that space. But because it can be networked, someone else can be in the same dance as you, but they would have different logics according to where their space is, too, so they can move together.

"Art is important as a way of knowing you're alive and experiencing and also understanding the world. There's this thing that I would call artistic intelligence that's really important that we develop. It feels important that we have artwork in digital space and artwork that leads us to the kinds of futures that we want, because digital space is going to be a part of the futures that we're the near future, at least."

S+D2 also requires thinking about dance in more dimensions than those available on stage.

"When you're choreographing for movement capture, it's being seen up and down," Akinleye said. "I can move the choreography so that it becomes below you. So there's a different view of what the movement looks like if you're looking down on it, or if you're looking up on it, or if you're looking directly in front of you. A lot of the work has been around using the full potential of having that ability to play with movement so that it has size and texture and also angle. I've never done that before."

There is no timetable for S+D2's release, but Akinleye is hoping for some time in 2025.

"My next step is to look at some performance spaces or festivals," Akinleye said. "Maybe one at MIT."