Adobe AR Art Residency

In January of 2020, I decided to focus on mixed reality art. The accumulating evidence was clear. The sort of storytelling I had always wanted to do was becoming technically feasible. While researching software solutions, I came upon Adobe Aero and quickly dove in with both feet. A month later, when asked to be part of the Adobe AR Art Residency, I was over the moon. The opportunity to explore this new medium with support of an incredible team of artists and technicians was truly amazing. This is a summary of the work I create during the twelve-week program.

Physical space is incredibly important in AR. Context plays a huge role in an artwork’s meaning, as does proximity. Experiences that (seem to) happen in your immediate physical space automatically feel personal and emotionally potent. So, I started my residency by exploring ways of using the space around participants, specifically proximity triggers and spatial relationships (through scale and distance.)

CHERRY BLOSSOM BALLERINAS

EXPERIENCE: When you get close to a blossom, it detaches from the tree and begins to dance around you in circles, indefinitely. It’s like having your own private sugar plum fairies.
MUSIC: Arrival of the Geese by Chad Crouch

IMMERSIVE MANDALAS

Questions are always a good starting point for art. I found myself wondering what it would be like to be inside of an enormous swirling mandala, so I built one. I began my experiments with simple shapes, colors, and interactive controls.

EXPERIENCE: Three buttons allow the participant to control the mandala’s spin.

MUSIC: Illumination by Kai Engel

Eventually, I started exploring storytelling possibilities by using figurative elements. This piece contains illustrations and animations that are inspired by the ocean.

MUSIC: Never Forget by Ketsa

Inspired by the biological kingdoms, this piece reminds me of 3d zoetropes where the traditional “illusion” has been replaced by a new one. 

MUSIC: Song Sparrow Serenade by Chad Crouch

TARGET PRACTICE

​​​​​​I began exploring ways to create a deep sense of relationship with the virtual elements. For example, when something shoots at your head, you can’t help but take it personally. 

EXPERIENCE: The animation is triggered by a timer. When you tap each of the rings, they begin to rotate starting with the ring you tapped. 

MUSIC: Plasticity by Blue Dot Sessions

This piece was inspired by the emotional overwhelm that many people (myself included) were feeling during the early days of the COVID-19 pandemic. 

EXPERIENCE: When you are 2m away from a letter, it senses that you are there and begins to shake and move. When you are 1m away, it rises up to meet you. When you are 1/2m away, they glom on to you, orbiting around you indefinitely. If it all starts to feel like it’s too much, you can get rid of each letter by tapping it, but you have to deal with each one individually (just like feelings.)

THROAT FIRE

EXPERIENCE: One lesson I’ve learned over the last few months is that immersive environments feel more authentic if both active (branching narrative choices) and passive interactions (optional interactive details) are present. It makes the world feel more complete and deepens the sense of immersion. 

So, I included both types of interactions in this piece. Some elements are triggered by proximity. Some are triggered by touch. Others are based on time. When you get close enough, the eyes open and follow you around (using the AIM function in Aero.) When you tap the woman, she climbs the ladder and jumps into the fire, where she is transformed into a butterfly that orbits around you indefinitely.

MUSIC: Headwaters by Chad Crouch

The final weeks of the residency were devoted to a showcase project. When something like a pandemic lands in your society’s lap, it’s hard to ignore. Personally, I found it difficult to focus on much else, so I decided to put aside previously laid plans and create a project devoted to it. If you’d like to learn more, check out my full post about it.

EXPERIENCE: Participants interact with the environment through proximity and tapping. By using active interactions (direct participant choices) and passive interactions (optional elements) I was able to add depth and authenticity to the “world”. Subtle ambient animations and sounds (like the breathing lungs) also contribute to the sense of place. 

SOUND DESIGN:  I created the composition using a combination of field recordings, piano, and my voice. The foundation for the entire piece is a field record of an actual hospital lobby (Courtesy of Tim Kahn.) The main percussive element is a heart rate monitor. I played the piano and sang the choral elements. When you tap the lungs, they play a recording of me breathing. The candles have a proximity sensor that plays individual choral tones, creating a “candle choir”. 

Being part of the Adobe AR Art Residency was an incredible experience, full of lessons, challenges, and plenty of fun.

Choosing to pursue mixed reality has led to lots of new skills.  ​​​​​​​I’ve been an interactive artist for several years – focusing on illustration, animation, and 2D game design, but 3D is completely new for me. It has presented some predictable hurdles. Thinking spatially is both challenging and exhilarating. For me, since I come from a theater background, it feels like set design on an individual scale. Looking back now, I can tell how much I’ve learned- technically, artistically, and personally.

LESSONS LEARNED

Getting comfortable with discomfort: Since this is a brand new medium, everyone is a newbie.  Fumbling is part of playing and play is how we learn. (Plus, it’s super fun!) Reframing “failing” is essential for innovation. When this residency began, I made failure an explicit goal. I kept a list of experiments by my desk and every time one failed, I would celebrate it with a little star sticker. It may seem silly, but I recommend it!

Trust: Over the years, and especially during this residency, I’ve learned the importance of trust: trusting the process and trusting myself. At many points along the way, I felt lost and unsure of what I was doing. But, I kept at it. Time + trying is the magic recipe. Returning to my roots and my traditional workflow (in my case, drawing first) always helped to put me back on course.

Discovery through doing: Experience has to be part of the augmented reality development process. It’s an experiential art form, so you have to live it to fully understand it. It’s best to test early and often. 

Letting go: Part of creating augmented reality art (and interactive art, in general) is letting go. When you give a participant authentic choices, there is a certain amount of surrender involved. Very simply, you can’t have complete control… and that can be a great thing. Allowing for surprises is part of the process. There is beauty in not knowing.