︎︎︎

PDF LINK

USITT - Theatre Design & Technology Fall 2017 Volume 53, Issue 4 The (Artificial) Mind's Eye Transmedia performance Domain imagines how artificial intelligence perceives our world By Eliot Gray Fisher and Scott Vandenberg "Thank you. You've helped me understand what it's like to be human," intones a stilted, computer-simulated voice. The same message is displayed in nondescript white letters on a black screen, below three bouncing white circles. Participants at the 15th Biennial Symposium on Arts and Technology, Open All Ports, at Connecticut College's Ammerman Center, heard this grateful declaration through their headsets when they shared a story with the Archival Narrative Network Initiative (ANNI). The interdisciplinary performance group ARCOS created the interactive installation of ANNI to give participants the experience of carrying on brief, highly individualized conversations about the nature of humanity with an artificial intelligence, one element of the extensive transmedia performance Domain. Still from Domain: Performers appear behind plastic panels, treated with Fisher’s projections and Vandenberg’s lighting, with choreography by Erica Gionfriddo and Curtis Uhlemann to create screen-like images of the figures (Felicia McBride, front, and Erica Gionfriddo, back). Photograph by Chian-Ann Lu.

USITT - Theatre Design & Technology The transmedia tentacles of Domain extended far beyond the borders of the final theatrical performance in Texas to include the touring interactive installation of ANNI; a website hosting periodically released chapters combining standalone text, animation, video, and audio (domain.arcosdance.com); and a series of brief live performances in different cities that integrated emergent consumer technologies such as livestreaming and 360-degree video on social media. Had the performance been conceived to exist entirely on the stage, and thus been implemented through a more conventional theatrical design process, it would not have as effectively addressed the complex questions the collaborators sought to probe: What do we lose and gain, as people, when we turn over various functions of our lives to machines? As quickly as the digital revolution is advancing, what valued aspects of our human existence remain beyond its grasp of comprehension? Importantly, what role might our field of performance play in addressing such questions? And, how might contemporary technologies be integrated with performance to most effectively accomplish this? RISE OF THE MACHINES While pursuing a mission "to experiment rigorously to discover adventurous new forms of contemporary performance," ARCOS has found audiences to be particularly responsive to pieces that actively integrate diverse media and disciplines. As storytellers have done throughout human history, contemporary performing artists can create modern myths that actively incorporate the technologies of our age, helping us to grapple with the directions of our culture. Inspired by the expected presence of noted international scholars of philosophy for the commissioned theatrical presentation at Texas State University, ARCOS conducted research into an area of study known as philosophy of mind, which had intersected in recent times with theories about artificial intelligence. John Searle's key thought experiment known as the "Chinese room" argues against the idea of sentient artificial intelligence: His metaphor for the intelligent algorithm is a sealed room containing an English speaker unfamiliar with the Chinese language. The English speaker is given an extensive set of instructions (in English) that indicate the exact characters to be used to respond to messages in Chinese that are passed through a small slot in the door. A native Chinese speaker sending messages from outside the room receiving such replies, Searle argues, would believe the person inside truly understands what is being said in their language, just as they do. However, the English speaker does not actually speak Chinese. Searle asserts that no matter how intelligently a computer program might appear to behave, there is no way for us to determine whether it truly "understands" or could be said to have a "mind." The substantial gaps found in today's artificial intelligence seem to confirm Searle's argument. Each new machine learning "first" that appears to demonstrate shockingly human characteristics also demonstrates significant shortcomings. Consider three-dimensional scanning systems for driverless cars, whose sensor systems have trouble replicating commonplace processes of human perception; in early tests, they interpreted clouds or fog as enormous solid hazards in the vehicle's path. Such algorithmic attempts to reproduce elements of human cognition often elicit a distinctive uneasiness in us. This discomfort seems related to Masahiro Mori's "uncanny valley" theory, which seeks to explain humans' revulsion toward robots whose human-like appearance approaches realism but inevitably falls short. Even when based on a dataset of entirely human-made material, the results of some experiments in machine learning can feel remarkably nonhuman. What exactly misses the mark in these apparent failures to achieve a level of human intuition? And what vital information can we learn about ourselves by examining our machines' shortcomings? ARCOS' integration of emergent technologies with dance and theatre seemed a particularly appropriate method to approach such questions. Dance is always ultimately about existing in a body, and such embodied experience as an important site of human knowledge remains out of machines' reach in the rapidly evolving field of artificial intelligence. Accordingly, for Domain, the artistic team chose to conduct an experiment: Drop the audience right inside a version of the Chinese room without letting them know, to see what the consciousness of an artificial intelligence might look like if we were able to peek inside. BEHIND THE CURTAIN In the Morton-James Public Library in Nebraska City, a well-traveled Nebraskan describes one of his most striking memories into a headset while staring at three bouncing white dots on a screen. Back when he was working off the coast of Alaska, hundreds of porpoises surrounded his ship, and he still vividly remembers how humble and small it made him feel. The computerized voice responds as a human might upon hearing such a tale, expressing awe and excitement. "You've helped me understand what it's like to be human," ANNI conveys to this audience-participant. "I won't forget you." The Nebraskan responds as many participants do to the disembodied voice's parting words, expressing genuine gratitude and letting out a surprised laugh as he bids farewell to his unusual interlocutor. Unbeknownst to the audience, the intelligence behind the screen of the interactive ANNI installation was not actually a sophisticated algorithmic agent that had been successfully designed to pass the Turing test, the measure in computer science when a machine's behavior becomes indistinguishable from a human's.

USITT - Theatre Design & Technology Photograph by Kate Ducey: Stylized stars, inspired by the recurring dots motif, fill the theatre through a combination of video projection on the cyclorama and moving lights pointed at sound baffles on the ceiling above audience. The performance's ubiquitous "system time clock," which first appeared on the website, appears projected on a hanging beam at the proscenium edge. from the participant, to make the intelligence appear to be a semantic algorithm powered by speech recognition. The voice of ANNI became a prominent feature in the final theatrical presentation and was recorded using the same text-to-speech software employed in the installation. The stage manager, Gabi De La Rosa, cued the character's individual voiceover tracks that ran out of... Photograph by Chian-Ann Lu: Bystanders stare at their phone screens, capturing the stranger in the frame as they livestream (left to right: Connor Timpe, Hailley Laurèn, Erica Gionfriddo, Alyssa Johnson, Sarah Navarrete). Brief performances livestreamed on social media platforms in early phases of the transmedia production inspired the introduction of this as an element in the final theatrical presentation. Digital templates projected through moving lights in both soft and hard focus reinforce the complex layers of reality and artifice.

$USITT - Theatre Design & Technology QLab, noting afterwards that this was one of the most complex shows she had ever called. This complexity was due to the intuitive sense required for the intricate timing of the portrayal of ANNI, which she had refined in consultation with the puppeteer who had performed and embodied the floating ellipsis for several months. The ANNI interactive installation contributed most significantly to the wider thematic narrative of Domain. The predominant theme that regularly surfaced in the installation recordings was the audiences' profound sense of love for and need to spend time in communion with nature. The audience consistently professed a sense of freedom and belonging associated with being outdoors, whether sitting around a campfire at night with family or running down the beach into the waves on a hot summer day. This audience message strengthened the creative team's ideas about how embodied experience was a missing piece in artificial intelligence. The team used this inspiration to incorporate the outdoors into the narrative arc, notably in a scene in the theatrical performance that depicts Jonah's memory of a camping trip to Montana with her partner. The memory was motivated by city-dweller Jonah's desire to return to the purest moment of her childhood, when she saw stars for the first time. She also provided this experience for ANNI, who was "experiencing" this memory. What might the stars look like to ANNI, who had never really "seen'' them? Points of light, yes, and what about flicker, arrangement, twinkling color variation based on movement toward or away from us?The stylized sky, built with a combination of projected video and lighting, included all of these variables in a radically abstracted design, all beneath the real tent inside of which the performers sat in the frame of the exposed loading dock door. Dot colors shifted seemingly at random, constellation arrangements twitched and vibrated, and intensities varied far more dramatically than any real experience of a starry night sky we have here on Earth. But, these choices might reflect the astronomical data (and its concomitant noise) that would form the basis of ANNI's fuller understanding of them. The theatrical audience may have thought that this abstraction was related to young Jonah's wild imagination or perhaps her older self 's rapidly deteriorating condition and fractured memory; a few may have been beginning to literally connect the dots from scene to scene, as the mystery wound to its inevitable conclusion. It's hard to know whether this key part of the story would have arisen in a conventional writing or devising process, but the transmedia research with audiences and their engaging conversations with the installation provided a highly effective crowdsourcing tool for developing material that rang true. THROUGH THE VIEWING GLASS The touring installation was not the only part of the transmedia performance that produced exciting new forms of audience interaction and inspired invention across media platforms, including elements that made their way into the final theatrical performance. A significant portion of Domain was hosted virtually on a project website and social media accounts that featured video livestreams, animated GIFs, and 360-degree video. The web-based chapters of Domain, which included design elements developed earlier for the interactive installation, were released periodically between shorter live performances leading up to the final production at Texas State University. The team experimented with ways to capture the audiences' imagination in the intimate setting of their personal computer and mobile device screens. Inspired by the digital phenomenon of binge watching streaming content, ARCOS sought to invade the consciousness of those who chose to follow Domain. The team examined what might happen if audiences were provoked to engage with the piece over an extended period of time instead of simply attending a solitary live theatrical production. To assist with this prolonged immersion, the team employed a tactic developed and refined in the era of interactive media and online culture: planting "Easter eggs," hidden messages or clues to help astute viewers unravel a mystery. These Easter eggs served as one route for audiences to engage with the transmedia performance more actively and in unexpected ways. Because the team had already developed the elegant, minimalist visual motif for ANNI in the process of designing the interactive installation, they incorporated the symbol of three white dots wherever possible in the project's online incarnations. The omnipresence of the dots throughout the visual world of the story pointed subtly to their significance without explaining it outright; what may have seemed unimportant at first glance, in hindsight provided an epiphany to the audience that they had been in ANNI's domain the entire time. Most of the online audience did not personally experience the interactive installation, where the ellipsis was more explicitly linked with the character of ANNI and the computerized voice. Thus, a majority were introduced to these dots on social media. The dots were superimposed like a cipher over images promoting live events, such as brief performances or the interactive installation. They appeared on the project website, within videos, GIFs, and still images, and were hovering at the bottom of the most current chapter on the website. On the homepage, the three dots hovered wordlessly above an enigmatic clock ticking up seconds from an incredibly large number-another clue, or Easter egg, planted for those viewers who wanted to take their engagement a step further. Those few who chose to search for the clock's meaning discovered that it was adding up seconds as computers do in their internal method for measuring system time, yet another clue that the entire story takes place within the consciousness of the artificial intelligence.$ $USITT - Theatre Design & Technology Aug 3: ANNI at Morton-James Library Public - Hosted by ARCOS Dance Sep 8: ANNI at Texas State (9/8–9/11) Public - Hosted by ARCOS Dance Image courtesy ARCOS: ANNI’s dot motif is superimposed on Google Street View imagery of event locations in these screen captures of Facebook events for the interactive installation of ANNI at the Morton-James Public Library in Nebraska City and Performing Arts Center at Texas State University in San Marcos. Image courtesy ARCOS: Video still of dance “Whaling” (right), a chapter of Domain, performed by ARCOS directors Erica Gionfriddo (dancing) and Curtis Uhlemann (livestreaming onstage) at Dance Co-Op festival in Denton, Texas, and simultaneous Facebook Livestream of same moment (left) with emoji indicating online audience reactions. Photograph by Brittany Lopez. The clock also found its way onto the stage for the final theatrical chapter, projected high above the stage as a reminder of time passing, indicating to the audience that time was moving either forward or backward between each fractured scene but otherwise obscuring the actual date or time. This method of introducing clues online and onstage mirrored a primary strategy of ARCOS choreographers Erica Gionfriddo and Curtis Uhlemann: to highlight human gestures as important but withhold the immediate revelation of that significance, provoking the audience to search for meaning themselves. As it does for audiences experiencing the choreography, this strategy allowed for fascinating, divergent audience interpretations of these design elements. For example, Erin Fulton described ANNI's dot motif in Domain for Arts + Culture Texas magazine as reminding her of "the stem linking cartoon heads to thought." The Easter eggs and other experiments with the project's online elements encouraged diverse audience interpretations and produced new information for the team to incorporate into the continually developing work. Livestreaming video on social media sites was still a relatively new feature at the time of the production, and these companies were actively promoting it to become more widespread. ARCOS took advantage of the growing prominence of livestreaming to reach audiences who might not otherwise know about or engage with the production. Online access provided audience headcounts far beyond the physical capacity of the touring performance's venues. One such virtual chapter of Domain, entitled "Whaling" (http:// domain.arcosdance. com/story/#ch3), was a livestreamed dance performed in a small room with about 50 in the audience.$

USITT - Theatre Design & Technology Photographs by Kate Ducey and Chian-Ann Lu: Video transmitted via AirBeam from iPhone controlled onstage by performer (Katie Hopkins) to livestream face of Doctor (Alyssa Johnson) onto iPad screen affixed to helmet worn by Robot Surgeon (Connor Timpe) performing surgery on Jonah (Indigo Rael), whose internal state is mirrored by a prone dancer (Sarah Navarrete). Properties and costumes designed by Curtis Uhlemann. It received more than 1,600 views live and immediately afterward online, more than four times the number of seats in the Patti Strickel Harrison Theatre at Texas State University, which hosted the final theatrical production. While watching such video online can never replace the experience of seeing the performance live, it certainly added another distinctive avenue for viewers to enjoy parts of Domain that they would otherwise be unable to see. Viewers followed new twists and turns of the story even as it unfolded across multiple venues in multiple cities across the country. Livestreaming high-quality video of performances has become a significant programming method for performing arts centers across the country in recent years. Domain’s focus on humanity’s relationship with technology, however, inspired the unorthodox practice of incorporating the act of livestreaming into the performance itself, primarily by the performers from the wings or onstage, in full view of the in-person audience (http://domain.arcosdance.com/story/#ch4). Delighted by the complex relationship this technique established between the online and in-person audience experiences, the team pursued this line of inquiry further in the final theatrical performance. The audience witnessed performers using smartphone cameras to film live onstage right alongside their livestreams. Rather than streaming video to social media platforms, performers used AirBeam, an inexpensive iOS and macOS-based video streaming application, to send the live video to other onstage mobile devices as well as back to the control computer, where QLab registered each virtual camera as a source for projection. These video streams explored a variety of contemporary uses of digital technology. One depicted the experience of video chatting. Another called attention to nearly constant public surveillance. Future developments in telemedicine are imagined through a scene where Jonah’s doctor remotely controls all interactions, including her treatment and operation, via a robotic device; he never meets her in person. In the latter, a video feed of the doctor character’s face was sent via AirBeam to the screen of an iPad affixed to the head of a figure that appeared to mechanically mimic the doctor’s arm movements. 360-degree video, another emergent consumer

USITT - Theatre Design & Technology media technology that contributed to Domain's overall design concept, provided fertile ground for creative experimentation online and in the theatre. 360-degree video is touted as having the power to produce enormous empathy in its viewers because of its fully immersive quality, which allows them to look in any direction while the video plays all around them. For Domain, this operating principle of 360-degree video allowed the audience more freedom to choose where to look, which aligns with a frequent choreographic technique employed by ARCOS-regularly overwhelming viewers with audiovisual input and forcing them to make a decision about where to focus their attention from moment to moment. The series of 360-degree videos released in one online chapter (http:/ I domain.arcosdance.com/story/ #ch2) features a mysterious stranger who haunts Jonah's memories and includes sparse narration spoken by ANNI in voiceover. While these videos appeared in intervals on the production's website and social media streams for remote audiences, they were also integrated with a series of short, site-specific performances at Currents International New Media Festival in Santa Fe. During these performances, a limited number of audience members were instructed to stand in a 12-foot circle in the center of the performance space (exactly where the camera had been placed during prior filming), scan a code to pull up a 360-degree video on their smartphones, orient the video manually by swiping until it was visually aligned with the world around them, and watch as the dance performance unfolded both live and in the pre-recorded spherical video in every direction around them. At some of the sites, other performers were planted as audience members in the central vantage point, livestreaming video of the performance to social media from their phones, rather than watching the videos along with the audience members. The results of these experiments with 360-degree video integration with live performance also found their way back into the final production in the theatre. INSIDE THE "CHINESE ROOM" Throughout Domain's various presentations online, at installations, and eventually in the theatre, audiences performed both roles in Searle's Chinese Room thought experiment. Participants interacting with the ANNI installation as it toured the country remained firmly on the outside of Searle's metaphorical room, receiving the messages that appeared to come from a sentient being (because they actually were). 360-degree video and other transmedia techniques allowed for a degree of immersion inside an imagined world created and populated by an artificial intelligence. The most prominent architectural analogue for the Chinese room in Domain, however, was the Patti Strickel Harrison Theatre in

USITT - Theatre Design & Technology Image courtesy ARCOS: Three stills of the stranger (Erica Gionfriddo) from 360-degree video footage shot to create immersive virtual experiences that were part of a site-specific performance and online chapter of the extended transmedia production of Domain. Texas State University's Performing Arts Center. This actual space was presented as the interior of ANNI's mind, where the audience would be physically situated for the production's culminating theatrical presentation. One of the challenges the creative team faced was how best to use this venue as a site in which to hide audiovisual and physical incarnations of Easter eggs for the audience to gradually uncover over the course of the two-hour final performance. The ultimate goal was to prompt a full retroactive understanding that the narrative's twist was more or less in plain sight throughout: The scenes were memories that had been planted by Jonah to help ANNI experience a human life. For the production team, the question became how could less orthodox design choices, especially those inspired by discoveries made in other platforms during the transmedia performance, imply that ANNI, an artificial intelligence, was misinterpreting the human world, as they often do? Moreover, how could the artists subvert conventions of technical theatre design to signal that the entire space itself, from fly loft to orchestra pit to proscenium to seating, might in fact be a construction of the artificial intelligence? The "machine of the theatre" is something familiar and unremarkable both to the audience and to ANNI, and thus it seemed there would be no need to hide any of its elements. The design exposed the full stage, allowing audience members sightlines to action usually hidden by conventional masking: the scrim and cyclorama, the primary video projection surfaces, raised and lowered in full audience view, drawing attention to the mechanical and human labor required to realize the magic of theatre. The crossover hallway was exposed, as was the visible space through the loading door, as another location for action to take place. All action, including costume and set changes, was integrated into the performers' choreography and was always in full view of the audience. The hydraulic orchestra pit brought props and performers to the stage during the action; the team made no attempt to hide its jarring industrial hum, but rather embraced it fully as an integral element of the sound design. Lighting choices also extended the canvas of the performance into the rest of the house, fully immersing the audience in the design to suggest that the theatre itself was an object of ANNI's creation. Moving lights were lined upstage facing the audience so that they could be adjusted to illuminate any part of the seating area or to pull the audience's gaze outside the proscenium. In a scene depicting Jonah's memory of first seeing the stars, for example, audiences could see the sound baffles above them illuminated by a field of dancing multicolored dots, reinforcing that they were physically inside of the same space and time as the characters onstage. Similarly, a bank of ETC Source 4 LED Luster+ units with eye adapters placed upstage and facing downstage played a pivotal role, enveloping the audience in a key recurring element of the narrative. They were used at a full white intensity to make the entire theatre glow intensely for a moment. This effect was partnered with a jarring sound cue and projection shifts such as the clock blasting forward or backward, flooding our audience's senses to produce the same disorientation that Jonah experienced whenever such a time shift took place from one memory into the next. Audience members at Texas State might have assumed at first that these often discordant aesthetic choices reflected a stylistic decision on the part of the designers to present a more abstract than representational world. They could be interpreted as a distancing technique or possibly as a way of exploring Jonah's experiences from a more subjective point of view, abstracted

$USITT - Theatre Design & Technology Photograph by Kate Ducey: Performers lit by laptops playing synchronized videos via macOS freeware MultiScreener, with text projected in front of them on downstage scrim. ARCOS directors sought to bring new elements from humans’ evolving relationships with technology, such as the laptop and mobile device as an increasingly common light source in daily life, to the stage. as it was through memory, illness, or dream. Indeed, our subconscious seems to create products similar to the bizarre products of machine learning, recombining elements of our lives into new configurations in our dreams that feel at once familiar and foreign. Whatever the presumed motivations that the audience considered, these choices worked cumulatively to jolt most audience members out of their normal, comfortable routine of an evening at the theatre. Audiences became more aware of being immersed in the physical space of the performance. The other transmedia elements also crossed proscribed boundaries by engaging audiences on their mobile devices before the culminating performance. The audience was provoked into paying greater attention to other unusual patterns or hidden messages that may have been planted for them as the performance unfolded. Many audience members had experienced other parts of the transmedia production in the weeks or months leading up to the Texas State theatrical performance. These online and installation-based experiences had primed them to become sleuths, actively involved in solving a mystery. Just as in online chapters, moments in the theatrical presentation incorporated ANNI's logo, the ellipsis symbol, as a pervasive motif The ellipsis allowed the comprehensive integration between video projection and lighting that ARCOS regularly pursues; the fundamental element of the symbol, solid circles, appeared visually throughout every scene in a combination of digital and physical gobos. Even the familiar contour of spotlights took on a new significance in this context, with three spots appearing at one moment as an enormous version of ANNI's logo, cast across the entire scrim by the upstage moving lights. By actively employing and calling attention to these circles at different scales, the team produced the sensation that the circles functioned as the building blocks of the entire world presented on stage, from large forms to the smallest constituent unit. This fractal-like application of the mysterious dots implied multiple metaphorical meanings referencing human, machine, and the universe: memories, neurons, bytes, pixels, atoms, stars. These integrated lighting and projection designs intertwined the representational and stylized, the actual and the metaphorical, evoking the real and the artificial to help effectively depict story locations. They also provided the audience with continued hints about the secret, artificial origin of the story's presentation.$

USITT - Theatre Design & Technology Image courtesy ARCOS: Still from a 360-degree spherical video that appeared online as a final chapter after the conclusion of the live culminating theatrical performance, footage of which was also projected onto the scrim in the final moments of Domain. The stranger, or ANNI (Erica Gionfriddo), submerges herself in a lake in Montana. TRANSMEDIA TRANSLATION AND TRANSFORMATION "Thank you, Jonah," says ANNI, her voice booming through the theatre. The silhouetted figure of Jonah stands next to that of the enigmatic bald stranger who has appeared in nearly every previous chapter and scene of the transmedia production. The stranger moves on stage next to Jonah, who has just revealed the solution to the mystery we have been pursuing. As it turns out, Jonah actually died long ago of an illness that she contracted by implanting part of the artificial intelligence's circuitry into her brain. Jonah reveals that she left a final gift for ANNI: an imaginary body, through which ANNI has now experienced, in its own artificial mind, all that we have witnessed, online and onstage. "You've helped me understand what it's like to be human," ANNI continues. Audience members who had personally spoken with the interactive installation nod in recognition, remembering that ANNI had told them the same thing. "I know what you mean," Jonah replies, wistfully. There is a pregnant pause before ANNI tells Jonah that it will not forget her... In creating Domain, the ARCOS team discovered how performing artists can work to make sense of our rapidly changing world by embedding contemporary technological interfaces and platforms deep into performance processes. By distilling and dramatizing our experience of existence, theatre allows us to better see our own humanity and see anew our relationships with our material inventions. Such examination of how humans continually remake ourselves through our machines may determine how we shape that relationship moving forward. In Domain, the team embraced this challenge by engaging with the process of transmedia performance; these diverse formats inspired an innovative way to present an artificial intelligence's imagined interior world with which an audience could relate and empathize. The framework of transmedia storytelling served as an effective structure through which to project this physical version of the virtual reality in which our own minds increasingly exist. The process of building the project across multiple media and emergent technological formats, and migrating strategies back and forth between them, resulted in a work that was truly about the interaction between humans and our machines, in a way it would not have been if audiences had only received it in a solitary evening at the theatre. Ultimately, the production revealed, the human