Our first attempt at basing a show around the use of a mobile phone app (app-based theatre) was with The Beautiful Ones, whose first half was an app-based audio guide with some very basic multiple choice capabilities. Lost Shoreline, more than a year later, relied on an app for the whole show to function, with significantly more complex interaction and structure. In the process of making these two shows we’ve come a long way in understanding what’s possible, so we can really start to take advantage of new possibilities for the theatre.
Why use an app?
For both of these shows, the app was used to solve problems. One of the simpler problems it solves is hardware costs; it’s like having everyone in the audience bring a computer for you to use. Of course, this alone is not really enough to justify it if you could use a simpler means. If your only requirement is to play a sequence of audio tracks, you could use mp3 players and save a considerable amount of time and effort in getting an app to do the same thing. If your only requirement is to let a few people vote anonymously, you could give them some buttons like this…
A big problem that we are using apps to try and solve is interactivity. Interactivity in the theatre tends to be limited in scope, because the number of audience members who can genuinely ‘interact’ simultaneously is limited to the number of performers/crew. Other types of ‘technical’ interactivity, such as radio voting systems (e.g. Ontroerend Goed’s Fight Night) allow you to go beyond this limitation, but the limitation then becomes the imagination of the creators of the system itself, and in this case there is only a single way to interact: a simple multiple choice question which can be selected by number at the time it is announced.
Another possibility is to make use of a pre-existing app (such as a messaging app like WhatsApp) to facilitate interaction. While it might not seem as polished, this enables many more new ways of interacting, connecting audience members to one another and creating more game-like options, as well as things like image/audio transfer. This is a great solution in some ways, because it is free, reliable and secure, not to mention easy to set up without any programming knowledge. The limitation of this is that it still has to be managed manually, and we come back to the limitation of how many interactions one performer can deal with simultaneously.
To go further than this the app itself needs to deal with certain kinds of logic specific to the show, managing what to do at certain times or in the event of certain combinations of situations. This means the app can take on the role of dealing with all basic interactions and timing and facilitating distribution of ‘real’ performers’ interactions, multiplying the amount of interaction that is possible… but this is going to require a custom-built app.
From The Beautiful Ones to Lost Shoreline, App-based Theatre
To find ways of storytelling using programming, we were inspired very much by the mechanics of ‘choose your own adventure’ and RPG computer games. The Beautiful Ones was the beginning of this idea, building a mechanic that offered very basic multiple choice options, a bit like a more linear old-fashioned RPG. Lost Shoreline was an evolution from this in much the same way as RPGs have evolved to be less linear, with the popularity of the ‘open world’ design. The freedom of the form makes storytelling much more challenging, increasing the amount of content that must be created, as well as multiplying the possibilities for how different players/participants might experience what you create. In spite of the challenges, we decided this direction was more appropriate to the content of Lost Shoreline, which drew inspiration from post-structuralism and historiographic metafiction.
One of the biggest challenges for us when working with new technology in theatre is finding ways to make the technology fundamentally serve the functional needs of the show, rather than decorate it superficially. However, integrating technology like this involves a lot more risk. If the ‘technology’ involved is limited to projections in the background etc., then the show can carry on if something fails, and the audience might not even notice. If the ‘technology’ is responsible for the structure, timing, location and audio of the whole show, then if something fails, there is no show any more. So while this makes the use of technology arguably more integrated and innovative, it makes the whole thing a lot scarier to create.
The Development of Lost Shoreline
When we started our creative meetings with the full team for Lost Shoreline, I still didn’t really know what was possible in terms of technology. At that time, I was still planning to create a self-contained app that didn’t require an internet connection to function, and worked more like a multiple-choice adventure game with mp3s and a basic GPS map. About 4 months before the show, the skeleton of an app that I’d made was still only a kind of custom google-maps thing with some audio capability.
However, in the creative meetings we kept coming up against the issue of organising the audience to make sure we didn’t end up with 40 people in a small room at the same time. We were coming up with ways involving probability to try and make it as unlikely as we could, but whatever we did, without a connection between the phones, there would still be a possibility of having the whole audience in the same place at the same time.
Since most of the team didn’t have much experience with this tech, their ideas weren’t limited by what they thought was possible. Even though Michelle and I had some experience from The Beautiful Ones, even I wasn’t exactly sure what was going to be possible for me to implement in time for the show with just me doing the programming. Ideas like ‘make it seem like someone’s phoning them at a specific time, and play audio when they answer’ seem easy to me because it’s mostly interface design, but seem almost impossible if you don’t know how it works. But things that sound simple like ‘send a notification to the closest 4 audience members to a specific location’ is actually much more complex to set up, because to calculate the closest four, you need live access to the location data of all audience phones. Still, not as bad as checking if their photo is of a bird. Anyway, that meant I had to set up a database the store the data from the phones in real time, and we had to make sure everyone had internet access. Luckily the audience was small enough to allow us to use the free version of Firebase for the database and it was much easier to set up than trying to set up my own.
Once we had a database everything became a lot easier to manage, and things felt a lot safer because we were able to make a lot of fundamental changes to the show without having to update the app, by getting data and even audio updates from the database instead. We were able to trigger and change times on the fly during the show in case anything went wrong. Basically, I realised that I should have built everything from the database up, and I pretty much had to rewrite everything I’d done up to this point. This was a great learning experience for me, and I started having more ideas than I could possibly implement – not just for this show, but for other apps as well. However, it was also a bit terrifying because I was learning how to do things for the first time and immediately putting them into an app that had to work perfectly the first time it was used with the full audience.
What The App Does
It would be pretty boring (or very niche) to go over every function in the app, but I want to highlight what I think are some of the cleverer bits.
The app works like a silent disco, with everyone hearing the audio in sync (down to milliseconds), but what you hear depends on your location. There are actors walking around who are also being tracked by GPS. You can see the actor’s location in your app as an animated coloured dot on your map, and if you’re within a certain distance you will hear the audio that relates to that actor. The volume gets louder the closer you get to the actor
There is an interactive overlay of how the shoreline of Kowloon changed over time with reclamation projects. You can slide to different dates and the map will let you see exactly how the shoreline was at that time, and where you are in relation to it. There’s a subtle soundscape that tries to emulate the sound of the sea according to how close you are to whatever you set the shoreline year to.
The timing of events is taken care of by the app so that everything stays in sync. People receive phone calls, dots pop up on the map, new functions get revealed, and like I mentioned before, the closest people to certain locations at certain times get summoned there. Sometimes these all happen together, and sometimes they are determined according to arrival time, or according to location. There is even a system in case two or more people need to stay together for some reason (e.g. carer, child etc.) which can be set up at the start of the show.
All of the phone numbers on the road can be dialled within the app (it has a phone-like interface) in order to hear an audio track that corresponds to the location that number can be found – usually about the street that the number can be found on. We got most of the phone numbers manually by photographing all the places we could, but if someone dialled a number not in the database, then their GPS coordinate at the time they dialled it would be added to another database for checking. These were then added back into the main database after each show, so we gradually built up a full database of the locations of all the phone numbers on the streets – the audience did it for us without even knowing.
If an audience member has a problem, they can use the app to call us directly – we also get their GPS coordinates so we can go straight to their location and help them. Thankfully this was mostly not used, except for a few people requesting battery chargers, or on on occasion someone who was seeing the show for the second time without the app being reset properly. We were able to run directly to them without causing them any delay.
What We Could Improve
One of the biggest problems we encountered was the potential inaccuracy of the GPS. When I first built the GPS-based audio functions, I was using very finely calibrated distance to volume and even panning according to bearing, but in real world testing this proved to be disorientating due to the amount of buildings and footbridges interfering with the GPS signal. This would be a great thing to try in a flat and open space, but it’s not an ideal system in Hong Kong. I had to remove the panning and reduce the sensitivity of the volume a lot to account for the potential inaccuracy, so it was still able to function well enough for our purposes. I considered other ways of detecting user location, from masses of battery powered bluetooth beacons, to ushers in charge of updating individual audience members’ locations… but I think in the end I will go back to giving the user more agency in telling the app where they are and triggering what they want to hear manually, rather than trying to make everything automated.
Another difficulty when working with this level of technology is time. Even though we had more than twice as long as a normal rehearsal period to create the show, it was still crazy for me to try and do the programming and testing on my own, and meant that I couldn’t be involved in as much of the creative content as I would have liked. Because the technical aspects and the content of the show were so mutually dependent, it wasn’t really possible to create the app first and find the content for it, so both sides of things had to develop together. I think this kind of show would benefit from having the process spread out much more to allow time for app development in between meetings, but the current funding/rehearsal model for ‘theatre’ shows like this doesn’t really allow for a process that’s conducive to integrating new technology and content. Not that that will stop us trying.
The Future
For our next production, I'm switching to writing in React Native so that I can keep largely the same codebase for both iOS and Android versions. As well as keeping me sane, it's allowing me to push more towards integrating things with our website and automating things like the ticketing process. More to follow soon...