Continued from Part 1 which can be found here: Making App-based Theatre
One of the biggest challenges for us when working with new technology in theatre is finding ways to make the technology fundamentally serve the functional needs of the show, rather than decorate it superficially. However, integrating technology like this involves a lot more risk. If the ‘technology’ involved is limited to projections in the background etc., then the show can carry on if something fails, and the audience might not even notice. If the ‘technology’ is responsible for the structure, timing, location and audio of the whole show, then if something fails, there is no show any more. So while this makes the use of technology arguably more integrated and innovative, it makes the whole thing a lot scarier to create.
The Development of Lost Shoreline
When we started our creative meetings with the full team for Lost Shoreline, I still didn’t really know what was possible in terms of technology. At that time, I was still planning to create a self-contained app that didn’t require an internet connection to function, and worked more like a multiple-choice adventure game with mp3s and a basic GPS map. About 4 months before the show, the skeleton of an app that I’d made was still only a kind of custom google-maps thing with some audio capability.
However, in the creative meetings we kept coming up against the issue of organising the audience to make sure we didn’t end up with 40 people in a small room at the same time. We were coming up with ways involving probability to try and make it as unlikely as we could, but whatever we did, without a connection between the phones, there would still be a possibility of having the whole audience in the same place at the same time.
Since most of the team didn’t have much experience with this tech, their ideas weren’t limited by what they thought was possible. Even though Michelle and I had some experience from The Beautiful Ones, even I wasn’t exactly sure what was going to be possible for me to implement in time for the show with just me doing the programming. Ideas like ‘make it seem like someone’s phoning them at a specific time, and play audio when they answer’ seem easy to me because it’s mostly interface design, but seem almost impossible if you don’t know how it works. But things that sound simple like ‘send a notification to the closest 4 audience members to a specific location’ is actually much more complex to set up, because to calculate the closest four, you need live access to the location data of all audience phones. Still, not as bad as checking if their photo is of a bird. Anyway, that meant I had to set up a database the store the data from the phones in real time, and we had to make sure everyone had internet access. Luckily the audience was small enough to allow us to use the free version of Firebase for the database and it was much easier to set up than trying to set up my own.
Once we had a database everything became a lot easier to manage, and things felt a lot safer because we were able to make a lot of fundamental changes to the show without having to update the app, by getting data and even audio updates from the database instead. We were able to trigger and change times on the fly during the show in case anything went wrong. Basically, I realised that I should have built everything from the database up, and I pretty much had to rewrite everything I’d done up to this point. This was a great learning experience for me, and I started having more ideas than I could possibly implement – not just for this show, but for other apps as well. However, it was also a bit terrifying because I was learning how to do things for the first time and immediately putting them into an app that had to work perfectly the first time it was used with the full audience.
What The App Does
It would be pretty boring (or very niche) to go over every function in the app, but I want to highlight what I think are some of the cleverer bits.
The app works like a silent disco, with everyone hearing the audio in sync (down to milliseconds), but what you hear depends on your location. There are actors walking around who are also being tracked by GPS. You can see the actor’s location in your app as an animated coloured dot on your map, and if you’re within a certain distance you will hear the audio that relates to that actor. The volume gets louder the closer you get to the actor.
There is an interactive overlay of how the shoreline of Kowloon changed over time with reclamation projects. You can slide to different dates and the map will let you see exactly how the shoreline was at that time, and where you are in relation to it. There’s a subtle soundscape that tries to emulate the sound of the sea according to how close you are to whatever you set the shoreline year to.
The timing of events is taken care of by the app so that everything stays in sync. People receive phone calls, dots pop up on the map, new functions get revealed, and like I mentioned before, the closest people to certain locations at certain times get summoned there. Sometimes these all happen together, and sometimes they are determined according to arrival time, or according to location. There is even a system in case two or more people need to stay together for some reason (e.g. carer, child etc.) which can be set up at the start of the show.
All of the phone numbers on the road can be dialled within the app (it has a phone-like interface) in order to hear an audio track that corresponds to the location that number can be found – usually about the street that the number can be found on. We got most of the phone numbers manually by photographing all the places we could, but if someone dialled a number not in the database, then their GPS coordinate at the time they dialled it would be added to another database for checking. These were then added back into the main database after each show, so we gradually built up a full database of the locations of all the phone numbers on the streets – the audience did it for us without even knowing.
If an audience member has a problem, they can use the app to call us directly – we also get their GPS coordinates so we can go straight to their location and help them. Thankfully this was mostly not used, except for a few people requesting battery chargers, or on on occasion someone who was seeing the show for the second time without the app being reset properly. We were able to run directly to them without causing them any delay.
What We Could Improve
One of the biggest problems we encountered was the potential inaccuracy of the GPS. When I first built the GPS-based audio functions, I was using very finely calibrated distance to volume and even panning according to bearing, but in real world testing this proved to be disorientating due to the amount of buildings and footbridges interfering with the GPS signal. This would be a great thing to try in a flat and open space, but it’s not an ideal system in Hong Kong. I had to remove the panning and reduce the sensitivity of the volume a lot to account for the potential inaccuracy, so it was still able to function well enough for our purposes. I considered other ways of detecting user location, from masses of battery powered bluetooth beacons, to ushers in charge of updating individual audience members’ locations… but I think in the end I will go back to giving the user more agency in telling the app where they are and triggering what they want to hear manually, rather than trying to make everything automated.
Another difficulty when working with this level of technology is time. Even though we had more than twice as long as a normal rehearsal period to create the show, it was still crazy for me to try and do the programming and testing on my own, and meant that I couldn’t be involved in as much of the creative content as I would have liked. Because the technical aspects and the content of the show were so mutually dependent, it wasn’t really possible to create the app first and find the content for it, so both sides of things had to develop together. I think this kind of show would benefit from having the process spread out much more to allow time for app development in between meetings, but the current funding/rehearsal model for ‘theatre’ shows like this doesn’t really allow for a process that’s conducive to integrating new technology and content. Not that that will stop us trying. Look out for news of something along these lines coming up…