Climate Action

Interactive cinema: How films could alter plotlines in real time by responding to viewers’ emotions

Solving this issue of breaking the immersive experience remains a key question for artists exploring interactive film.

Solving this issue of breaking the immersive experience remains a key question for artists exploring interactive film. Image: Unsplash/Jake Hills Avatar of user Jake Hills Jake Hills

Richard Ramchurn
Assistant researcher, University of Nottingham
Share:
Our Impact
What's the World Economic Forum doing to accelerate action on Climate Action?
The Big Picture
Explore and monitor how Climate Crisis is affecting economies, industries and global issues
A hand holding a looking glass by a lake
Crowdsource Innovation
Get involved with our crowdsourced digital platform to deliver impact at scale
Stay up to date:

Climate Crisis

  • A new film about a pair of climate activists uses machine-learning technology to adapt the story in line with the viewer's emotional response.
  • Through the use of a computer camera and software, the film effectively watches the audience as they view footage of climate disasters and are implicitly asked difficult questions.
  • In this article, its director shares his narrative approach and views on interactive technology as a way of expanding the filmmaker’s toolkit.

Most films offer exactly the same viewing experience. You sit down, the film starts, the plot unfolds and you follow what’s happening on screen until the story concludes. It’s a linear experience. My new film, Before We Disappear – about a pair of climate activists who seek revenge on corporate perpetrators of global warming – seeks to alter that viewing experience.

What makes my film different is that it adapts the story to fit the viewer’s emotional response. Through the use of a computer camera and software, the film effectively watches the audience as they view footage of climate disasters. Viewers are implicitly asked to choose a side.

I chose to use this technology to make a film about the climate crisis to get people to really think about what they are willing to sacrifice for a survivable future.

Storytelling has always been interactive: traditional oral storytellers would interact and respond to their listeners. For almost a century, film directors have been experimenting with interactivity – the past decade has seen an explosion of interactive content.

Streaming services give viewers the opportunity to choose their own adventure. However, letting the viewer control the action has long posed a challenge: it’s at odds with narrative immersion, where the viewer is drawn into the world created by the story.

One of the most prominent recent experiments in interactive film, Netflix’s Bandersnatch, clearly illustrates this. Here the action stops to ask the user what to do next – breaking the flow of the story and actively involving the viewer. Solving this issue of breaking the immersive experience remains a key question for artists exploring interactive film.

The interactive film, Before We Disappear uses emotional cues from the viewer to edit the film in real time.
The interactive film, Before We Disappear uses emotional cues from the viewer to edit the film in real time. Image: AlbinoMosquito Productions Ltd

The films I create and direct take a different route, leveraging non-conscious control to influence a film as the audience watches. My previous brain-controlled films, The Moment (2018) and The Disadvantages of Time Travel (2014), used brain computer interfaces (BCIs). These systems use computers to analyse electrical signals from the brain, allowing people to effectively control a device with their minds.

Using this data from the brain, audiences create a non-conscious edit of the film in real time – reinforcing the films’ respective stories of science-fiction dystopia and a wandering, daydreaming mind.

However, the BCI interface requires specialised equipment. For Before We Disappear, I wanted to use a technology more readily available to audiences, that could allow films to be shared over the internet.

Controlling the narrative

Before We Disappear uses an ordinary computer camera to read emotional cues and instruct the real-time edit of the film. To make this work, we needed a good understanding of how people react to films.

We ran several studies exploring the emotions filmmakers intend to evoke and how viewers visually present emotion when watching. By using computer vision and machine learning techniques from our partner BlueSkeye AI, we analysed viewers’ facial emotions and reactions to film clips and developed several algorithms to leverage that data to control a narrative.

Behind the scenes of filming Before We Disappear, an interactive film that changes plotline in response to audiences' emotions.
Behind the scenes of filming Before We Disappear, an interactive film that changes plotline in response to audiences' emotions. Image: AlbinoMosquito Productions Ltd

While we observed that audiences tend not to extensively emote when watching a film, BlueSkeye’s face and emotion analysis tools are sensitive enough to pick up enough small variations and emotional cues to adapt the film to viewer reactions.

The analysis software measures facial muscle movement along with the strength of emotional arousal – essentially how emotional a viewer feels in a particular moment. The software also evaluates the positivity or negativity of the emotion – something we call “valence”.

We are experimenting with various algorithms where this arousal and valence data contributes to real-time edit decisions, which causes the story to reconfigure itself. The first scene acts as a baseline, which the next scene is measured against. Depending on the response, the narrative will become one of around 500 possible edits. In Before We Disappear, I use a non-linear narrative which offers the audience different endings and emotional journeys.

Emotional journey

I see interactive technology as a way of expanding the filmmaker’s toolkit, to further tell a story and allow the film to adapt to an individual viewer, challenging and distributing the power of the director.

Discover

How is the World Economic Forum ensuring the responsible use of technology?

However, emotional responses could be misused or have unforeseen consequences. It is not hard to imagine an online system showing only content eliciting positive emotions from the user. This could be used to create an echo chamber – where people only see content that matches the preferences they already have.

Or it could be used for propaganda. We saw in the Cambridge Analytica scandal how large amounts of personal information were collected from Facebook and used for political advertising.

A film scene of two women sitting in a dingy interior. interactive film
The new film explores solutions to the climate crisis. Image: AlbinoMosquito Productions Ltd

Our research aims to generate conversation about how users’ emotion data can be used responsibly with informed consent, while allowing users to control their own personal information. In our system, the data is analysed on the users’ device, rather than, say, the cloud.

Have you read?

Big business, big responsibility

Non-conscious interaction is big business. Platforms such as TikTok and YouTube use analysis of users’ past interactions on the platforms to influence the new content they see there. Users are not always aware of what personal information is being created or stored, nor can they influence what algorithms will present to them next.

It’s important to create a system where audiences’ data is not stored. Video of the viewer or facial expression data should not be uploaded or analysed anywhere but on the player device. We plan to release the film as an interactive app, incorporating an awareness of potential abuse of the user’s data, and safeguarding any personal data on the device used to watch it.

Adaptive films offer an alternative to traditional “choose-your-own-adventure” storytelling. When the story can change based on the audiences’ unconscious responses rather than intentional interaction, their focus can be kept in the story.

This means they can enjoy a more personalised experience of the film. Turns out the old traditions of storytelling may still have much to teach us in the 21st century.

Loading...
Don't miss any update on this topic

Create a free account and access your personalized content collection with our latest publications and analyses.

Sign up for free

License and Republishing

World Economic Forum articles may be republished in accordance with the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International Public License, and in accordance with our Terms of Use.

The views expressed in this article are those of the author alone and not the World Economic Forum.

Related topics:
Climate ActionNature and BiodiversityEmerging Technologies
Share:
World Economic Forum logo
Global Agenda

The Agenda Weekly

A weekly update of the most important issues driving the global agenda

Subscribe today

You can unsubscribe at any time using the link in our emails. For more details, review our privacy policy.

Shifting spaces: Could tackling climate change in cities help solve the youth mental health crisis?

Natalie Marchant and Julie Masiga

July 19, 2024

About Us

Events

Media

Partners & Members

  • Sign in
  • Join Us

Language Editions

Privacy Policy & Terms of Service

© 2024 World Economic Forum