Author: Jenna Ng and Oliver Tomkins
Format: Video
Duration: 33′ 25″
Published: July 2023
https://doi.org/10.37186/swrks/13.1/11

A descriptive transcript can be accessed here.

---

Research Statement

Research Questions
The video essay’s starting point is to explore the implications of highly realistic images that appear, interact and socialise with human users, often seemingly “live” in real-time. Usually created with imaging technologies such as holographic projection or virtual reality (VR) and often boosted with AI and machine learning, these images radically break down boundaries between virtual and actual realities, truth and fakery, real and unreal.

The essay’s research questions interrogate these new/renewed confrontations between viewer and image: what is the state of reality out of media that so freely and near-seamlessly mix virtuality and actuality? How do we express and make sense of this reality? How does it change the image’s relations to truth? How does it relate to the wider contexts of mis/disinformation and deepfakes that colour contemporary times?

Approach
The essay connects its questions to issues of the image’s ontology, indexicality and relations to truth long discussed in classic and contemporary film theory (see “Context” below). The increasing cultural dominance and realism of the digital image renews the urgency of these questions. We argue that these images in question not only present a renewed articulation of reality. They also engender a certain disposition of dramatic discombobulation, bewilderment and disorientation resulting from the freewheeling vacillation of actuality and virtuality across the image’s blurred and merged boundaries. While abstract, this mood and setting, too, are central to the argument that address the video essay’s research questions.

These considerations formed key motivations for our making a video and eschewing a more straightforward written text. The video thus stands as both an exposition of the new virtuality and an audiovisual rendering of its poetics. For instance, the bookended structuring of depicted and described scenes frame the essay’s argument as splashes of imagination for the audience, or as self-reflective “mood boards,” in a way. We wrote the narration with a conscious rhetoric and meter of elegy and pensiveness. We chose images and music in the same creative vein. In these ways, we also made the video in acute consciousness of creating it as para-academic work via what Joy and Musciandaro identified in their panel discussion as “the multivalent sense of something that fulfills and/or frustrates the academic from a position of intimate exteriority” (as quoted in Boshears 2014, 179). We found this multivalence in the audiovisual expressiveness of the video. In a medium that is not only more accessible, but also enables greater creative space in and affordances with which to convey mood, emotion and setting, it amplifies the words of the argument, grants it colour and vividness, and re-sounds the evocations of this new state of reality.

Context / Significance
The essay continues existing discussion on where and how visual media have always pushed the boundaries between the virtuality of images against the actuality of the viewer’s environment. Here, we use “virtuality” to specifically refer to the particular ontological status of an object, rather than its more common conceptions as or associations with digital/computational or mimetic representation. In this respect, we adopt Friedberg’s (2009) approach of diving back into the term’s definition and etymology, whereby she quotes (as do we) the Webster Dictionary’s definition of “virtual” (stemming from its Latin root “virtus” for strength or power) as “of, relating to, or possessing a power of acting without the agency of matter; being functionally or effectively but not formally of its kind” (as cited on 8). In this sense, as Friedberg puts it, “the virtual is a substitute – ‘acting without agency of matter’ – an immaterial proxy for the material” (ibid). Virtuality is thus a register of representation with a different kind of reality and materiality, or what Friedberg calls a “second-order materiality, liminally immaterial” (11). It is not a direct mimetic copy, but a transfer or transformation of the object to another appearance or another plane of meaning (see also Gunning 2019).

In this respect, many scholars have commented on the significance of the boundaries between virtual and actual realities, most notably in relation to the image as the virtual. For instance, Derrida (1987) highlights the function of the virtual as parergon: “neither work (ergon) nor outside the work [hors d’oeuvre] neither inside nor outside, neither above nor below, [the boundary] disconcerts any opposition but does not remain indeterminate and it gives rise to the work” (Derrida 1987, 9; emphasis added). To Krauss (1985), the painting’s frame is that boundary. It “crops or cuts the represented element out of reality-at-large,” and “announces that between the part of reality that was cut away and this part there is a difference” (115). In relation to cinema, Friedberg (2009) refers to the “ontological cut,” taking the term from Victor Stoichita who used it to refer to the demarcation between the portable panel painting and the wall: “Like the frame of the architectural window and the frame of the painting, the frame of the moving-image screen marks a separation – an ‘ontological cut’ – between the material surface of the wall and the view contained within [the frame’s] aperture” (Friedberg 2009, 5 and repeated at 157). Applied to other media from paintings and trompe l’oeil to photography and cinema to VR and immersive environments, scholars have extensively discussed the ontology of the image in both its separations and muddying against the referent (e.g. Bazin 1971; Daney 1972; Andrew 1976; Cardinal 1986; Elkins 1996; Baudrillard 2002; Carbone 2016; Peretz 2017). This discourse was renewed at the millennium’s turn with the advent of computer-generated imagery (e.g. Mulvey 2006; Rodowick 2007; Prince 2012; Kim 2016). More recently, the virtuality of the image received renewed attention from new angles, such as cinema marketing (Atkinson 2014); posthumanism (Brown 2013; Brown and Fleming 2020); and computational culture (Jones 2020), to name a few examples.

Our essay advances this scholarship in two ways. Firstly, it distinguishes a corpus of images that we argue herald different actual/virtual relations from its precedents. Discourse on the virtuality of the image generally focus on dematerialization (e.g. Hayles 1999) or immersiveness (e.g. Grau 2003). Studying the images of the new virtuality, our essay extends the understanding of virtuality from transfer to vacillation. We identify these images of the new virtuality as beyond representation. Rather, they are a complex dialogue that trades and wheels across the virtual and actual. In this dialogue, virtuality stands on multiple paradoxes: they are believed yet disbelieved; manipulated yet recognisable; realistic yet fake. In turn, an intense discombobulation arises, out of which even more virtuality is sought, experienced and consumed. The video’s second advancement is to connect the new virtuality to the wider politics of post-truth, mis/disinformation and deepfakes. These politics, too, rest on the notion of disappearing difference without positive terms – between scene and reality; event and representation; record and referent. Our argument of the new virtuality’s play between real and unreal thus also reads virtuality as a critical signal fire that demonstrates post-truth’s pliable subjectivity and its concomitant challenges to frameworks of disbelief, reliability and certainty.

In this sense, media – and its virtualities – also convey important ideas of how orality, literacy and linearity (e.g. Ong [1982] 2002; Flusser [1986] 2015) tie into history and historical consciousness. We thus argue for the virtuality of the image as the new battleground for understanding the politics of current and future realities and histories. Key understandings for this battleground are how virtuality is built, constructed and believed; the spaces and structures in which it operates; the aims and ambitions as driven by its creators; the consequences of its power. The New Virtuality thus highlights new areas and questions for the meanings of our realities: not by way of content or constitution, but the assertion (and diminishment) of boundaries and difference.

As a research trajectory, this video essay extends ideas from the first author’s latest book, The Post-Screen Through Virtual Reality, Holograms and Light Projections: Where Screen Boundaries Lie (Ng 2021). In The Post-Screen, Ng identifies these images of near-seamless actual/virtual integrations as “the post-screen” in how they subvert screen boundaries and demarcations. While the book re-formulates the notion of the screen, the video essay expands its critical focus beyond the screen to the concept of virtuality itself and its ensuing engagements with understandings of reality. It continues and extends Ng’s work of studying these images and understanding what they mean. The video is also part of a larger multi-media portfolio online piece created by both authors to further explore these ideas of virtuality (Ng and Tomkins 2022). Leveraging the connective pages of a website, the piece interweaves written essays, image galleries, creative fiction and interactive story as a hybrid creative-academic work. Alongside the video, the website (at www.thenewvirtuality.com) aims to be an accessible and readable work about our ideas on contemporary images and notions of reality.

Methods
Our methodology for the video essay derived primarily from Screen, Cultural and Media Studies, including readings of film and media theory; archival research (including reviews and paratexts); textual readings; and media archaeology (Parikka 2012). In particular, we employed the last to draw an arc of media history in which to situate the new virtuality, tying the notion of media boundaries from panoramas to VR to demonstrate its thematic continuities and deep contexts. From these contexts, we elucidated the new virtuality by connecting theory with readings across multiple media sites of paintings, digital art, cinema, performance, VR, apps and architecture to illustrate and support our argument.

Outcomes
The essay conveys three pieces of new knowledge. The first (over Sections II – III) explicates the new virtuality as mapped across a historical arc of “old” media and inspirations. These exemplifications range from “found-footage horror”; 3D cinema; film marketing practices; and Victorian visual entertainment such as Pepper’s Ghost projections and panoramas; to René Magritte’s painting La Condition Humaine as a key thematic exemplar. The arc demonstrates media’s consistent manoeuvrings of boundaries between actuality and virtuality, and their plays with difference. Demarcations of boundaries, such as via the painting’s frame, cinema screen bezel, theatre stage proscenium etc, are undone or subverted as much through radical marketing and staging practices as from imaging technologies such as VR and holographic projections.

The second (IV – V) is the essay’s key argument on reaching the arc’s current end point: realistic imaging technologies that present engagement live and in real-time across increasingly unstable actual/virtual boundaries. The user confronts a new state of virtuality, one that we argue is not quite virtual nor actual, nor an amalgamation of the two. Rather, it is a limbic space of vacillation between virtuality and actuality. The key exemplar here is the virtual human – highly realistic 3D images of humans who appear onscreen and onstage in real-time, but do not exist in actuality. We draw examples from a wide range of media, including live mixed reality from East Asia (China and Japan) which might be new to anglophone scholars. In the new virtuality, the simulacra collapses into the real, where reality and illusion no longer have their old semantic values as counters or opposites to each other. Instead, they point to a regime of truth values borne of no difference and without positive terms, whereby difference itself is not about the sign but about those that surround it.

The third (VI) is our exploration of the implications from this loss of difference as profound discombobulations of reality and a resulting hunger for ever more virtual images of “reality”. The new virtuality thus elides the represented, constituting a black hole of representability that paves the way for a gluttony of virtuality, a devouring of a surfeit of images and virtual life. In a sense, this gluttony received an unexpected boost when various parts of the world largely retreated into virtual life under the Covid-19 lockdowns of 2020-21. While these lockdowns have been mostly lifted, virtual working and living still continue today. With subdued yet sure determination, the virtual invades the actual as human activities and socialisations shift into the former in rejection of the latter. In this vein, too, the metaverse beckons, with its new political, capital and power structures.

Impact
The video was launched on August 31, 2022 with announcements on Twitter and LinkedIn. A trailer for the video was also shown and publicised at two conferences – the inaugural Digital Creativity, Industry and Culture (DCIC) conference at the University of York on September 20, 2022 which had 130+ attendees consisting of academics, practitioners and industry partners; and the Live Xinema IV 2022 online conference on September 23, 2022. At the former, we also circulated information leaflets, and the first author presented a talk on the project. The first author further presented the project at a discussion salon with students and academics at the Centre for Film and Ethics, Queen Mary University of London, on November 29, 2022. Both authors also presented the work at the Interactive Film & Media (IFM) Annual Conference 2023 in June earlier this year. The trailer for the video will also be screened at the official launch of the School of Arts and Creative Technologies at the University of York in June 2023 as part of the showcase of interactive media work and research from the School.

On May 4, 2023, the video essay won the Special Jury Prize at the Learning on Screen annual awards, with praise from the judges on the work being “an absolutely fascinating film, which was excellently cut and curated to tell the story of AI in an amazing and thoroughly engaging way” (Learning on Screen, 2023). The larger practice-based work of the multi media website (see “Context/Significance” above) is also the recipient of the 2023 John Culkin Award for Outstanding Praxis in the Field of Media Ecology (awarded by the Media Ecology Association). It has also been shortlisted for a Media, Communication and Cultural Studies Association (MeCCSA) Practice-Based Research of the Year Award 2023, whose results will be announced at the MeCCSA conference in September.

Future dissemination plans include presenting this work at other public fora, such as the York Festival of Ideas, a flagship showcase of talks and other events between the University of York and the city. We anticipate substantive interest from all sectors on this topic, particularly as we reflect on how these images are already key constituents of our worlds of entertainment, work and play, and their potentials for much more.

Bibliography
Andrew, D. (1976) The Major Film Theories: An Introduction. Oxford: Oxford University Press.

Atkinson, S. (2014) Beyond the Screen: Emerging Cinema and Engaging Audiences. New York: Bloomsbury.

Baudrillard, J. (2002) Screened Out. Trans. Chris Turner. London; New York: Verso.

Bazin, A. (1971) “An Aesthetic of Reality: Neorealism”. In: H. Gray, ed. and trans. What is Cinema? Vol. II. Berkeley; Los Angeles, CA: University of California Press, 2005, pp. 16-40.

Boshears, P. (2014) “Para-Academic Publishing as Public-Making”. In: A. Wardrop and Deborah Withers, eds. The Para-Academic Handbook: A Toolkit for Making-Learning-Creating-Acting. Bristol: HammerOn Press, pp. 175-188.

Brown, W. (2013) SuperCinema: Film-Philosophy for the Digital Age. New York; Oxford: Bergahn.

Brown, W. and D.H. Fleming. (2020) The Squid Cinema From Hell: Kinoteuthis Infernalis and the Emergence of Chthulumedia. Edinburgh: Edinburgh University Press.

Cardinal, R. (1986) “Pausing over Peripheral Detail”. Framework. 30-31, pp. 112-130.

Carbone, M. (2016) Philosophy-Screens: From Cinema to the Digital Revolution. Trans. Marta Nijhuis. Albany: State University of New York Press.

Daney, S. (1972) “The Screen of Fantasy (Bazin and Animals)”. In: I. Margulies, ed. and M.A. Cohen trans. Rites of Realism: Essays on Corporeal Cinema. Durham, NC: Duke University Press, 2002, pp. 32-41.

Derrida, J. (1987) The Truth in Painting. Trans. Geoff Bennington and Ian McLeod. Chicago, IL: University of Chicago Press.

Elkins, J. (1996) The Object Stares Back: On the Nature of Seeing. New York: Harvest.

Flusser, V. ([1986] 2015) Into Immaterial Culture. Rodrigo Maltez Novaes, ed. and trans. Milton Keynes: Metaflux Publishing.

Friedberg, A. (2009) The Virtual Window: From Alberti to Microsoft Cambridge. MA: MIT Press.

Grau, O. (2003) Virtual Art: From Illusion to Immersion. Cambridge, MA: MIT Press.

Gunning, T. (2019) “Moving Through Friedberg’s Properly Adjusted Virtual Window”. In: E. Dimendberg, ed. The Moving Eye: Film, Television, Architecture, Visual Art and the Modern. Oxford: Oxford University Press, 2019, pp. 13-31.

Hayles, K. (1999) How We Became Posthuman: Virtual Bodies in Cybernetics, Literature and Informatics. Chicago: University of Chicago Press.

Kim, J. (2016) Between Film, Video, and the Digital: Hybrid Moving Images in the Post-Media Age. New York: Bloomsbury.

Krauss, R. (1985) The Originality of the Avant-garde and Other Modernist Myths. Cambridge, MA: MIT Press.

Jones, N. (2020) Spaces Mapped and Monstrous: Digital 3D and Visual Culture. New York: Columbia University Press.

Learning on Screen. (2023) “Winners and Nominations 2023.” May 5, 2023. Accessed June 20, 2023. https://learningonscreen.ac.uk/awards/nominations-2023/.

Mulvey, L. (2006) Death 24x a Second: Stillness and the Moving Image. London: Reaktion.

Ng, J. (2021) The Post-Screen Through Virtual Reality, Holograms and Light Projections: Where Screen Boundaries Lie. Amsterdam: Amsterdam University Press.

Ng, J. and O. Tomkins. (2022) “The New Virtuality.” August 31, 2022. Accessed September 13, 2022. http://thenewvirtuality.com.

Ong, W.J. ([1982], 2002) Orality and Literacy. New York; Abingdon: Routledge.

Parikka, J. (2012) What is Media Archaeology?. Cambridge; Malden: Polity.

Peretz, E. (2017) The Off-Screen: An Investigation of the Cinematic Frame. Stanford, CA: Stanford University Press.

Prince, S. (2012) Digital Visual Effects in Cinema: The Seduction of Reality. New Brunswick, NJ: Rutgers University Press.

Rodowick, D.N. (2007) The Virtual Life of Film. Cambridge, MA: Harvard University Press. Webography

Filmography
2060 (Youku, 2021, China)
Avatar 2: The Way of Water (James Cameron, 2022, USA)
Bicycle Thieves (Vittoria De Sica, 1948, Italy)
The Blair Witch Project (Daniel Myrick, Eduardo Sanchez, 1999, USA) District 9 (Neill Blomkamp, 2009, South Africa)
Fight Club (David Fincher, 1999, USA)
Gemini Man (Ang Lee, 2019, USA)
House of Wax (Andre DeToth, 1953, USA)
The Irishman (Martin Scorsese, 2019, USA)
The Matrix (Wachowskis, 1999, USA)
Meeting You (Kim Jong-woo, 2020, Republic of Korea)
Minority Report (Steven Spielberg, 2002, USA)
Ringu (Hideo Nakata, 1998, Japan)
Spider-Man: Far From Home (Jon Watts, 2019, USA)

Television
Doctor Who, “Time of Angels” (wri. Steven Moffat, dir. Adam Smith, 2010, UK) Years and Years (wri. Russell T. Davies, dir. Simen Cellan Jones, 2019, UK)

Ludography
Gorn (Devolver Digital, 2019)
Star Wars: Secrets of the Empire (THE VOID, 2017)
We Wait (Aardman Digital, 2017)

 

---

Peer Reviews

All reviews refer to original research statements which have been edited in response to what follows:

Review 1: Invite resubmission with major revisions of practical work and/or written statement
As the pace of tech innovation continues to accelerate, I am grateful for this essay as a resource of instances of deepfakes, virtual protests, holographic performances, VR games, and all manner of wizardry from Pepper’s Ghost to a cherub escaping a Rubens painting. It’s quite a smorgasbord. And while any viewer could think of another example that might have been included, every one of them would be superfluous, because there’s no way to be exhaustive in the space of half an hour. Jenna does the crucial and praiseworthy work of supplying excellent test cases to sketch the scope of the field. However, I cannot recommend this video essay for publication in its current form for two interconnected reasons: It lacks clarity in its terminology, and that lack of clarity undermines the argument.

Had the subject of the essay been the inadequacy of terms like “virtual” to describe contemporary technological mediation and intervention, this essay might have sounded a very necessary alarm. Throughout, I was struck by how many different shades of meaning the word “virtual” has been forced to carry. It’s straining to the point of bursting due to all the different phenomena we’ve tried to shove inside it. But instead of supplying a new vocabulary for a new era, or even just demonstrating the dangerous slipperiness of existing vocabulary, this essay bases everything on a binarism of actual and virtual, of real and unreal; and despite invoking Baudrillard once, it does not sufficiently clarify what is meant by actual and virtual, and often deploys them contradictorily.

For instance: the essay claims that a virtual influencer named Mikayla “looks, sounds, and moves like a real person,” and that she is “entirely realistic… seeming to be actual people with actual lives.” This essay was my introduction to Mikayla, and I was not taken in by the illusion, though I briefly had been by Shudu Graham, the other virtual model. So, does virtuality represent an aspirational attempt at photorealistic verisimilitude? I would accept that definition if not for the fact that a moment later “virtual” is deployed as an apparent synonym for “cartoonish” in the context of Hatsune Miku. Then, in the final Burj Khalifa example, the essay argues that the unusual elevation constitutes a virtual view of the city. How exactly? Here, no technology of mediation is even in play. Ultimately, I think these concerns could be addressed by un-writing a great deal of the essay. One option would be to use these examples to meditate on how the word “virtual” has come to encompass too much, and why that’s a problem. Another option might be to attempt a new classification and critique its limitations/effectiveness. And there are other options as well.

Whatever you choose, don’t be afraid to say fewer words and draw fewer explicit conclusions. Allow your examples to speak for themselves, and to speak to each other through editing, and allow the viewer a space to consider what “virtuality” is, means, or is-becoming nowadays.

Review 2: Accept submission subject to minor revisions of written statement
Ng and Tomkins’ video essay presents a highly engaging examination of the aesthetic and affective characteristics of the real and virtual boundary, contributing new and original analyses of numerous case study examples from film and media, which holistically provide an incredibly useful genealogy to the phenomenon under question. The video benefits from high production values – made from a wealth of found footage excerpts, the skilfully mixed soundtrack with voice over and music makes the video’s arguments even more engaging and compelling.

Structured by six chapters, the video essay leads the viewer through different facets of ‘the new virtuality’. The introduction presents two examples which sets the scene for the vast spectrum of visual phenomenon under consideration – from holographic illusions created by digital overlays onto the real world (augmented reality) to full blown CGI simulations representing persistent digital worlds (the metaverse). The accompanying research statement builds on extant scholarship which has hitherto examined the real to virtual border in arts, technology, and culture, and provides useful insights into the creative process and a clear rationale for the ordering and structuring of the videos content. The statement also reveals that the video essay is part of an even broader portfolio of work which includes an image-rich companion website making for an excellent teaching resource and a key point of reference which can be expanded and updated. This additional resource proves incredibly useful in providing further detail about the dizzying number of examples from which the video essay draws with some key touchstone examples left un-discussed e.g., the deep fake video of the Queen’s alternative Christmas message of 2020.

It is the video’s fifth chapter, ‘The Virtual Human’ where the key underpinning themes and questions of the video essay are synthesised, and that is to me, the most fascinating. The ‘beings of the new virtuality’ include digital super model Shudu Gram, Lil Miquela – a social media superstar, Japanese virtual performer Hatsune Miku, and the now infamous ‘ABBAtars.’ Ng and Tomkin’s argue that these CGI created individuals showcase their virtuality and their existence signals the beginnings of the widespread acceptance of the virtual performer in contemporary media. This is only set to advance in further moves towards the live experience of virtual performance – where large-scale live audiences will be able to synchronously see and interact with the CGI characters through augmented/mixed reality eye wear. This will lead to new questions requiring further attention and examination. The video essay opens up these questions and more, and will undoubtedly stimulate further enquiry drawing from the impressive wealth of examples and insight that Ng and Tomkins have scrupulously curated.

The video essay not only encourages viewers to question image making and reception in a post-truth era where AI-generated images proliferate – it also successfully captures a significant transitionary moment in media history. The ‘new virtuality’ is a temporally bound moment, situated in a liminal gap, which will soon close, given the rapid evolution of new digital imaging technologies. The video essay will therefore become a very important reference point which has revealed and recorded key insights into this enthralling and complicated temporary space before the joins become invisible, before the line disappears, and before the gap no longer shows.

All reviews refer to original research statements which have been edited in response to the above