Research Statement

Research Questions

Where You End and I Begin extends my work using ImageJ, a public domain scientific image analysis software program, to explore, abstract, and reconfigure film texts in experimental, research-driven ways. Building on prior work, this video explores the consequences of viewing narrative cinema via algorithm, specifically edge detection methods developed by Canny, Sobel, Laplace, Hessian, Bernsen, and others which isolate lines and contours. By performing these operators on individual film frames, I create new versions of film scenes that are abstracted and reduced to their outlines, akin to watching a film from the point of view of an edge detection algorithm. As strange as this sounds, it is a common visual trope in science fiction or war films to show an audience point-of-view shots of heads-up displays, wireframes, thermal imaging, cross hairs, ray-tracings, overlays, and other visual tricks that separate and identify figures from a background, augmenting reality by dividing the world, and our attention, neatly into shapes. So, while I subvert the practical intention of a tool like ImageJ, I do so in order to simultaneously create new algorithmic knowledge about media texts as well as to test new modes of using supposedly neutral and objective computational image analysis.

How does this scientific visualization of the world fit with narrative cinema’s compulsive coupling of humans? For example, when the Terminator visually assesses his surroundings, ultimately deciding on one tough biker to subject to a more thorough algorithmic striptease, he does so not to understand or analyze his new environment, but rather to find the best body whose clothes he can inhabit. Similarly, the agents in Enemy of the State triangulate Gene Hackman’s identity through his outlines, reproducing his likeness onscreen to bring the recluse closer to them. And in A Scanner Darkly, an animated film made by tracing outlines of live actors, Keanu’s scramble suit cycles through a motley variety of looks to obscure the real person beneath, finally revealed by a dolly shot that takes us past a network of circuits inside the suit. Strong lines and outlines become interfaces for characters to relate to one another, mediating our knowledge of those outside our skin, helpfully marking the limen where we end and others begin.

Film philosophers have long considered the question “What do three dimensional shapes look like in two dimensions?” One of the most in-depth answers to this question was provided by Rudolf Arnheim, who uses gestalt theory to explain the possibility of meaning-making in early cinema. Thus cinemagoers are able to make sense of a man raising a newspaper in front of his face not by thinking that his jaw has been horrifically displaced, but by understanding that the paper’s sharp line is on a different three-dimensional plane from the man, the jaw only visually cut off by the newspaper when seen from this particular two-dimensional perspective. In cinema, then, we must understand edges working at a double level: as a way of framing three-dimensional space for a flat two-dimensional screen and as an aesthetic line that can be read for meaning in its own right.

In contrast to the antiseptic, military-scientific computational point of view of a film like The Terminator, the shot from Crouching Tiger, Hidden Dragon, where Mu Bai presses Shu Lien’s hand against his cheek, is for me one of the most romantic images of world cinema. The barest consummation of their lifelong love, the smallest touch here activates a larger passion. We see how small gestures have great meaning; their unwillingness or inability to express their feelings makes the tiniest of contact highly charged, even as they discuss the impermanence of the physical world. The space between lovers is elevated to an excruciating degree in The Lover, although there the relationship is further complicated by racial, class, social, national, and age dimensions. Seen as a series of lines inching closer and closer, the lovers’ edge-detected hands highlight the diminishing physical space between themselves, and when they connect the lines separating them are joined in one shape.

These images have a unique aesthetic quality different from the original. What beauty is there in the foregrounding of edges? How can a unique affective response come from otherwise scientific analysis? A simplifying, a distillation, a reduction of the messiness of physical experience to the image: the effects of edge detection experimentation are a surprise to me and others. Are these edge algorithms capable of drawing out an aspect of narrative films otherwise hidden from human vision? Or does the increasing presence of these algorithms in social media, digital photography, and smartphone editing applications rather suggest that we are becoming more effectively drawn towards computer vision, being trained to think visually in algorithmic terms?

Criteria

As described further in the methods section below, I attempted a scholarly project in a poetical, rather than explanatory, vein. One useful criteria, then, might be the extent to which the work demonstrates and activates new knowledge in the viewer’s mind without recourse to descriptive explanation.

Context

This work extends from my earlier scholarly videographic work that uses ImageJ to analyse film and media texts. This work itself was inspired by a tradition of surrealist methods of criticism that intentionally deform texts in order to perform an alternative reading strategy, usually one opposed to narrative. Likewise, I ask, what does it mean for a human to watch a film from the perspective of a computer, or of a computer vision algorithm? While there has been some digital humanities work on moving image texts, much of this has been informed by traditional narratological concerns like average shot length (cf. Salt). Instead, I wanted to manipulate corpora of film texts to create what Franco Moretti might consider a “distant viewing.” I am also inspired by fine artists such as photographer Hiroshi Sugimoto’s Theaters, a series of long-exposure photographs of movie theaters lit by projector light, and computational artists such as Jason Salavon and Jim Campbell, who independently worked with averaging processes in the late 1990s and 2000s.

From another direction, my work builds on a community of scientific and medical practitioners who work with ImageJ’s open architecture to create and share java-based plugins. The open source community around this software exemplifies a newer kind of humanities practice that values transparency, collaboration, reproducibility, and open access.

Methodologically, however, Johanna Drucker has demonstrated that there is a tendency for these kinds of digital tools used in information visualization to primarily reflect “positivistic, strictly quantitative, mechanistic, reductive and literal” values which “preclude humanistic methods from their operations because of the very assumptions on which they are designed: that objects of knowledge can be understood as self-identical, self-evident, ahistorical, and autonomous” (86). From this tradition of digital humanities, I pose unasked questions of media texts and question the self-evident function of scientific imaging tools, particularly the differences between varieties of edge detection algorithms. Do edge detection algorithms work differently on men versus women? What about people of color?

This current work is a natural extension of my personal research trajectory. Inspired by Lev Manovich’s Software Studies Initiative, my initial work used ImageJ to create summed images of films, compressing their frames into a single color field (http://www.digitalhumanities.org/dhq/vol/11/1/000276/000276.html). I next experimented with sequentially summing films, where successive intervals of a few seconds are summed and then played so that in each summed frame one frame drops away and one is added (https://vimeo.com/172398889). The ghostly outlines of characters and camera movements next led me to pursue this current application of various edge detection algorithms to narrative cinema, first at an academic conference (https://vimeo.com/208869731) and now more extensively in “Where You End and I Begin.”

Methods

My methods derive from videographic criticism, scientific image analysis, and a deformative tradition of cultural analysis.

My videographic practice is informed by Christian Keathley’s distinction between “analytical and explanatory” work and “poetical and expressive” work, where “if the goal is still the production of some knowledge, the challenge for the ‘digital film critic’ is to situate herself somewhere in the middle of these alternatives, borrowing the explanatory authority of one and the poetical power of the other.” While videographic criticism as it has been popularized tends towards the essayistic (with explanatory voiceover, scholarly quotation, or blocks of explanatory text), my work uses juxtaposition and overlap in a poetical way to generate knowledge. In part 1 we see contemporary cinema’s interest in visualizing vision, these are mirrored in part 2’s demonstration of various edge detection algorithms.

This may seem at odds with my extensive use of the scientific image analysis software ImageJ, with its rationalist, positivist focus on clarifying and displaying microscopic and hidden natural structures. However, precisely because of this scientific tool’s seeming neutrality and inappropriateness for studying narrative film, ImageJ becomes a work station for a new digitally-informed media studies that productively questions itself and implicit assumptions about appropriate methods of analysis.

For me, this method takes the form of what Lisa Samuels and Jerome McGann call “deformance,” which argues against criticism as solely a way to sniff out an encoded, critical interpretation that gives a work value and instead argues that “all interpretation is a function of the poem’s systemic intelligibility” (italicized in original). Thus, my critical method sets aside the notion of applying an interpretation to a text and instead works to discover a text’s inherent systems by performing actions on it that render it strange or surreal: deforming a text as a way of performing its structure. This is explicitly so in “Where You End and I Begin”’s examination of edges.

Outcomes

This work can be useful to a number of other practitioners in diverse interdisciplinary fields: videographic scholars might take the deformative example as model to productively manipulate and defamiliarize moving image texts often considered sacrosanct, media studies scholars might consider new ways to examine the haptic potentialities of both traditional cinema as well as virtual reality and other immersive media environments, and performance art scholars might apply similar methods to studying movement in their field.

Dissemination

This work was self-funded and extends from an earlier presentation on edge detection algorithms given at a scholarly conference, SCMS 2017. While I described the basic methods there, this work is entirely new and has not been shown outside of the Vimeo link.

References

Rudolf Arnheim, Film as Art (Berkley: U of California Press, 1957).

Johanna Drucker, “Humanistic Theory and Digital Scholarship,” in Debates in the Digital Humanities, ed. Matthew K. Gold, 85-95 (Minneapolis: University of Minnesota Press, 2012).

Christian Keathley, “La Caméra-stylo: Notes on Video Criticism and Cinephilia,” in The Language and Style of Film Criticism, eds. Andrew Klevan and Alex Clayton, 176–191 (New York: Routledge, 2011).

Kevin L. Ferguson, “Digital Surrealism: Visualizing Walt Disney Animation Studios,” Digital Humanities Quarterly 11.1 (2017). http://www.digitalhumanities.org/dhq/vol/11/1/000276/000276.html

———, “The Slices of Cinema: Digital Surrealism as Research Strategy,” The Arclight Guide to Media History and the Digital Humanities, eds. Charles R. Acland and Eric Hoyt. (Falmer: REFRAME/Project Arclight, 2016): 270–299. http://projectarclight.org/book/

———,“Volumetric Cinema,” [in]Transition: Journal of Videographic Film and Moving Image Studies 2.1 (2015), 20 minutes. http://mediacommons.futureofthebook.org/intransition/2015/03/10/volumetric-cinema

Lev Manovich, Software Studies Initiative, http://lab.softwarestudies.com/. Accessed November 1, 2017.

Franco Moretti, Graphs, Maps, Trees: Abstract Models for a Literary History, Verso, London (2005).

Barry Salt, Film Style and Technology: History and Analysis (London: Starword, 2009).

Lisa Samuels and Jerome McGann, “Deformance and Interpretation,” New Literary History 30, no. 1 (1999): 25–56.

A.I. Artificial Intelligence (dir. Steven Spielberg, 2001)

Avatar, (dir. James Cameron, 2009)

Cliffhanger (dir. Renny Harlin, 1993)

Crouching Tiger, Hidden Dragon (dir. Ang Lee, 2000)

Disneyland Story, Robert Florey (dir. Wilfred Jackson, 1954)

Enemy of the State (dir. Tony Scott, 1998)

The Empire Strikes Back (dir. Irvin Kershner, 1980)

E.T. the Extra-Terrestrial (dir. Steven Spielberg, 1982)

Fight Club (dir. David Fincher, 1999)

Frozen (dir. Chris Buck, Jennifer Lee, 2013)

Ghost in the Shell 2: Innocence (dir. Mamoru Oshii, 2004)

Hardware (dir. Richard Stanley, 1990)

The Lover (dir. Jean-Jacques Annaud, 1992)

Lucy (dir. Luc Besson, 2014)

Le mystère Picasso (dir. Henri-Georges Clouzot, 1956)

Only God Forgives (dir. Nicolas Winding Refn, 2013)

Prometheus (dir. Ridley Scott, 2012)

A Scanner Darkly (dir. Richard Linklater, 2006)

Star Trek: The Wrath of Khan (dir. Nicholas Meyer, 1982)

“Take on Me,” a-ha (dir. Steve Barron, 1985)

Terminator 2: Judgment Day (dir. James Cameron, 1991)

The Thirteenth Floor (dir. Josef Rusnak, 1999)

Total Recall (dir. Paul Verhoeven, 1990)

Tron (dir. Steven Lisberger, 1982)

Variety (dir. Bette Gordon, 1983)

Peer Reviews

All reviews refer to original research statements which have been edited in response to what follows:

Review 1: Accept submission subject to minor revisions of written statement.

The strength of this work lies in its poetry. There is great elegance and confidence in the choices made and an intellectual underpinning that shines through. Yearning and melancholy emerge from the film both in unison with, and sometimes in spite of, the source films being analysed. Lovers from Crouching Tiger Hidden Dragon all but disappear in one scene, their camouflaged outlines grasping for purchase on a deconstructed, diegetic world, their frailty nonetheless defied by Michelle Yeoh’s character’s proclamation that “Not everything is an illusion”. Yet the pathos of this moment is surely the idea that even the illusion in this case is on the cusp of vanishing.

The structuring of the work in two halves is another of its strengths, creating the sense that edge-detection (or similar) is already used for narrative and aesthetic purposes in the films being analysed, causing us to wonder about the relationship between the author’s own deployment of edge-detection techniques and those of the original texts. There is a hint of auto-criticism that emerges from this choice (maybe those films are doing some of what this film is doing), but it also increases no end the dialectical play and instability of the ‘analysis’ and the poetic expression of the film. Upon first viewing one is very likely to not know which visual effects are native to a film and which ones have been introduced. Again, it is the choice to operate in precisely this liminal and uncertain realm, and in fact to really master it, that makes this film compelling.

Not all films lend themselves to the research context. I think this film does as long as we allow for the author’s claim that it is scholarship in a ‘poetic’, rather than ‘explanatory’, vein. I agree that the film excels as a kind of poetic thinking on the screen. Whether or not one allows for the idea of ‘scholarship’, itself, in a poetic mode is a slightly different matter. Certainly the author has adopted a fairly light touch in terms of the use of scholarhsip, choosing not to over-egg the submission by insisting on its theoretical weight, for instance, or yoking it to certain discourses unnecessarily. Perhaps one way of filling out the written work a little more might be to reflect on its situated-ness vis-à-vis the academy, on its poetic self-sameness for instance (i.e. its self-evident qualities), or on its status as a video essay.

To undertake a bit more of this work of situating the production would allow for more nuanced and developed reflections than the following: “I intentionally avoid overtly analytical techniques such as voiceover, scholarly quotation, or extended blocks of explanatory text, instead using juxtaposition and overlap in a poetical way to generate knowledge.” While I find this sort of statement clear, I feel that on occasion false arguments or dichotomies are raised as a technique for making arguments that don’t go very far.

At one point the author says that meaning is not just demonstrated by the video practice, but is in fact generated. It is good to generate, rather than just demonstrate, meaning, but this is a bit obvious, and the film is so much stronger than this binary allows for. Indeed, in the next section this discussion becomes more nuanced and engaging, culminating in the idea of ‘deformance’.

Similarly, the following relies on an idea of ‘unstated assumptions’ as a way of bolstering the author’s avowedly more critical approach to the software: “So, while I subvert the practical intention of a tool like ImageJ, I do so in order to simultaneously create new algorithmic knowledge about media texts as well as to question the unstated assumptions of supposedly neutral computational image analysis.” In the context in which this wonderful work is being reviewed, I feel that one would not anticipate ‘unstated assumptions’ being made by the artist-scholar about technology. What artist would? My complaint here is only that sometimes the rhetorical force of the writing falls short.

Does the Terminator really “colonize his identity” in the scene in T2 when Arnie takes the biker’s leathers? Whereas the metaphorical ‘striptease’ works as an evocative metaphor, perhaps the colonization spoken of is no more than a metaphor, and if it is, perhaps it should not be there. Indeed the triplet that follows is more confusing than it is poetic, when you say that Arnie’s character does all of the following: “inhabit the man, to wear his clothes and colonize his identity”. Which is it? Perhaps there is room to make an explicit connection between inhabitation and colonization as outcomes of the film’s aesthetic of liminality, but this connection should be made in a less gestural manner. How does the Terminator scene involve a ‘coupling’ of humans?

Although this is a film that works in the gaps between explanation and expression, between bodies and representation, the metaphors articulated in writing don’t quite match the elegance of the film and could be nuanced or developed through minor revisions. Whereas the film communicates delicately through gestural forms of expression, the writing needs to work harder to marry these to statements about how the film achieves what it achieves.

Final word to the lovely moment at 8 minutes where Chow Yun-Fat’s character’s face cycles through a series of different algorithmic treatments after he finally makes a physical connection with his love. This cycling of visual styles suggests the character’s very subjectivity is fracturing, or flashing back or multiplying before our eyes, as though there is no longer a single vessel for him to inhabit, only a swirl of mystery or even dissolution. Although the author says the romantic gesture of hands touching in Crouching Tiger Hidden Dragon is one of the most romantic s/he has seen, there is a fantastical and mysterious new gesture to be found in the algorithmic treatment of that gesture. This kind of elaboration of cinematic tropes and gestures is what makes the submitted film so delightful.

Review 2: Accept submission for publication with no amendments.

As a criteria for review of Where You End and I Begin the author suggests, the critical lens “might be the extent to which the work demonstrates and activates new knowledge in the viewer’s mind without recourse to descriptive explanation.” So that is how I shall approach this review, on the basis of my discovery of new knowledge, whilst parsing through the poetical exploration of a future vision of edges – how computers might see, and possibly some day come to poetically understand, narrative cinematic scenes.

Where You End and I Begin presents a poetic exploration of edges and liminal spaces represented in film through a juxtaposition of standard science fiction interfaces in film such as heads-up displays and robot vision, alongside interpretive and deformative explorations of the edges and linina of deeply human narrative film sequences. The author clearly argues, through both the scholarly videographic work and written submission, for new ways of reading and understanding film and media texts through the use of algorithms and scientific visualization software.

In “part i: where i end” of this scholarly videographic work, the author reveals filmic scenes of futuristic interfaces -edges and outlines – and other tropes that have been used by filmmakers to imagine how a computer might see the environment and people within it. In “part ii: and you begin,” the author utilizes the critical practice of deformation, as theorized by McGann and Samuels. Here, the poetry is made evident in this very juxtaposition of human-seeing-as-computer as articulated by many of the film directors in the sci-fi heavy “part i”, while the author uses computer software to poetically visualize how a computer parses human touch and interaction through their edges, especially in the very human film scenes selected in both The Lover and Crouching Tiger, Hidden Dragon.

This submission presents an original approach to scholarly videographic work in its computer software-aided interpretive, rather than explanatory, approach. The use of scientific visualization software ImageJ is used for its edge detection capabilities to emphasize the liminal spaces often unseen in cinematic imagery and the filmic frame. In this video essay, whilst the computer vision imagery in “part i where i end” relies predominantly on sequences from well-known science fiction films, the software deformation applied to the film frames in “part ii and you begin” poetically applies algorithmic edge detection to the most humanistic of film frames.

As the author asserts, “Strong lines and outlines become interfaces for characters to relate to one another, mediating our knowledge of those outside our skin, helpfully marking the limen where we end and others begin.” However, in the film scenes used in this video essay to demonstrate the use of outlines for characters to understand one another, the “other” in many of these scenes are not completely human characters, but rather machines, in the case of the T-800 in Terminator, or a facial scanning algorithm to aid humans in understanding the other, in the case of Enemy of the State.

Following on from this, does the use of ImageJ to create outlines of characters in dramatic narrative films begin to imagine a near future interface for which computers might relate to humans through an understanding of their edges and liminal spaces? Might an edge detection analysis of this kind help computers make sense of a touch between lovers or the activation of deep passion? In the case of the scene from Crouching Tiger, Hidden Dragon, the use by the author of several algorithmic filters within ImageJ in succession, at the most intimate of moments between the characters, allows us to imagine that the computer vision/edge detection software cannot compute the human tenderness that is taking place in the scene, as if it is looking for the right edges to help it understand. But as stated from the beginning, the reviewer is assuming poetical new knowledge through the software-enabled deformation in this critical video essay

Perhaps viewers are becoming trained to think visually in more algorithmic terms, as the author suggests. That makes this work worth further exploration into how both we as humans – and computers – might understand the potential new knowledge of narrative film and filmmaking brought about by computer vision, artificial intelligence and increasingly human algorithms.

All reviews refer to original research statements which have been edited in response.