User manual | VOL NO , Live visuals have become a pervasive component of our

VOL 19 NO 3 VOLUME EDITORS LANFRANCO ACETI, STEVE GIBSON &
STEFAN MÜLLER ARISONA EDITOR ÖZDEN ŞAHİN
Live visuals have become a pervasive component of our contemporary
lives; either as visible interfaces that re-connect citizens and buildings
overlaying new contextual meaning or as invisible ubiquitous narratives
that are discovered through interactive actions and mediating screens.
The contemporary re-design of the environment we live in is in terms of
visuals and visualizations, software interfaces and new modes of
engagement and consumption. This LEA volume presents a series of
seminal papers in the field, offering the reader a new perspective on the
future role of Live Visuals.
LIVE VISUALS
LEA is a publication of Leonardo/ISAST.
Editorial Address
Leonardo Electronic Almanac
Copyright 2013 ISAST
Sabanci University, Orhanli – Tuzla, 34956
Leonardo Electronic Almanac
Istanbul, Turkey
Volume 19 Issue 3
July 15, 2013
Email
ISSN 1071-4391
[email protected]
ISBN 978-1-906897-22-2
The ISBN is provided by Goldsmiths, University of London.
Leonardo Electronic Almanac, Volume 19 Issue 3
Web
» www.leoalmanac.org
lea publishing & subscription information
» www.twitter.com/LEA_twitts
» www.flickr.com/photos/lea_gallery
Editor in Chief
Lanfranco Aceti [email protected]
Almanac/209156896252
volume Editors
Lanfranco Aceti, Steve Gibson & Stefan Müller Arisona
Copyright © 2013
Editor
Özden Şahin
» www.facebook.com/pages/Leonardo-Electronic-
Co-Editor
Özden Şahin [email protected]
Leonardo, the International Society for the Arts,
Managing Editor
Live Visuals
Sciences and Technology
John Francescutti [email protected]
Leonardo Electronic Almanac is published by:
Art Director
Leonardo/ISAST
Deniz Cem Önduygu [email protected]
211 Sutter Street, suite 501
San Francisco, CA 94108
Editorial Board
USA
Peter J. Bentley, Ezequiel Di Paolo, Ernest Edmonds, Felice
Leonardo Electronic Almanac (LEA) is a project of Leonardo/
Frankel, Gabriella Giannachi, Gary Hall, Craig Harris, Sibel Irzık,
The International Society for the Arts, Sciences and Technol-
Marina Jirotka, Beau Lotto, Roger Malina, Terrence Masson,
ogy. For more information about Leonardo/ISAST’s publica-
Jon McCormack, Mark Nash, Sally Jane Norman, Christiane
tions and programs, see http://www.leonardo.info or contact
Paul, Simon Penny, Jane Prophet, Jeffrey Shaw, William
[email protected]
Uricchio
Leonardo Electronic Almanac is produced by
Cover Image
Passero Productions.
The Encounter, Elif Ayiter, 2010, Screenshot of Cinematic Play
Session in Second Life. © Elif Ayiter. Used with Permission.
Reposting of this journal is prohibited without permission of
Leonardo/ISAST, except for the posting of news and events
listings which have been independently received.
The individual articles included in the issue are © 2013 ISAST.
2
LEONARDOELECTRONICALMANAC VOL 19 NO 3
ISSN 1071-4391
ISB N 978 -1-9 0 6 897-2 2-2
ISSN 1071-4391
ISB N 978 -1-9 0 6 897-2 2-2
VOL 19 NO 3 LEONARDOELECTRONICALMANAC
3
The Leonardo Electronic Almanac
acknowledges the kind support
for this issue of
music and performing arts professions
Ron Sadoff, Director
Music Technology
Music Composition
B.m., m.m., ph.D.
B.m., m.m., ph.D.
Including a new 3-Summer M.M.
concert music, Jazz, film scoring,
electro-acoustic, songwriting
New York UNiversitY is aN affirmative actioN/eqUal opportUNitY iNstitUtioN.
immersive audio, computer music, informatics,
cognition, recording and production
4
LEONARDOELECTRONICALMANAC VOL 19 NO 4
3
IISSSSNN 110
07711--44339911
IISSBBNN 997788--11--990
066889977--2262 -- 0
2
• Study with a premier faculty who are active in the local and international music field,
including Juan pablo Bello, morwaread farbood, phil e. Galdston, paul Geluso,
tae Hong park, kenneth peacock, agnieszka roginska, robert rowe, s. alex ruthmann,
ronald sadoff, David schroeder, mark suozzo, and Julia wolfe
• work within a large and enriching university environment in the heart of New York city
• Have access to state-of-the-art facilities including the James l. Dolan music
recording studio, one of the most technologically advanced audio teaching facilities
in the United states
• Collaborate with an outstanding variety of department performance groups, along
with choreographers, visual artists, writers, filmmakers, and scholars in other fields
• take advantage of special courses offered abroad and during the summer
visit www.steinhardt.nyu.edu/music or call 212 998 5424 to learn more.
Job:
I S BA1310_06_MusicTech
N 978 -1-9 0 6 897-2 2-2
VOL 19 NO 3 LEONARDOELECTRONICALMANAC
Publication: LEA Journal
Size: 6.5” x 9.5” (no bleed) [177.8 mm x 254 mm trim]
ISSN 1071-4391
5
C O N T E N T S
C O N T E N T S
Leonardo Electronic Almanac
Volume 19 Issue 3
8
176
EDITORIAL Lanfranco Aceti
BACK TO THE CROSS-MODAL OBJECT: A LOOK BACK AT EARLY
AUDIOVISUAL PERFORMANCE THROUGH THE LENS OF OBJECTHOOD
Atau Tanaka
12
REVISITING CINEMA: EXPLORING THE EXHIBITIVE MERITS OF CINEMA
FROM NICKELODEON THEATRE TO IMMERSIVE
ARENAS OF TOMORROW Brian Herczog
190
STRUCTURED SPONTANEITY: RESPONSIVE ART MEETS CLASSICAL
MUSIC IN A COLLABORATIVE PERFORMANCE OF ANTONIO VIVALDI’S
FOUR SEASONS Yana (Ioanna) Sakellion & Yan Da
22
THE FUTURE OF CINEMA: FINDING NEW MEANING THROUGH
LIVE INTERACTION Dominic Smith
202
INTERACTIVE ANIMATION TECHNIQUES IN THE GENERATION AND
DOCUMENTATION OF SYSTEMS ART Paul Goodfellow
30
A FLEXIBLE APPROACH FOR SYNCHRONIZING VIDEO WITH LIVE MUSIC
214
SIMULATING SYNESTHESIA IN SPATIALLY-BASED REAL-TIME AUDIOVISUAL PERFORMANCE Steve Gibson
A ‘REAL TIME IMAGE CONDUCTOR’ OR A KIND OF CINEMA?: TOWARDS
LIVE VISUAL EFFECTS Peter Richardson
Don Ritter
46
AVATAR ACTORS Elif Ayiter
230
64
MULTI-PROJECTION FILMS, ALMOST-CINEMAS AND VJ REMIXES:
SPATIAL ARRANGEMENTS OF MOVING IMAGE PRESENCE Gabriel Menotti
240
78
88
108
MACHINES OF THE AUDIOVISUAL: THE DEVELOPMENT OF “SYNTHETIC
AUDIOVISUAL INTERFACES” IN THE AVANT-GARDE ART SINCE THE
1970S Jihoon Kim
256
272
134
AVVX: A VECTOR GRAPHICS TOOL FOR AUDIOVISUAL PERFORMANCES
284
306
ARCHITECTURAL PROJECTIONS: CHANGING THE PERCEPTION OF
ARCHITECTURE WITH LIGHT Lukas Treyer, Stefan Müller Arisona &
Gerhard Schmitt
IN DARWIN’S GARDEN: TEMPORALITY AND SENSE OF PLACE Vince
Dziekan, Chris Meigh-Andrews, Rowan Blaik & Alan Summers
LEONARDOELECTRONICALMANAC VOL 19 NO 3
ISSN 1071-4391
ISB N 978 -1-9 0 6 897-2 2-2
HOW AN AUDIO-VISUAL INSTRUMENT CAN FOSTER THE SONIC
EXPERIENCE Adriana Sa
GATHERING AUDIENCE FEEDBACK ON AN AUDIOVISUAL PERFORMANCE
Léon McCarthy
Nuno N. Correia
6
VISUALIZATION TECHNOLOGIES FOR MUSIC, DANCE, AND
STAGING IN OPERAS Guerino Mazzola, David Walsh, Lauren Butler,
Aleksey Polukeyev
TEXT-MODE AND THE LIVE PETSCII ANIMATIONS OF RAQUEL MEYERS:
FINDING NEW MEANING THROUGH LIVE INTERACTION Leonard J. Paul
OUTSOURCING THE VJ: COLLABORATIVE VISUALS USING THE
AUDIENCE’S SMARTPHONES Tyler Freeman
164
OF MINIMAL MATERIALITIES AND MAXIMAL AMPLITUDES: A
PROVISIONAL MANUAL OF STROBOSCOPIC NOISE PERFORMANCE Jamie
Allen
NEW PHOTOGRAPHY: A PERVERSE CONFUSION BETWEEN THE LIVE
AND THE REAL Kirk Woolford
124
148
LIVE AUDIO-VISUAL ART + FIRST NATIONS CULTURE
Jackson 2bears
ISSN 1071-4391
322
CHOREOTOPOLOGY: COMPLEX SPACE IN CHOREOGRAPHY WITH REALTIME VIDEO Kate Sicchio
336
CINEMATICS AND NARRATIVES: MOVIE AUTHORING & DESIGN FOCUSED
INTERACTION Mark Chavez & Yun-Ke Chang
352
IMPROVISING SYNESTHESIA: COMPROVISATION OF GENERATIVE
GRAPHICS AND MUSIC Joshua B. Mailman
ISB N 978 -1-9 0 6 897-2 2-2
VOL 19 NO 3 LEONARDOELECTRONICALMANAC
7
E D I T O R I A L
E D I T O R I A L
When Moving Images
Become Alive!
experience that digital and live visuals are rendering
of social ideals. The conflict is, therefore, not solely in
increasingly visible.
the elitist or participatory forms of consumption but
also in the ideologies that surround the cultural behav-
“Everything I said on the subject [the nature of aura]
“Look! It’s moving. It’s alive. It’s alive... It’s alive, it’s mov-
iors of visual consumption.
was directed polemically against the theosophists,
ing, it’s alive, it’s alive, it’s alive, it’s alive, IT’S ALIVE!”
whose inexperience and ignorance I find highly
Object in themselves, not just buildings, can and may
Frankenstein (1931)
repugnant. . . . First, genuine aura appears in all things,
soon carry live visuals. There is the expectation that
not just in certain kinds of things, as people imagine.”
Those who still see – and there are many in this
volume “Architectural Projections” by Lukas Treyer,
camp – visuals as simple ‘decorations’ are living in
Stefan Müller Arisona & Gerhard Schmitt).
a late 19th century understanding of media, with
2
and should project the label and its textured images
The importance of digital media is undeniably evident.
to the viewer. People increasingly expect the object
Within this media context of multiple screens and sur-
to engage with their needs by providing the necessary
information that would convince them to look into
no realization that an immense cultural shift has hap-
Images appear over the architecture of the buildings
faces the digitized image, in a culture profoundly visual,
pened in the late 20th century when big data, sensors,
as another structural layer, one made of information
has extended its dominion through ‘disruptive forms’
it, play with it, engage with it, talk to it, like it and ulti-
algorithms and visuals merged in order to create 21st
data that relays more to the viewer either directly or
of sharing and ‘illegal’ consumption. The reproducibili-
mately buy it.
century constantly mediated social-visual culture.
through screens able to read augmented reality infor-
ty of the image (or the live visuals) – pushed to its very
mation. But live visuals relay more than images, they
limit – has an anarchistic and revolutionary element
Ultimately there will be no need to engage in this
Although the visuals are not actually alive, one cannot
are also linked to sound and the analysis of this link-
when considered from the neocapitalistic perspective
process but the environment will have objects that,
fail to grasp the fascination or evolution that visuals
age provides us with the opportunity “to think about
imbued in corporative and hierarchical forms of the
by reading previous experiences of likes and dislikes,
and visual data have embarked upon. It is no longer
the different ways in which linkages between vision
construction of values. On the contrary, the reproduc-
present a personalized visual texture of reality.
possible to see the relationship of the visual as lim-
and audition can be established, and how audio-visual
ibility of the image when analyzed from a Marxist point
ited to the space of the traditional screens in the film
objects can be composed from the specific attributes
of view possesses a community and social component
Live visuals will provide an environment within which
theater or at home in the living room with the TV. The
of auditory and visual perception” (see “Back to the
for egalitarian participation within the richness of con-
purchasing does not mean to solely acquire an object
mobility of contemporary visuals and contemporary
Cross-modal Object” by Atau Tanaka).
temporary and historical cultural forms.
but rather to ‘buy’ into an idea, a history, an ideology
‘embeddedness’ of visuals onto and into things is a
iPads and iPhones – followed by a generation of
The digital live visuals – with their continuous potential
daily practice. The viewers have acquired expecta-
smarter and smarter devices – have brought a radi-
of integration within the blurring boundaries of public
re-defines one’s experience of the real based on previ-
tions that it is possible, or that it should be possible,
cal change in the way reality is experienced, captured,
and private environments – will continue to be the
ously expressed likes and dislikes.
to recall the image of an object and to be able to have
uploaded and shared. These processes allow reality
conflicting territory of divergent interests and cultural
that same object appear at home at will. The process
to be experienced with multiple added layers, allow-
assumptions that will shape the future of societal en-
In this context of multiple object and environmental
of downloading should not be limited to ‘immaterial’
ing viewers to re-capture, re-upload and re-share,
gagements. Reproducibility will increasingly become
experiences it is also possible to forge multiple individ-
digital data, but should be transferred to 3D physical
creating yet further layers over the previous layers
the territory of control generating conflicts between
ualized experiences of the real; as much as there are
screens has pushed boundaries – so much so that
objects.
8
one no longer has to read a label – but the object can
1
or a socio-political lifestyle. It is a process of increased
visualization of large data (Big Data) that defines and
that were already placed upon the ‘original.’ This lay-
original and copy, and between the layering of copy
multiple personalized experiences of the internet and
ering process, this thickening of meanings, adding of
and copies, in the attempt to contain ideal participa-
social media through multiple avatar identities (see
Images are projected onto buildings – not as the tra-
interpretations, references and even errors, may be
tory models of democracy. The elitist interpretation of
“Avatar Actors” by Elif Ayter). The ‘real’ will become
ditional trompe l’oeil placed to disguise and trick the
considered as the physical process that leads to the
the aura will continue to be juxtaposed with models of
a visual timeline of what the algorithm has decided
eye – but as an architectural element of the building
manifestation of the ‘aura’ as a metaphysical concept.
Marxist participation and appropriation.
itself; so much so that there are arguments, including
The materiality of the virtual, layered upon the ‘real,’
mine, that we should substitute walls with projected
becomes an indication of the compositing of the
Live visuals projected on public buildings and private
information data, which should also have and be
aura, in Walter Benjamin’s terms, as a metaphysical
areas do not escape this conflict, but present interpre-
perceived as having material properties (see in this
experience of the object/image but nevertheless an
tations and forms of engagements that are reflections
LEONARDOELECTRONICALMANAC VOL 19 NO 3
ISSN 1071-4391
ISB N 978 -1-9 0 6 897-2 2-2
ISSN 1071-4391
3
ISB N 978 -1-9 0 6 897-2 2-2
should be offered based on individualized settings of
likes and dislikes. This approach raises an infinite set
of possibilities but of problems as well.
VOL 19 NO 3 LEONARDOELECTRONICALMANAC
9
E D I T O R I A L
E D I T O R I A L
The life of our representation and of our visuals is
experience (see “Simulating Synesthesia in Spatially-
It is this control of the environment around us and us
our ‘real’ life – disjointed and increasingly distant from
Based Real-time Audio-Visual Performance” by Ste-
within that environment that will increasingly define
patience in complying with the guidelines and editorial
what we continue to perceive as the ‘real real,’ delu-
phen Gibson).
the role that live visuals will play in negotiating real
demands that made this issue one that I am particu-
and virtual experiences. The conflict will arise from
larly proud of, both for its visuals and for its content.
sively hanging on to outdated but comfortable modes
of perception.
If this fantasy of the images of society is considered
the blurred lines of the definition of self and other;
an illusion – or the reality of the simulacrum, which
whether the ‘other’ will be another individual or a cor-
My special thanks go to Deniz Cem Önduygu who has
The cinematic visions of live visuals from the 19th
is a textual oxymoron at prima facie – it will be de-
poration.
shown commitment to the LEA project beyond what
century have become true and have re-designed
termined through the experience of the live visuals
society unexpectedly, altering dramatically the social
becoming alive.
could be expected.
The potential problems of this state of the live visu-
structures and speeding up the pace of our physical
als within a real/virtual conflict will be discovered as
Özden Şahin has, as always, continued to provide
existence that constantly tries to catch up and play
Nevertheless, stating that people have illusory per-
time moves on. In the end this is a giant behavioral
valuable editorial support to ensure that LEA could
achieve another landmark.
up to the visual virtual realities that we spend time
ceptions of themselves in relation to a ‘real’ self and
experiment, where media and their influences are not
constructing.
to the ‘real’ perception of them that others have only
analyzed for their social impact ex ante facto; this is
reinforces the idea that Live Visuals will allow people
something that happens ex post facto.
If we still hold to this dualistic and dichotomist ap-
to manifest their multiple perceptions, as simulated
Lanfranco Aceti
Editor in Chief, Leonardo Electronic Almanac
proach of real versus virtual (although the virtual has
and/or real will no long matter. These multiple per-
Nevertheless, in this ex post facto society there are
been real for some time and has become one of the
ceptions will create multiple ever-changing personae
some scholars that try to understand and eviscerate
multiple facets of the ‘real’ experience), then the real
that will be further layered through the engagements
the problems related to the process of visuals becom-
is increasingly slowing down while the virtual repre-
with the multiple visual environments and the people/
ing alive. This issue collects the analyses of some of
sentation of visuals is accelerating the creation of a
avatars that populate those environments, both real
these scholars and embeds them in a larger societal
world of instantaneous connectivity, desires and aspi-
and virtual.
debate, hinting at future developments and problems
new extreme perception of consumer culture where the
that society and images will have to face as the live
object seen can be bought and automatically printed at
visuals become more and more alive.
home or in the office. Matt Ratto and Robert Ree, “Mate-
rations. A visuality of hyper-mediated images that, as
pollution, pervades and conditions our vision without
giving the option of switching off increasingly ‘alive’
live visuals.
4
The lack of ‘real’ in Jean Baudrillard’s understanding
In the end, these fantasies of identities and of worlds,
manifested through illusory identities and worlds
Director, Kasa Gallery
1. 3D printing the new phenomenon will soon collide with a
rializing Information: 3D Printing and Social Change,” First
within virtual contexts, are part of the reality with
The contemporary concerns and practices of live visu-
Monday 17, no. 7 (July 2, 2012),
which people engage. Although fantastic and illusory,
als are crystallized in this volume, providing an insight
http://firstmonday.org/ojs/index.php/fm/article/
these worlds are a reflection of a partial reality of the
into current developments and practices in the field of
view/3968/3273 (accessed October 20, 2013).
live visuals.
is speeding up the disappearance of the ‘real’ self in
identity of the creators and users. It is impossible for
favor of multiple personal existential narratives that
these worlds and identities to exist outside of the
are embedded in a series of multiple possible worlds.
‘real.’ This concept of real is made of negotiated and
This issue features a new logo on its cover, that of
It is not just the map that is disappearing in the pre-
negotiable frameworks of engagement that are in a
New York University, Steinhardt School of Culture,
cession of simulacra – but the body as well – as the
constant process of evolution and change.
Education, and Human Development.
2. Walter Benjamin, “Protocols of Drug Experiments,” in
On Hashish, ed. Howard Eiland (Cambridge, MA: Harvard
body is conceived in terms of visual representation:
University Press, 2006), 58.
3. “ The point here is not to issue a verdict in the debate
between Adorno and Benjamin, but rather to understand
the debate between them as representing two sides of
as a map. These multiple worlds of representations
The end of post-modernity and relativism may lead
My thanks to Prof. Robert Rowe, Professor of Music
an ongoing dialectical contradiction.” Ryan Moore, “Digital
contribute to create reality as the ‘fantasy’ we really
to the virtuality of truism: the representation of
and Music Education; Associate Dean of Research and
Reproducibility and the Culture Industry: Popular Music
wish to experience, reshaping in turn the ‘real’ identity
ourselves in as many multiple versions – already we
Doctoral Studies at NYU, for his work in establishing
and the Adorno-Benjamin Debate,” Fast Capitalism 9, no.
that continuously attempts to live up to its ‘virtual and
have multiple and concurrent digital lives – within the
this collaboration with LEA.
1 (2012), http://www.uta.edu/huma/agger/fastcapital-
fantastic’ expectations. Stephen Gibson presents the
world/s – ideological or corporate – that we will de-
reader with a description of one of these worlds with
cide or be forced to ‘buy into.’
live audio-visual simulations that create a synesthetic
10
possible. I also have to thank the authors for their
LEONARDOELECTRONICALMANAC VOL 19 NO 3
ISSN 1071-4391
ISB N 978 -1-9 0 6 897-2 2-2
ism/9_1/moore9_1.html (accessed October 30, 2013).
My gratitude to Steve Gibson and Stefan Müller Ari-
4. Paul Virilio, Open Sky, trans. Julie Rose (London: Verso,
sona, without them this volume would not have been
1997), 97.
ISSN 1071-4391
VOL 19 NO 3 LEONARDOELECTRONICALMANAC
ISB N 978 -1-9 0 6 897-2 2-2
11
A R T I C L E
A R T I C L E
A B S T R A C T
A ‘REAL TIME IMAGE
CONDUCTOR’ OR A
KIND OF CINEMA?
Towards Live Visual Effects
In this paper I describe a project that investigated methods for the incorporation
of filmic visual effects (VFX) into artworks and performed environments. VFX are
the computer-generated processes used in the film industry for manipulating live
action and animated elements.
Traditionally moving image visuals in a performative / gallery / club context
have been experienced as playback mediums, in which material is fixed in time
and played from beginning to end. Real-time visuals require the intervention of a
performer or a user to ‘cut up’ images live.
Since 2005 1 a number of film makers have moved away from narrative cin-
ema towards ‘live cinema’: remixing their films for audiences as a live performed
experience. This raises interesting possibilities to extend the genre within a performative art based approach.
Few filmmakers or VJs have incorporated ‘live’ visual effects as part of this
cinematic experience. It is the tension between remixing and creating images and
live visual effects that I identify as a key area for debate. Using the live cinema
works of Peter Greenaway and Mike Figgis I investigate how ‘live’ this cinema
really is or could ever hope to be. To further contextualize the possibilities for live
visual effects I describe and analyze: Miss Donnithorne’s Maggot: A performance /
interactive film of Sir Peter Maxwell Davis’s work of music theatre.
Sir Peter Maxwell Davies’s Miss Donnithorne’s Maggot
performed live by Alison Wells (mezzo soprano), Gemini (8
piece ensemble), Matt Cameron (visual effects) and Peter
Richardson (film maker). Screened and performed at the St
Magnus Festival in June 2012.
© Peter Richardson, 2012. Used with permission.
CINEMATOGRAPHIC LINEARITY
by
Peter Ric hardson
Professor of Film, School of Creative Arts, University
230
LEONARDOELECTRONICALMANAC VOL 19 NO 3
Since the early 20th century filmmakers have em-
narrative (novelistic) films then prevailed. From the
ployed linear, ‘novelistic’ methods to tell complex
late fifties ‘experimental’ filmmakers have sought to
stories to audiences seated in film theatres. Early
break out of traditional narrative structures and have
technical developments in film, notably those of the
proposed and demonstrated alternative cinematic
2
Of Hertfordshire, Hatfield U.K
Lumiere Brothers, who created immersive spectacles methods, often siting their work in the gallery as op-
Director, The Visual Effects Research Lab, University
and Georges Melies, who took a more narrative led
posed to the cinema. In his 1970 book Expanded Cine-
Of Dundee, Dundee U.K
approach to the new medium, occurred in Europe
ma
[email protected]
in the mid 1890s. By the early 1900s a nascent Eu-
television and described their forms as elements of a
ISSN 1071-4391
ISB N 978 -1-9 0 6 897-2 2-2
3 Gene Youngblood studied the tropes of film and
ropean studio system emerged in Denmark with the
‘closed system.’ Youngblood went on to study emer-
company Nordisk (1906) at its epicenter. The onset
gent experimental genres and the methods of artists
of the second world War (1914) saw the focus of this
who use video and film as their primary medium us-
young industry move to the wealthier and as yet non-
ing the lens of TV and cinema. The form has since
combatant United States of America. It was here that
become generally termed ‘media art.’ It is the term
the recognized mainstream ‘system’ of making films
‘media art’ that I use as the point of departure for this
emerged. The Hollywood method of making structural
paper. The works and methods examined here should
ISSN 1071-4391
ISB N 978 -1-9 0 6 897-2 2-2
VOL 19 NO 3 LEONARDOELECTRONICALMANAC
231
A R T I C L E
A R T I C L E
be considered as expanded cinematic practice and
mini-DV and ‘prosumer’ HD by both directors has led
VJ system consisting of a large plasma touchscreen,
not media art. Notions of expanded practices in film,
to innovative projects and outcomes that have bro-
developed by technical partner BeamSystems. Utilizing
London
it may be argued, made little impact on mainstream
ken the traditional boundaries between mainstream
this system, Greenaway projected the 92 Tulse Luper
tween experiencing his film Timecode (which had
audiences appealing more to ‘art house’ cinema-goers
cinema, art house and the avant garde. Both have and
stories on to 12 screens at Club 11 mixing the images
been released in 2000) in its cinema version and as
initially and then finding their audiences in the gallery
continue to ‘perform’ live versions of their works in
‘live.’ Greenaway then took the live performance on an
a new live performance (performed 10 years later),
6 Mike Figgis spoke of the difference be-
rather than the cinema. From the early 1970s filmmak-
the form of a VJ set: Tulse Luper (Greenaway) and
international tour NoTV Peter Greenaway Tulse Luper
claiming that dramatic irony could be increased in
ers such as Malcolm Le Grice (UK) and Paul Jeffrey
Timecode Live Remix (Figgis). These performances
VJ World Tour. The four-year tour took in 14 countries
the re-mixed version by the juxtaposition of score,
Sharits (U.S.A) took their work out of the cinema and
follow the traditional concept of the VJ performance:
including: Australia, Belgium, Brazil, France, Germany,
dialogue and effects. In his review of the Kings Place
into the gallery.
an operator (Video Jockey) using real time genera-
Italy, Mexico, Netherlands, Poland, Portugal, Russia,
performance for Sight and Sound (on line) Daniel Trill-
tive performance software playing back edited (often
Spain, Switzerland and the UK. Film, video, music and
appropriated) video clips. Importantly in Figgis’ and
crossover festivals hosted the 33 performances in
In imagining a cinema of remixed visual effects with
ing comments on Figgis’ assertions during the talk:
“He emphasized this in the live mix by rewinding key
rich, authored content, I want to examine the works of
Greenaway’s performances the VJs are themselves the
classic and modern theatres, open squares, industrial
sequences and playing them back so we could hear
two filmmakers who emerged not from media art but
authors of the footage they are playing back and re-
spaces, concert-halls, opera houses and museums. The
the sounds coming from a different part of the screen.
from the cinematic narrative linear tradition and have
mixing. They have (often) written the script, developed
mainstream cinematic version of the Tulse Luper Suit-
He also tried to show how different sounds coloured
now developed practices that could be considered as
the project and directed the cinematic narrative se-
case may be seen as one of the most ambitious and
our perceptions of the action. A snippet from Mahler’s
‘expanded.’ Additionally both have gone some way to-
quences that form the original film versions. The films
challenging of all Greenaway’s works. Talking to Salon.
Fifth Symphony, often used as Hollywood shorthand
wards deploying visual effects sequences in a live con-
have been edited, composited and mixed for cinema
com in 1997 and before principle photography on Tulse
for profound sadness, was played while people milled
text, especially within the works: Timecode Remixed
distribution. The final films have been released as tra-
Luper had begun Greenaway discussed the complexity
around aimlessly; the sound of a character chewing
by Mike Figgis and The Tulse Luper VJ Tour by Peter
ditional narrative films and have found their audience
of the film, its non traditional format and the proposed
gum noisily was faded in during a sex scene.”
7 Here
Greenaway. Greenaway and Figgis represent versions
through distribution to movie theatres. The cinematic
CD Rom and internet versions. “Is there an audience
then, we see the live performance directly impacting
of the same paradigm; both are known primarily for
experience is therefore fundamentally traditional. In
out there that go to the movies, watch TV, buy CD-
on meaning and context on the original ten year old
their feature length films, both have sought to break
the performative iteration the directors are decon-
Roms and are plugged into the Internet? It’s not as
work. Timecode, shot digitally and unedited, told four
out of traditional notions of cinema, both directors
structing, decompositing and unmixing their works live,
though we’re using the same information. Ultimately,
stories which were projected on cinema screens si-
were early adopters of new digital cinematic technolo-
which allows new meanings and contexts to emerge.
(Tulse Luper Suitcase) will be one big global encyclo-
multaneously, challenging audiences with its paradoxi-
gies – A TV Dante (Greenaway), Ten Minutes Older
The live show frees the directors from the confines
pedia about this phenomenon of there’s no such thing
cal complexity and simplicity. Figgis by this time was
The Cello (Figgis) – and both have made experimental
of linearity but requires a great deal of complex un-
as history.”
films aimed at cinema audiences, notably Timecode
picking before the remix versions can be effectively
5
already well known for his experimental projects and
had indeed performed an early version of Timecode
(Figgis) and The Tulse Luper Suitcases (Greenaway).
performed. Figgis talks of the freedom this gives him
It should be remembered that in 1997 the Internet,
live at the 2000 edition of the Edinburgh International
Both filmmakers have gone on to perform these
as being: ”something which I’ve always wanted to do
whilst being a firmly established part of life, was still
Film Festival before the film’s official theatrical release.
works as live VJ sets. It is worth mentioning that both
directors emerged not from the film school tradition
– and in straight cinema I’ve never been able to do it –
which is actually interact with the audience.”
but from Art School in Greenaway’s case and from
4
music performance and theatre in Figgis’ case. This
may go some way to explaining both directors’ urges
unable to stream video to any decently usable standard. By the 2005 launch of YouTube, moving image
content over the Internet had become ubiquitous. So
MISS DONNITHORNE’S MAGGOTT
Greenaway’s aspirations for an extended and expanded
HOW LIVE?
to break out of traditional modes of address. Impor-
232
In his pre Timecode performance talk at Kings Place
version of the Tulse Luper project incorporating the
Sir Peter Maxwell Davies’ Miss Donnithorne’s Maggot
Internet could easily be regarded as forward thinking if
is a challenging work of musical theatre that relates
the tragic story of the slow descent into insanity of
tantly for this paper both directors saw opportunities
On June 17th 2005 at Club 11 Amsterdam, Peter
not visionary. When talking to Hawthorne, Greenaway
to develop new audiences for their work by adopting
Greenaway played his first VJ performance as part
had yet to conceive of a live remixed version of Tulse
Eliza Donnithorne, an Australian society figure of the
and adapting new technologies / methods at an early
of the NoTV CNCDNC visual art club evening. Ac-
Luper and it would indeed be eight more years before
1880s. Written by Maxwell Davies and first performed
stage in the technologies’ development. The use of
companied by music created by DJ Serge Dodwell
the concept of the VJ world tour would be realized.
with the composer conducting his The Fires of Lon-
hand held digital image making technologies such as
(aka Radar), Greenaway used for his set a bespoke
LEONARDOELECTRONICALMANAC VOL 19 NO 3
ISSN 1071-4391
ISB N 978 -1-9 0 6 897-2 2-2
don ensemble at the Town Hall Adelaide, Australia
ISSN 1071-4391
ISB N 978 -1-9 0 6 897-2 2-2
VOL 19 NO 3 LEONARDOELECTRONICALMANAC
233
A R T I C L E
A R T I C L E
ST MAGNUS FESTIVAL
In February 2012 the Artist Recording Company
(ARC) commissioned the Visual Effects Research Lab
(VERL) to create a simple filmed recording of a new
performance of Miss Donnothorne’s Maggott by Alison Wells and Gemini conducted by Ian Mitchell. The
work would be performed over two nights at the 35th
St Magnus Festival on Orkney, Scotland. On listening
to Maxwell Davies’ original recording featuring The
Fires Of London with Mary Thomas in the role of
Donnithorne it was clear that the piece was unusual
and challenging; merely filming the live performance
seemed to greatly undervalue the work. After more
detailed research a concept for staging the piece
alongside a companion film (to be screened simultaneously with the performance) emerged. ARC agreed
an early concept to use a filmed version of the story
as part of a virtual stage set so that the Donnithorne
story could be told on two levels. Whilst developing
this concept, the notion of ‘performing’ the film as a
live remix began to emerge.
Maxwell Davies’ work is arranged as eight songs: 1.
Prelude, 2. Miss Donnithorne’s Maggott, 3. Recitative,
Sir Peter Maxwell Davies’s Miss Donnithorne’s Maggot
4. Her Dump, 5. Nocturne, 6.Her Rant, 7. Recitative
performed live by Alison Wells (mezzo soprano), Gemini (8
and 8. Her Reel. This format offered the opportunity
piece ensemble), Matt Cameron (visual effects) and Peter
to construct a film of eight discreet episodes which
Richardson (film maker). Screened and performed at the St
would be fixed in time but remixed live and in synch
with the stage performance when needed.
on the 9th March 1974 the work has rarely been per-
the fifth is a nocturne-interlude sung for her by the
Magnus Festival in June 2012. © Peter Richardson, 2012. Used
formed since. Amidst on-going debate concerning the
alto flute. Also as in the earlier work, the solo part
with permission.
validity and historical accuracy of the Donnithorne
is a tour de force of vocal effects, requiring a range
The adaptation would attempt to fuse live music the-
story, publishers Boosey and Hawkes’ sleeve notes
of three octaves, though Miss Donnithorne is gen-
atre, pre-shot structural narrative film and live visual
erally more songful in her madness than George III.
effects into a performed hybrid cinematic experience.
The temperature of the ensemble music is also a
Greenaway’s description of the act of performing or
capture the essence of the piece:
Miss Donnithorne was an Australian lady, appar-
little lower, more controlled, perhaps more lady-like,
ently one of the models for Miss Havisham in Dick-
if still expecting wildly brilliant execution.
ens’ Great Expectations; jilted at the last minute,
remixing his films live as being like a: “real time image
8
conductor”
lenge facing the lab was to produce a multi platform
she became a recluse, and the piece discovers her
234
9 became a highly apt simile. The chal-
performance experience whereby the audience would
ranting among the remnants of her wedding cake,
be offered the opportunity to either follow the work’s
which is decorated with instrumentalists. Like Max-
narrative live on stage or as a ‘live’ visual remix of the
well Davies’ mad king, she has eight songs, though
filmed narrative version remixed live on screens hung
LEONARDOELECTRONICALMANAC VOL 19 NO 3
ISSN 1071-4391
ISB N 978 -1-9 0 6 897-2 2-2
ISSN 1071-4391
ISB N 978 -1-9 0 6 897-2 2-2
VOL 19 NO 3 LEONARDOELECTRONICALMANAC
235
A R T I C L E
A R T I C L E
across the stage. The screen remix would provide context and help re-position for the audience the work’s
narrative in a contemporary / historical context.
The Visual Effects Research Lab team identified possibilities for the inclusion of ‘live visual effects’ elements
which could be triggered during the performance.
LIVE TECHNICAL CONSIDERATIONS
A central theme in the work is the notion of decay.
Decay is evident not least in Donnithorne’s appear-
Behind the scenes her maids prepare the house for
tempts to smash her way through the icing. In a series
The venue for the two performances was The Picka-
ance (she takes to the stage in the torn and decayed
the arrival of a hundred guests post-ceremony. Inter-
of close up shots we witness the total destruction of
quoy Centre, Kirkwall: a multi-use venue that was
remains of her wedding dress) but importantly in
cut with images of her groom (a naval officer) setting
the cake and as the camera pulls away we share in
transformed for the St Magnus Festival 2012 into a
the state of long-term neglect and disrepair that her
sail, the audience realizes that no wedding is going
Donnithorne’s frustration and despair.
theatre seating 250. Throughout the development
house is allowed to fall into. The wedding banquet is
to take place. Donnithorne now comes to the same
phase the VERL team considered various playback
(as the legend has it) left untouched by the guests
conclusion: with no groom there will be no wedding.
and mixing options for the live elements of the per-
and subsequently left on the hero’s table to decay and
We next find her pacing round her house obsessed by
rot for the rest of her days: a symbol of her decline
the sound of a metronome, which acts as a cypher for
into madness. Using the decaying food as a cypher for
DOES SHE SING?
formance. Initially a traditional VJ set-up emerged as
her grief and anguish. Unable to console herself she
The role of Miss Donnithorne was portrayed on cam-
and manipulate images the VJ would require source
Miss Donnithorne’s condition the team researched
attempts to flee the confines of the house only to find
era and live on the St Magnus stage by Alison Wells
material (i.e. the un-edited sequences from the Donni-
methods for filming time lapse sequences of real food
herself transported over a hundred years into a bleak
(Mezzo Soprano) whose acting experience is in opera
thorne film) playing back, through a Video Synthesizer,
decaying which could be sped up or slowed down to
and dank future.
and music theatre. Wells would be singing the part of
the existing video stream from disk or other storage
Donnithorne for the St Magnus Festival performances
media. Mixing hardware and VJ software such as
dramatic effect during the performance. (Greenaway
uses this technique to good effect in his 1986 film A
In a decaying modernist tower block the same prepa-
but would not be singing ‘on camera’ for the filmed
Modul8 would then allow the combining of multiple
Zed And Two Noughts). In order to enhance the decay
rations are being made for a far more meager cel-
and remixed versions, instead she would be acting the
streams of video and the visual effects stream, which
and, as the possibility of shooting real food decaying
ebration. The story has been transposed to the early
role on camera as interpreted and re-imagined by the
is then outputted to a screen or projector (a HD pro-
over a period of months was not an option for the
2000s where we find Donnithorne living amongst the
director (in a similar manner to the silent films of the
jector and an on stage single screen in this case). The
team, a package of visual effects shots was required.
detritus of a decaying city on the 12th floor of a tower
early twentieth century). This led to an interesting ex-
large file sizes of the additional visual effects sequenc-
The scene became fundamental to the performance.
block. Again she prepares the wedding breakfast only
periential complexity: Wells would be performing (sing-
es which were to be keyed live meant that options
As the stage set for the live work would include only
to find that her intended does not appear. Now lost
ing) and performing (acting) live, whilst the audience
for meaningful live remixing were becoming severely
a wedding cake on a table, the banquet (which is re-
in her own madness the story cuts between Donni-
in the theatre would be able to select their viewpoint
limited. The traditional set up would not be suitable
ferred to in a number of the eight songs) could play a
thorne’s two existences, drawing contrasting portraits
by choosing to follow the performance (singing and
for the live Donnithorne performance.
central role in the film and its subsequent remix.
of a sad decline into madness. Tormented by memo-
acting) live on stage or the performance (acting) as live
ries of her lost groom and terrorized by a gang of local
remixed narrative visual projected onto the screens
The film narrative was shot on 4:4:4 colour space over
youths Miss Donnithorne becomes a prisoner in her
whilst listening to the (live / sung) performance.
six days on location in Dundee, Scotland. Shot in the
own home(s). Intercut with the story are images of
traditional cinematic linear method, using actors por-
the wedding banquet. Long tracking shots reveal the
traying the various imagined characters to which Don-
splendor of the food: glazed chickens, game birds, fish
nithorne refers throughout her eight songs, the direc-
in aspic, breads, cold meats, glistening potatoes and
tor (Richardson) interpreted freely on the themes and
tropes of the vocal score to produce a film which op-
VISUAL EFFECTS (LIVE?)
Having shot and edited the film the VERL team
THE REMIX
concentrated efforts on perfecting the visual effects
various deserts. At the head of the table Miss Don-
115 minutes of footage was generated during the
effects live. The tracking shots of the wedding ban-
nithorne’s tastefully decorated wedding cake adds the
shoot and edited using Final Cut Pro into a 33-minute
quet in its un-decayed state were filmed on location at
a stately home in East Scotland. The Art Director had
sequences whilst researching methods for mixing the
erates between two worlds: the contemporary urban
finishing touch. As the horror of Donnithorne’s situ-
film split into eight sections (songs). With the final
and the historical suburban Victorian.
ation unfolds the same shots are repeated, only this
outcome anticipated as somewhere between a VJ set
researched traditional Victorian wedding foods and
time the food is deteriorating before our eyes: the fish
accompanying a recital and a moving stage set, care
prepared the ‘prop’ food to be as historically accurate
dissolve into maggot infested slime, the chicken and
had to be taken to not to distract the audience from
as possible. The sequence was filmed in slow motion
tralian Victorian house. We find Miss Donnithorne
game birds decompose to dust and the cold meats
the performance by over re-mixing the filmed mate-
and camera data was recorded as the same moves
in a state of excitement on the eve of her marriage.
shrivel and disintegrate. As the camera reaches the
rial. Whilst the technical set up for the performance
would be replicated some weeks later for the visual
She carefully reviews the wedding preparations tak-
top of the cake we see the bride and groom figures
is not the main thrust of this paper it would be worth
effects decay sequence. Empty background plates ing great care to arrange the food and delicacies and
transform from models into the real Miss Donnithorne
discussing the options and choices made at this stage
were also shot for the forthcoming green screen
ultimately the wedding cake on the sumptuous table.
and her groom. Donnithorne falls to her knees and at-
in development by the live team.
replication shoot. On completion of principle photog-
The traditional film element opens on a grand Aus-
236
the most viable. As a basic requirement to perform
LEONARDOELECTRONICALMANAC VOL 19 NO 3
ISSN 1071-4391
ISB N 978 -1-9 0 6 897-2 2-2
ISSN 1071-4391
ISB N 978 -1-9 0 6 897-2 2-2
10
VOL 19 NO 3 LEONARDOELECTRONICALMANAC
237
A R T I C L E
A R T I C L E
References and Notes
raphy the food was stored for a month and allowed
the various cues of Davies’s eight songs with the live
context to a particularly challenging work of music
to rot. The wedding banquet including the table was
team who were set up on a balcony opposite the
theatre. The audience on both evenings gave mixed
mance during the NoTV CNCDNC visual art club evening
then reassembled in the green screen studio using the
stage. The performance duration was circa 35 minutes
responses to their experience anecdotally. Opera and
in Amsterdam.
camera data and Art Director’s on-set photographs to
with the film remix / visual effects elements account-
music theatre aficionados appeared to both hate and
accurately position the food in relation to the camera.
ing for a screen time of circa 30 minutes the VJ team
love the experience, however it is undeniable that the
The scene was re-filmed against green this time with
had scope to manipulate the timings of the visuals so
St Magnus Festival audience had witnessed an innova-
the decayed food replacing the pristine version of the
that they were synchronous with the performance
tive staging of Miss Donnithorne’s Maggott. Filmmak-
3. The term ‘expanded cinema’ was coined in the mid-1960s
previous month. In post-production the VERL team
whilst adhering to the ethos of the remix performanc-
ers such as Greenaway, Figgis and others have made
by Stan Vanderbeek to describe his multi projector film
used a mixture of Autodesk Maya (animation) and
es of Greenaway and Figgis.
many attempts to create a new, more expanded or
Nuke (compositing) to replace the background repli-
immersive version of the cinematic experience and
cate the camera moves and key, as if time lapse, the
their works have been well received by audiences
decayed food. These techniques are standard meth-
CONCLUSIONS
ods for visual effects sequences. A reconnaissance trip
2. Lumiere Brothers’ film L’arrivée d’un train en gare de La
Ciotat (The Arrival of a Train at La Ciotat Station) was
first shown in 1895.
experiments.
4. Michael Faulkner/D-Fuse, ed., VJ: Audio-Visual Art and VJ
Culture (London: Laurence King Publishing, 2006).
across the globe. In attempting to appropriate and (re)
5. Christopher Hawthorne, “Flesh and Ink,” Salon,1997, http://
expand their methodologies the VERL team sought to
www.salon.com/1997/06/06/greenaway970606/ (ac-
to the Orkneys gave the VERL team the opportunity
This paper charts the progress of the Donnithorne
produce a visual effects rich cinematic and immersive
to liaise with the Pickaquoy Centre producer, stage
project which began with aspirations towards a new
audience experience taking the VJ / club experience
crew, the Artist Recording Company producer and
form of performed visual effects played and effected
to a new audience at an internationally renowned clas-
engineer. The venue’s length and height meant that
live and ended with a far more traditional synthesis of
sical music and arts festival on a stunning Island off
the HD projector would be giving an image height of
live performance and accompanying film remix. It will
Scotland’s East coast.
two meters (the film was shot 16:9 aspect ratio) and
be some time before technology and in particular pro-
thus the audience would be able to fully appreciate
cessing and render time can deliver a games engine
the high resolution film from the furthest parts of the
like software for VJs. I have attempted to show that
cessed December 31, 2012).
6. The talk took place at Kings Place, London on Sept 15th
2010.
7. Daniel Trilling, “Review: Timecode Live: Auartet for Four
Cameras,” British Film Institute, http://old.bfi.org.uk/
sightandsound/reviews/timecode-live-2010.php (accessed
December 31, 2012).
8. Paul Griffiths, ed, Peter Maxwell Davies: Miss Donnithorne’s
Acknowledgements
auditorium. The on-stage performance would be cap-
there is a future for a visual effects rich live cinema
tured on four cameras and it was envisaged that live
and that the VERL team’s efforts to incorporate vi-
footage would be also be mixed within the VJ element
sual effects sequences live in the Miss Donnithorne’s
The author wishes to thank Professor Howard Burrell CEO
of the performance.
Maggot St Magnus Festival performance have pro-
The Artist Recording Company, University of Northampton
vided a ‘proof of concept’ in the spirit of Figgis and
for commissioning the project, Alison Wells for her stunning
content&task=view&id=34#.UMH49kL89UQ (accessed
Greenaway’s pioneering works. Looking to the future
performances and eternal patience, the VERL team at Duncan
December 31, 2012).
selected for the performance proved that the size and
more research and development is required into the
of Jordanstone College Of Art and Design, University Of
run time of the near un-compressed rendered visual
core technologies which enable live video mixing. It
Dundee in Scotland.
effects sequences would be beyond its capabilities.
was never the team’s intention to focus on adapting
Whilst the rendered effects amounted to a small pro-
or developing new technology, instead our focus was
portion of the entire performance the team felt that
on producing a large-scale live cinematic experience
the juxtaposition of the live performance with visual
with a high proportion of visual effects. If this were
Attempts to test the standard VJ set up the team had
238
1. On 17th June 2005, Peter Greenaway did a VJ perfor-
effects embedded within the film remix was a vital
achieved would it go some way towards altering audi-
element of the piece. The decision was taken to pre-
ences’ perceptions of how music theatre could be
render the decaying food sequence before the per-
staged and in particular bring new meaning and con-
formance and play the renders into the remix via final
text to Maxwell Davies’ post-modern work of 1974?
cut pro files live during the appropriate song sequence.
The performance was indeed immersive, the remixing
Richardson worked with Wells on the Pickaquoy stage
of the pre-shot sequences meshed well with the on-
during rehearsals to block her performance and agree
stage performance adding the intended meaning and
LEONARDOELECTRONICALMANAC VOL 19 NO 3
ISSN 1071-4391
ISB N 978 -1-9 0 6 897-2 2-2
Maggott Study Score – Hawkes Pocket Score 1290 (London: Boosey and Hawkes, 1974).
9. “Peter Greenaway – Live Cinema,” NoTV Visual Music, http://www.notv.com/index.php?option=com_
10.A plate is a term used to describe a shot of a background,
which will be used to replace a green screen background
in post production.
11.The Visual Effects Research Lab is funded by a grant from
the North Sea Intereg Programme of the European Union.
ISSN 1071-4391
ISB N 978 -1-9 0 6 897-2 2-2
VOL 19 NO 3 LEONARDOELECTRONICALMANAC
239

* Your assessment is very important for improving the work of artificial intelligence, which forms the content of this project