In ‘Analysing Musical Multimedia’ Nicholas Cook provides an extremely clear general theory of how media engage with one another in multimedia works in general (from opera through to commercials). Judging the book solely by its cover and thinking I had found a first year textbook, I almost placed it back on the library shelf. Thankfully, I checked the contents-

Chapter 1 – Synaesthesia and Similarity
Chapter 2 – Multimedia as Metaphor
Chapter 3 – Models of Multimedia
and so on, and decided it might be worth a shot.

Cook’s thoughts on synaesthesia were extremely refreshing, ‘Synaesthesia provides some hints as to what multimedia is: but, perhaps more importantly, it supplies an illuminating model of what multimedia is not.’ Cook believes the most useful way to study multimedia is in its element’s differences and interactions, rather than through the eyes of synaesthesia, a phenomenon based on similarity, duplication and translation.

Speaking on Skriabin’s (an apparent synaesthete) fifth symphony, Prometheus, which includes a part for a Tastiera per luce, or colour keyboard:

    The luce part literally does add little; for while the slower part has no discernible relationship to what is heard, the faster part simply duplicates information that is already present in the music. In neither case is there a substantial degree of perceptual interaction between what is seen and what is heard—which means that, in a significant sense, Prometheus does not belong to the history of multimedia at all. And to say this is to suggest there is a definite limit to what the phenomenon of synaesthesia can tell us about multimedia, because synaesthesia consists precisely of the duplication of information across different sensory modes. To demand something other than duplication is to go beyond the bounds of synaesthetic correspondence.

The paragraph alone makes Cook’s attack seem more severe than in correct context but there remains an interesting point. Synaesthesia is an amazing phenomenon, and has metaphorical relevance but it also has its limits; its subjective nature among synaesthetes and its complete adherence to similarity. We need to be armed with a wider approach to cross-media works. Cook sees synaesthesia as potentially ‘enabling condition for multimedia, but not a sufficient one.’

    To analyse music is also to be committed to the idea that we perceive notes in terms of the relationships between them: we perceive each note as influencing, and being influenced by other notes—or at any rate, if we do not, it is hard to see what we could be analysing. In a nutshell, we analyse the interaction between the elements of the music: that is what analysing music means. And exactly the same applies to multimedia. To analyse something as multimedia is to be committed to the idea that there is some kind of perceptual interaction between its various individual components, such as music, speech, moving images, and so on: for without such interaction there is nothing to analyse.

The rest of the book is based on this idea of interaction and the idea of multimedia as metaphor. Cooks sees in media interaction the potential for emergence (the result of putting medias together as more than the sum of its parts) and sees the metaphor model as invoking ‘similarity not as an end, but as a means.’ Cook talks about semiotics, motion and gesture (something I might elaborate later on with my reading of The Sonic Self) and arrives at a basic model for the analysis of multimedia.

For a full understanding, read the book, but I’ll explain simply and inadequately. We start at the top and decide whether two media’s relationships are consistent or coherent. A colour organ or keyboard would be a simple and perfect example of a consistent relationship (a relationship he sees as quite rare in current multimedia) where as coherence allows for differential elaboration. The next distinction is a fuzzy one, which Cook acknowledges; he also acknowledges the potential for media forms to shift between classifications. These two quotes might help:

    Conformance begins with originary meaning, whether located within one medium or diffused between all; contest, on the other hand, ends in meaning. And the association of conformant models with synaesthestic and metaphysical speculation demonstrates, conformance tends towards the static and the essentialized, whereas contest is intrinsically dynamic and contextual.
    The term ‘contest’ is intended to emphasize the sense in which different media are, so to speak, vying for the same terrain, each attempting to impose its own characteristics upon the other. One might develop the analogy by saying that each medium strives to deconstruct the other, and so create space for itself.
    The mid-point between these two extremes is represented by the third model of multimeda, complementation, which Figure 3.1 represents in negative terms as that which exhibits neither consistency nor contradiction… complmentation is readily associated with the succesive phases of multimedia production. The classical Hollywood film for example, for instance, was in general virtually complete before it was passed onto the composer for scoring: the composers job was understood as one of complementing….

The main point I wanted to get to was Cook’s ideas on contestation. The possible repercussion being that that works such as mine; generative AV work (dynamic and cross imposing characteristics), form an AV relationship of contestation, rather than harmony. Cook would be useful in supporting the argument that such works are linked through metaphor, motion, gesture (and illusion) rather than similarity or some sort of natural harmony.


August 10, 2007

“So long as the expression ‘Gesamtkunstwerk’ (or ‘integrated work of art’) means that the integration is a muddle, so long as the arts are supposed to be ‘fused’ together, the various elements will all be equally degraded, and each will act as a mere ‘feed’ to the rest. The process of fusion extends to the spectator, who gets thrown into the melting pot too and becomes a passive (suffering) part of the total work of art. Witchcraft of this sort must of course be fought against. Whatever is intended to produce hypnosis, is likely to induce sordid intoxication, or creates fog, must be given up.”

Brecht on Theatre: The Development of an Aesthetic. 1964

In Audio-Vision, Chion boldy suggests that our senses are separate and disparate, describing synchresis as falsely ‘natural’ and ‘immediate’ and ultimately an illusionary experience. Rather than balancing the argument with talk of synesthesia and it’s associated art works I want to focus on this idea of illusion.

An earlier post on David Lynch provides a wonderful example of Lynch exposing this illusion. In ‘Club Silencio’ Lynch creates a moment of almost reverse synchresis providing an AV experience that feels unnatural, stressing that we cannot help but be affected and fooled by this AV trickery. Lynch provides a similar effect through eerie drones, the lack of recorded sound, or audio and visual effects/distortions, (think the Red Room in Twin Peaks) expossing an inescapable illusion.

This blog post really got me thinking. Lynch is essentially and brilliantly pointing out, that illusion (or fantasy) is inescapable and essential to reality. Enter Lacanian thought/Slavoj Zizek.

In Cyberspace, Zizek writes about the interface and the ‘fantasy screens’ of computing:

    What is a fantasy screen, an “interface”? Sometimes we find it even in nature, as in the case of the Cerknica lake in Slovenia: this intermittent lake (during its seasonal eruption, the water throws fishes into the air) was experienced as a kind of magic screen, a miracle of something emerging out of a void. As early as the seventeenth century, this phenomenon intrigued natural scientists—a Slovene author, Janez Valvasor, became a member of the British Royal Academy for providing an explanation of this mystery (an intricate network of underground channels with different pressures). Perhaps, this is the most elementary definition of a mechanism: a machine which produces an effect in the precise sense of a ‘magical’ effect of sense, of an event which involves a gap between itself as the raw bodily materiality—a mechanism is that which accounts for the emergence of an “illusion.” The crucial point here is that the insight into the mechanism does not destroy the illusion, the “effect,” it even strengthens it insofar as it renders palpable the gap between the bodily causes and their surface-effect.

The illusion of synchresis could be seen in the same light. We may know the mechanics of its illusion, but the illusion is just as, if not more powerful. Lynch’s audiovisual disjunctions reveal a gap between surface effect and reality – could fused AV works through the creation of an illusion mechanism, through simulating an unreal relationship, reveal the same gap?

Zizek’s quote would suggest that the pursuit of audio visual fusion is not the pursuit of the pure and interrupted functioning of fantasy. A constructed AV relationship while an effective illusion could also be seen to render palpable the gap between audio and vision. Perhaps this is it’s lure?

The hard part now. Investigating this creatively…

The Real as the irreducible gap between vision and audio?
Could AV illusion offer us a peak into the impossible Real?
How to investigate the fantasmic aspects of the audio-visual relation?

In reverse?

July 31, 2007

In this post I believed I was working to the reverse process to Chion’s analysis; that I was adding visuals to audio rather than the other way round. The more I think about it, the more I realise that I may be wrong. The graphics system I am building is being built independently of my chosen soundtrack and to be honest, adding the audio is the final step. Certain properties of the visuals are permanent, while the soundtrack is completely interchangeable.

This process only differs to film sound in that the visuals are affected not only by the phenomena of ‘added value’, due to synchresis, but also directly in form and shape by audio analysis. Possibly an extension of Chion’s ideas on film sound?

    Sound is almost like a drug. It’s so pure that when it goes in your ears, it instantly does something to you. And you can tell if that’s working for you. – David Lynch

David Lynch’s attention to and unique use of audio in his films is well documented. Having read Chion’s Audio-Vision and recently purchasing his book David Lynch I’ve set about to see some of the Lynch films I haven’t seen with a critical ‘ear’. I hired Mulholland Drive and heard/saw a powerful and typically eerie audiovisual moment. In the Club Silencio scene Lynch insists that we notice an obvious audiovisual disjunction:

Many readings of the film decide that Club Silencio marks a portal where Dianne’s fantasies (the first two thirds of the film) end and she returns to her real life. Makes sense I guess. She freaks out, finds the box, opens it up, Betty disappears, Dianne gets sucked into a black void and bang – reality.

Timothy Takemoto’s reading sees the film in a different light, pointing to Zizek’s criticisms of such films and –

    Their revelation of a clear and definite reality to which the hero and audience can return. The truth Zizek claims, is more unsettling: there are only layers of fantasy behind which, at best, a “grey fog”.

As Zizek might have, this blog thought that, “The Real of Mulholland Dr is not Diane’s supposedly waking world, but the paradoxically entrancing insomniac realm of Club Silencio.” The Real in a Lacanian sense – Club Silencio as brief glimpse of the traumatic and impossible Real.

The audiovisual illusion in Club Silencio puts the audience through the same audiovisual disruption as Dianne. The truth is spelled out clearly, ‘There is no band. What you will hear are recordings’ but Chion’s audiovisual contract in which the phenomenon of synchresis, ‘the spontaneous and irresistible mental fusion, completely free of any logic, that happens between a sound and a visual when these occur at exactly the same time,’ gives the suited magician type guy an undeniable and definite power. It is impossible again, to void the woman’s ‘singing’ of the emotion held within the song. Betty, Dianne and the viewer cannot resist the seduction of the illusion and when the singer falls, and the music continues, we are all shocked.

Lynch has used a powerful illusion to give Dianne (and the viewer) a glimpse of reality. And what do we learn? Fantasy and illusion is inescapable and essential to reality. Now, what’s this got to do with audiovisual relations?

Chion, who provides the audiovisual theoretical framework for my work, also applied Lacanian psychoanalysis to Hitchcock and Lynch. (I know, a weird co-incidence). When you look at some of Chion’s terms; audiovisual illusion, disparity, projection and contamination, there’s a definite deep, dark, psychoanalytic feel to them. Then we have Lynch, exploiting this illusion with powerful moments of audio-visual disjunction and disruption, and with some Lacanian thought, potentially remarking on the human psyche, and the irresistible seduction of fantasy in constructing the self.

What’s left is a pretty intense and dark approach to audiovisual relations, which potentially has little to do with the pursuit of audiovisual fusion/harmony. Beyond pointing out the phenomenon of synchresis, Chion did not write on the artistic potential of pursuing this ‘irresistible fusion’. He rather approaches the technicalities of film sound, particularly in films like Lynch’s where audiovisual dissonance is used to powerful effect.

We know the audiovisual illusion is a powerful one but is the pursuit of audiovisual fusion, the pursuit of a fantasy? Are the senses as disparate as Chion thinks so?

    “Every time I hear sounds, I see pictures. Then, I start getting ideas. It just drives me crazy” –David Lynch

Michel Chion’s ‘Audio-Vision: Sound of Screen’ provides a rare theoretical framework for studying the audiovisual relationship. Chion forges terms such as synchresis, spatial magnetization, acousmatic sound, reduced listening, rendered sound and Acousmêtrē highlighting the lack of theory regarding sound which he accredits to, “something about sound that bypasses and surprises us, no matter what we do.”

Chion begins by pointing out that there is no “natural and pre-existing harmony between image and sound” and makes a point with definite conceptual implications for the fused AV practitioner:

    Visual and auditory perception are of much more disparate natures than one might think. The reason we are only dimly aware of this is that these two perceptions mutually influence each other in the audiovisual contract, lending each other their respective properties by contamination and projection.

The reassociation of image and sound is the fundamental stone upon which film sound is built. Using example after example Chion highlights the ease in which the viewer can be fooled by sounds ambiguous nature. This reassociation is done for many reasons – often because a simulated or rendered sound seems more real than the original sound. Synchresis explains the phenomena and simultaneously highlights the perceptual possibility of audiovisual construction while revealing the unfeasibility of a true and unique audiovisual harmony:

    The spontaneous and irresistible mental fusion, completely free of any logic, that happens between a sound and a visual when these occur at exactly the same time.

Excluding the small section on music video, it is important to note that Chion’s observations describe the process of adding sound to image – the reverse process to which many fused AV producers work. Chion’s notions of the audiovisual illusion and added value are particularly attractive providing the process works both ways. Imagine the following excerpt with the words sound and image substituted for each other:

    By added value I mean the expressive and informative value with which a sound enriches a given image so as to create the definite impression, in the immediate or remembered experience one has of it, that this information or expression “naturally” comes from what is seen, and is already contained in the image itself.

Chion does later point out the danger of applying musical analogy to film and explains the relationship between counterpoint and harmony by studying audiovisual dissonance; points in audiovisual experience in which the audio has no tangible counterpoint to the film. I’ll be thinking about this some more and applying some other theories of consonance and dissonance in fused AV and visual music.

I just received a big box of books in the mail including David Lynch by Chion, a filmmaker who is definitely worth studying considering his powerful and strange use of audio/music in film. It will also be interesting to apply John Whitney’s quest for Digital Harmony to Chion’s thoughts of audio visual disparity.