Sound is almost like a drug. It’s so pure that when it goes in your ears, it instantly does something to you. And you can tell if that’s working for you. – David Lynch

David Lynch’s attention to and unique use of audio in his films is well documented. Having read Chion’s Audio-Vision and recently purchasing his book David Lynch I’ve set about to see some of the Lynch films I haven’t seen with a critical ‘ear’. I hired Mulholland Drive and heard/saw a powerful and typically eerie audiovisual moment. In the Club Silencio scene Lynch insists that we notice an obvious audiovisual disjunction:

Many readings of the film decide that Club Silencio marks a portal where Dianne’s fantasies (the first two thirds of the film) end and she returns to her real life. Makes sense I guess. She freaks out, finds the box, opens it up, Betty disappears, Dianne gets sucked into a black void and bang – reality.

Timothy Takemoto’s reading sees the film in a different light, pointing to Zizek’s criticisms of such films and –

    Their revelation of a clear and definite reality to which the hero and audience can return. The truth Zizek claims, is more unsettling: there are only layers of fantasy behind which, at best, a “grey fog”.

As Zizek might have, this blog thought that, “The Real of Mulholland Dr is not Diane’s supposedly waking world, but the paradoxically entrancing insomniac realm of Club Silencio.” The Real in a Lacanian sense – Club Silencio as brief glimpse of the traumatic and impossible Real.

The audiovisual illusion in Club Silencio puts the audience through the same audiovisual disruption as Dianne. The truth is spelled out clearly, ‘There is no band. What you will hear are recordings’ but Chion’s audiovisual contract in which the phenomenon of synchresis, ‘the spontaneous and irresistible mental fusion, completely free of any logic, that happens between a sound and a visual when these occur at exactly the same time,’ gives the suited magician type guy an undeniable and definite power. It is impossible again, to void the woman’s ‘singing’ of the emotion held within the song. Betty, Dianne and the viewer cannot resist the seduction of the illusion and when the singer falls, and the music continues, we are all shocked.

Lynch has used a powerful illusion to give Dianne (and the viewer) a glimpse of reality. And what do we learn? Fantasy and illusion is inescapable and essential to reality. Now, what’s this got to do with audiovisual relations?

Chion, who provides the audiovisual theoretical framework for my work, also applied Lacanian psychoanalysis to Hitchcock and Lynch. (I know, a weird co-incidence). When you look at some of Chion’s terms; audiovisual illusion, disparity, projection and contamination, there’s a definite deep, dark, psychoanalytic feel to them. Then we have Lynch, exploiting this illusion with powerful moments of audio-visual disjunction and disruption, and with some Lacanian thought, potentially remarking on the human psyche, and the irresistible seduction of fantasy in constructing the self.

What’s left is a pretty intense and dark approach to audiovisual relations, which potentially has little to do with the pursuit of audiovisual fusion/harmony. Beyond pointing out the phenomenon of synchresis, Chion did not write on the artistic potential of pursuing this ‘irresistible fusion’. He rather approaches the technicalities of film sound, particularly in films like Lynch’s where audiovisual dissonance is used to powerful effect.

We know the audiovisual illusion is a powerful one but is the pursuit of audiovisual fusion, the pursuit of a fantasy? Are the senses as disparate as Chion thinks so?

    “Every time I hear sounds, I see pictures. Then, I start getting ideas. It just drives me crazy” –David Lynch

In ‘Untitled Iterations’, Ben Bogart writes on generative art, consciousness and perception in a straightforward and useful way.

    Consciousness is the process of turning noise into pattern. When we look around the world and listen we do not see or hear the cascade of unimaginable amounts of interacting particles, from waves of light scattering off objects to the molecules that carry the sound we hear; rather, we see a defined, understandable structure of our world. We do not see the world as science defines it. We instead collectively participate in the process of creating the structure that comprises our world.

It’s easy to forget the infinite amount of raw data the human brain filters and processes every day. Without getting into perception or philosophies of consciousness, (something I know little about) it’s clear that the human brain has an amazing ability to find pattern and structure in chaos. A perfect example – noise rock.

    Noise and pattern are not two concepts that are mutually exclusive. In fact, noise and pattern are markers at two ends of a single continuum that classify all types of structure.

For example, we could place Sunn o))) with their sub-bass drones of distortion fairly concretely and deeply into the noise side of the spectrum.

We could place a minimal, repetitive electronic number (something I again, know little about) closer to the pattern end.

Brisk, the band I’ve chosen to ‘visualise’ won’t settle on its place on the spectrum. ‘Hell or High Water’ dances up and down it at will. Six instruments all screaming at once, generally in dissonance, turns to two guitars playing a pleasing melody. Sometimes with, and sometimes without a typical song (verse/chorus) structure. Generally adhering to the laws of rhythm, sometimes barely, sometimes with overlapping time signatures and complex polyrhythms. It’s possible, for the few who are entertained by this toying with chaos that the kick is in decoding the chaos. Or to use a cheesy analogy; possessing the correct, obscure codec.

From Australian academic Paul Carter’s ‘Material Thinking’:

    We pointed out that, in ancient Greek thought, chaos did not have its present-day meaning. It signified ‘the yawning, or gaping open of time and space to permit creation … Greek chaos imagines the interpenetration of lines, a crossing that does not cancel out but mutually transforms.

Bogart considers, ‘the vast area between noise and pattern as chaos.’ So, this is what I’m getting at:


Interesting is the computer’s struggle to successfully interpret and create pattern from raw audio data. Picking up even the simplest of patterns; a beat, even in wild, heavy music is a straightforward task for any person but beat detection is still, to this day, an unreliable technology.

Compared the rich abilities of the ear and brain, asking a computer to visualise music seems a strange concept. The question is – can applying structure, systems and process to music full of confusion, noise and chaos reveal new qualities? Enhance this process of creation, creating a new space of interpretation? Or at least create a worthwhile viewing experience? I’m hoping so in the context of the video clip.

I’ve just installed a bigger hard drive in my Xbox and put most of my music on it. XBMC comes with a variety of visualisers and when faced with the choice of which one, I chose none. It’s not the way I generally want to experience music. My media centre screensaver is now a black screen…

Michel Chion’s ‘Audio-Vision: Sound of Screen’ provides a rare theoretical framework for studying the audiovisual relationship. Chion forges terms such as synchresis, spatial magnetization, acousmatic sound, reduced listening, rendered sound and Acousmêtrē highlighting the lack of theory regarding sound which he accredits to, “something about sound that bypasses and surprises us, no matter what we do.”

Chion begins by pointing out that there is no “natural and pre-existing harmony between image and sound” and makes a point with definite conceptual implications for the fused AV practitioner:

    Visual and auditory perception are of much more disparate natures than one might think. The reason we are only dimly aware of this is that these two perceptions mutually influence each other in the audiovisual contract, lending each other their respective properties by contamination and projection.

The reassociation of image and sound is the fundamental stone upon which film sound is built. Using example after example Chion highlights the ease in which the viewer can be fooled by sounds ambiguous nature. This reassociation is done for many reasons – often because a simulated or rendered sound seems more real than the original sound. Synchresis explains the phenomena and simultaneously highlights the perceptual possibility of audiovisual construction while revealing the unfeasibility of a true and unique audiovisual harmony:

    The spontaneous and irresistible mental fusion, completely free of any logic, that happens between a sound and a visual when these occur at exactly the same time.

Excluding the small section on music video, it is important to note that Chion’s observations describe the process of adding sound to image – the reverse process to which many fused AV producers work. Chion’s notions of the audiovisual illusion and added value are particularly attractive providing the process works both ways. Imagine the following excerpt with the words sound and image substituted for each other:

    By added value I mean the expressive and informative value with which a sound enriches a given image so as to create the definite impression, in the immediate or remembered experience one has of it, that this information or expression “naturally” comes from what is seen, and is already contained in the image itself.

Chion does later point out the danger of applying musical analogy to film and explains the relationship between counterpoint and harmony by studying audiovisual dissonance; points in audiovisual experience in which the audio has no tangible counterpoint to the film. I’ll be thinking about this some more and applying some other theories of consonance and dissonance in fused AV and visual music.

I just received a big box of books in the mail including David Lynch by Chion, a filmmaker who is definitely worth studying considering his powerful and strange use of audio/music in film. It will also be interesting to apply John Whitney’s quest for Digital Harmony to Chion’s thoughts of audio visual disparity.

New recursive structure

June 26, 2007

The first song, ‘The Power of Independent Trucking’ by Big Black, the second, ‘Sporting Life’, by Young Marble Giants.

YouTube absolutely destroyed the quality. I’d recommend you watch it on Vimeo where it looks slightly better:

Render Test 2: Structure 2 from njmcgee on Vimeo

(Vimeo won’t embed in a hosted blog (and doesn’t look that good anyway). I’ll be thinking about a way to get better quality video embedded. Any ideas? Let me know.)

Best results so far with heavy rock music. Moments of real rhythm, and surprisingly dynamic visual response. The exponential relationship between element size and fft values allows individual legs to be simultaneously effected by their corresponding spectrum band with great effect. I will probably tone things down – reduce the number of elements and better control over the overall scale. Oh, and for some reason one of the legs gets stuck…

Thankfully, as I continue to bash out ‘programs’, complex visual outcomes I have not, or could not have planned are emerging.
As I tip toe towards generativity I’m wondering how to approach a ‘final’ piece – considering the unique movements, forms and reactions each time sketch is run.

Windowed Averaging

June 18, 2007

Recently I’ve been keeping things very simple – eight frequency bands driving all my graphics. My previous experiments used 64 bands, split into four groups of sixteen with Ess kindly working out the average for these groups. My past methods were based completely on event triggering – when a spectrum band peaks above its group’s average by a certain amount something happens.

I’ll definitely work a feature like this back into the mix. If tamed correctly it finds drum hits (particularly the snare) and highlights big dynamic changes really well.

For now, I’ve stripped things back to find more complex ways of interpreting these spectrum values. The plan is to have these always changing spectrum values controlling movement and speed. Turns out it’s quite tricky – even with audio dampening, getting smooth results will take a bit more work. Until then I’ve been thinking up ways to visually highlight structural change beyond a keyboard or time based trigger.

Here is St. Anthony’s Fire (the song I will be using, right click to download) split into eight frequency bands and averaged every 60 frames (two seconds). Not exactly a sexy piece of data visualisation I know but useful enough.

(right click to see a larger version)

The song is very heavy and a total mess of frequencies but windowed averaging has made the structural changes quite clear to the trained eye. I’ve circled the points in which I could set thresholds or even better link a windowed average to a particular variable to get a nice gradual transition rather than an instant shift. Hopefully I can give these six sections a completely different look/structure and provide smooth transitions to create a clip that’s always flowing and moving. Generative animation’s beauty is its capacity to provide output, shapes and movements beyond the programmer’s initial intention – visuals that are somewhat ‘alive’. I’d like to turn the visuals into a character of sorts, a complex and wild beast that twists, turns and changes with the music. Avoiding hard cuts and instant transitions will help the visuals to live and breathe.