In ‘Analysing Musical Multimedia’ Nicholas Cook provides an extremely clear general theory of how media engage with one another in multimedia works in general (from opera through to commercials). Judging the book solely by its cover and thinking I had found a first year textbook, I almost placed it back on the library shelf. Thankfully, I checked the contents-

Chapter 1 – Synaesthesia and Similarity
Chapter 2 – Multimedia as Metaphor
Chapter 3 – Models of Multimedia
and so on, and decided it might be worth a shot.

Cook’s thoughts on synaesthesia were extremely refreshing, ‘Synaesthesia provides some hints as to what multimedia is: but, perhaps more importantly, it supplies an illuminating model of what multimedia is not.’ Cook believes the most useful way to study multimedia is in its element’s differences and interactions, rather than through the eyes of synaesthesia, a phenomenon based on similarity, duplication and translation.

Speaking on Skriabin’s (an apparent synaesthete) fifth symphony, Prometheus, which includes a part for a Tastiera per luce, or colour keyboard:

    The luce part literally does add little; for while the slower part has no discernible relationship to what is heard, the faster part simply duplicates information that is already present in the music. In neither case is there a substantial degree of perceptual interaction between what is seen and what is heard—which means that, in a significant sense, Prometheus does not belong to the history of multimedia at all. And to say this is to suggest there is a definite limit to what the phenomenon of synaesthesia can tell us about multimedia, because synaesthesia consists precisely of the duplication of information across different sensory modes. To demand something other than duplication is to go beyond the bounds of synaesthetic correspondence.

The paragraph alone makes Cook’s attack seem more severe than in correct context but there remains an interesting point. Synaesthesia is an amazing phenomenon, and has metaphorical relevance but it also has its limits; its subjective nature among synaesthetes and its complete adherence to similarity. We need to be armed with a wider approach to cross-media works. Cook sees synaesthesia as potentially ‘enabling condition for multimedia, but not a sufficient one.’

    To analyse music is also to be committed to the idea that we perceive notes in terms of the relationships between them: we perceive each note as influencing, and being influenced by other notes—or at any rate, if we do not, it is hard to see what we could be analysing. In a nutshell, we analyse the interaction between the elements of the music: that is what analysing music means. And exactly the same applies to multimedia. To analyse something as multimedia is to be committed to the idea that there is some kind of perceptual interaction between its various individual components, such as music, speech, moving images, and so on: for without such interaction there is nothing to analyse.

The rest of the book is based on this idea of interaction and the idea of multimedia as metaphor. Cooks sees in media interaction the potential for emergence (the result of putting medias together as more than the sum of its parts) and sees the metaphor model as invoking ‘similarity not as an end, but as a means.’ Cook talks about semiotics, motion and gesture (something I might elaborate later on with my reading of The Sonic Self) and arrives at a basic model for the analysis of multimedia.

For a full understanding, read the book, but I’ll explain simply and inadequately. We start at the top and decide whether two media’s relationships are consistent or coherent. A colour organ or keyboard would be a simple and perfect example of a consistent relationship (a relationship he sees as quite rare in current multimedia) where as coherence allows for differential elaboration. The next distinction is a fuzzy one, which Cook acknowledges; he also acknowledges the potential for media forms to shift between classifications. These two quotes might help:

    Conformance begins with originary meaning, whether located within one medium or diffused between all; contest, on the other hand, ends in meaning. And the association of conformant models with synaesthestic and metaphysical speculation demonstrates, conformance tends towards the static and the essentialized, whereas contest is intrinsically dynamic and contextual.
    The term ‘contest’ is intended to emphasize the sense in which different media are, so to speak, vying for the same terrain, each attempting to impose its own characteristics upon the other. One might develop the analogy by saying that each medium strives to deconstruct the other, and so create space for itself.
    The mid-point between these two extremes is represented by the third model of multimeda, complementation, which Figure 3.1 represents in negative terms as that which exhibits neither consistency nor contradiction… complmentation is readily associated with the succesive phases of multimedia production. The classical Hollywood film for example, for instance, was in general virtually complete before it was passed onto the composer for scoring: the composers job was understood as one of complementing….

The main point I wanted to get to was Cook’s ideas on contestation. The possible repercussion being that that works such as mine; generative AV work (dynamic and cross imposing characteristics), form an AV relationship of contestation, rather than harmony. Cook would be useful in supporting the argument that such works are linked through metaphor, motion, gesture (and illusion) rather than similarity or some sort of natural harmony.