Hectic times

October 8, 2007

Exegesis and audiovisual work due in a matter of weeks. Terrifying for two reasons:

    1. My HDD died. Partition stuffed. Some pictures, documents, video and music (expensively) recovered, but all my code is gone.

    2. I’ve just started full-time work at a wonderful place called Icelab.

Although coding again from scratch is frustrating, it’s most likely a good thing. The cleanliness and quality of my code will surely be better for it. Thought my computer would be happier with a fresh, clean hard drive and a re-install of windows, but no, its surely dying.

So rather than wrapping up my honours work I’ve been at work learning Motion 3 and how to build websites properly. Expect a decently designed blog at some stage.

After getting my head around Motion’s 3D navigation I discovered a really impressive piece of software. Purely as an animation or graphics tool its daunting, but its video features and connectivity with Final Cut make for a piece of software with potential that I am yet to even grasp. Playing with nice macs and flash new software is fun.

My Reas/Fry processing book came in the mail and it’s fantastic. If only I had it at the start of the year. Countless small programming concepts I’ve spent hours frustrating over are explained concisely and effortlessly. Solid hard-cover, well laid out, and cleverly structured. I shipped to Australia from Barnes and Noble. Took about two weeks, but it was far cheaper than buying here.

Finally did a little bit of programming tonight. Some results:





September 4, 2007

Currently writing exegesis rather than blog posts.

Another clue as to how my animation might look. This time with paint instead of pencil:

    Considering arts techniques from the broad perspective of the present, I observed that the best “computer art” did not compare
    well with lacework from Belgium made a century ago. But the computer possessed a unique capability of making very complex pattern flow. One could plan exacting and explicit patterns of action and distinctive motions as intricate as lace, but in a way no Belgian lace maker would ever imagine. – John Whitney, 1980.

This 1975 film is reportedly John Whitney’s first foray into computer graphics. Until ‘Arabesque’, Whitney used a converted mechanism of a World War II M-5 Antiaircraft Gun. Essentially a twelve-foot-high analog computer of amazing complexity; where design templates were placed on three different layers of rotating tables and photographed by multiple-axis rotating cameras.

In Digital Harmony (1980), the book that describes his life’s work, his hypothesis –

    …assumes the existence of a new foundation for a new art. It assumes a broader context in which Pythagorean laws of harmony operate. These laws operate in a graphic context parallel to the established context of music. In other words, the hypothesis assumes that the attractive and repulsive forces of harmony’s consonant/dissonant patterns function outside the dominion of music.

Whitney acknowledges that, ‘Music does not need images any more than paintings need sound’ but saw in computing, ‘a visual medium which is more malleable and swifter than musical airwaves. That medium is light itself.’

The book often communicates personal opinion rather than rigorous argument but Whitney makes some original and interesting points. It seems Whitney is not really pursuing visualisation or a tightly fused AV form. Whitney’s search is instead for abstract graphics with the fluidity, expressiveness and structural qualities of music. Whitney begins the book by highlighting the inherent spatial and visual qualities of music and damning early ‘visual music’ inventions:

    Most people visualize music as two-dimensional, with time represented by the horizontal lines and pitch by vertically arrayed symbols, as is the convention on paper. But the perception of music is not two dimensional. The ears reside at the center of a spherical domain. We hear from all around. We hear music as patterns of ups and downs, to and fro in a distinctly three-dimensional space – a space within.

    The eye, more outwardly oriented, perceives objects and events outside at the point where our eyes focus. Yet the eye enjoys design equally as well as the ear. The mind’s eye shares with the ear any inward experience of architectonic spatial constructions and would perceive them with the same pleasure, were they to exist.

    The fact is, however, that these interior fluid visual edifices hardly exist. Anyone can visualize an architectural fantasy of music dancing in the head, but manifesting in reality is another matter! Each century since Leonardo, a vision, grand and obscure as its myth, compelled one or two inventors to struggle with the pathetic inadequacies of the color organ. Twentieth-century abstract art has been a training ground for visual response to musical experience, but in the mind’s eye, architecture in motion lies at the root of our enjoyment of music. Many people, with closed eyes at a concert, are “watching” the music, but after all these centuries, there still exists no universally acceptable visual equivalent to music! It should exist and it will soon.

Whitney also documents his and others failed attempts at experimental film based endeavours:

    Pointing their cameras at the world, all those “symphonists” inadvertently recorded the stasis of the world, even as they filmed its busiest moments – its winds and storms and birds and water and city traffic. Those films are not symphonies, I thought, poetry perhaps, but not liquid architecture, not music.

    …wherever I pointed my camera, I failed to discover that special quality of any material possessing the controllable visual fluidity that I desired … pointing my camera anywhere resulted in recording images of somewhere. If the camera’s record is unclear, blurred by the smear of too fast panning or being out of focus, the sense of somewhere as place is simply flattened. The spatial content of an image is flattened. The eye resists the attempt to domesticate abstraction. This sort of deception hardly satisfies the eye, because the sense of being (or seeing) somewhere is so strong. The eye is the natural master of pattern recognition. The eye demands satisfaction by invoking in us strong feelings of puzzlement.

And makes the important point that, “No abstraction in my camera had the generative potential, the capability to propagate fluid patterns or especially, the liquid variability of the intervallic families of music tones.”

This is where the computer comes into play and Whitney’s argument gets interesting. Whitney sees a parallel between musical tones and generative animation. Whitney sees music as an abstract and generative form in itself:

    There is no such thing as the harmonic organization of musical tone in nature. Occasionally a stone may ring like a bell, birds pattern “song,” but there are few natural bells, fewer natural flues where the winds sound organ tones. Even the whistle of the wind is eerie and non-musical. Patterning of musical tones is a man-made reality of the aural world, universally accepted as such, but nowhere looked upon as an abstraction that has been extracted (or abstracted) out of the natural environment, nowhere regarded as a manifestation of the environment.

Whitney in deciding that music is not an abstracted picture of anything, allows for his second level of pure abstraction and generation. He focuses on three qualities applicable to both forms:

    A benchmark was reached when I began to apprehend the relationship of the three terms: differential, resonance and harmony. First, motion becomes pattern if objects move differentially. Second, a resolution to order in patterns of motion occurs at points of resonance. And third, this resolution at resonant events, especially at the whole number ratios, characterizes the differential resonant phenomena of visual harmony.

    What I knew about music confirmed for me that emotion derives from the force-fields of musical structuring in tension and motion. Structured motion begets emotion. This, now, is true in a visual world, as it is a truism of music.

Digital Harmony, the documentation of a life’s work is the most comprehensive study of generative animation and its musical potential that I have found yet. It provides some useful counterpoints when compared to Chion’s deconstruction of audio visual relations. A simple reading of Chion would state that audio is predominately temporal while vision is predominately spatial but Whitney’s musical ‘liquid architecture’ metaphor is a wonderful one. Regardless, I’m starting to side with Chion’s idea of ‘audiovisual illusion’ and perhaps through a lifetime of work and focus, Whitney has merely become a better magician.

This is not to say Whitney is wasting his time. Magic is an art form. This also doesn’t devalue his ideas of visual consonance, dissonance, harmony and disharmony. A work where consonance and dissonance is linked between audio and visual, temporally and structurally without doubt creates moments of audio-visual resonance. These ideas are particularly interesting in regards to my choice of song and visual aesthetic.

Evil and spikey.

July 24, 2007

Hand drawn element sketch no. 3 (click to enlarge)

Hand-drawn elements

July 23, 2007

First attempt at using hand-drawn elements. Took a little more care with the etching and transparency settings. (Click to enlarge)

In motion.

July 11, 2007

Sorry about the quality. I just couldn’t be bothered with setting up for rendering, rendering, loading into after effects, and rendering again….

Transparent GIFs

July 11, 2007

I’ve been thinking about using GIF files as elements for a while now and finally got around to it. Was an easy change to make but boy does it make a difference. Particularly in motion, the result has an interesting cut-out effect suggesting conventional animation or motion graphics, rather than the typical processing look.

The aesthetic possibilities are endless but my generative potential is somewhat hampered. I’ll have a play with pixel manipulation to see what I can come up with. A colour-sound relationship could be fun.

Next I’ll insert some hand-drawn elements so that things look less ‘cartoony’.

New recursive structure

June 26, 2007

The first song, ‘The Power of Independent Trucking’ by Big Black, the second, ‘Sporting Life’, by Young Marble Giants.

YouTube absolutely destroyed the quality. I’d recommend you watch it on Vimeo where it looks slightly better:

Render Test 2: Structure 2 from njmcgee on Vimeo

(Vimeo won’t embed in a wordpress.com hosted blog (and doesn’t look that good anyway). I’ll be thinking about a way to get better quality video embedded. Any ideas? Let me know.)

Best results so far with heavy rock music. Moments of real rhythm, and surprisingly dynamic visual response. The exponential relationship between element size and fft values allows individual legs to be simultaneously effected by their corresponding spectrum band with great effect. I will probably tone things down – reduce the number of elements and better control over the overall scale. Oh, and for some reason one of the legs gets stuck…

Thankfully, as I continue to bash out ‘programs’, complex visual outcomes I have not, or could not have planned are emerging.
As I tip toe towards generativity I’m wondering how to approach a ‘final’ piece – considering the unique movements, forms and reactions each time sketch is run.

Windowed Averaging

June 18, 2007

Recently I’ve been keeping things very simple – eight frequency bands driving all my graphics. My previous experiments used 64 bands, split into four groups of sixteen with Ess kindly working out the average for these groups. My past methods were based completely on event triggering – when a spectrum band peaks above its group’s average by a certain amount something happens.

I’ll definitely work a feature like this back into the mix. If tamed correctly it finds drum hits (particularly the snare) and highlights big dynamic changes really well.

For now, I’ve stripped things back to find more complex ways of interpreting these spectrum values. The plan is to have these always changing spectrum values controlling movement and speed. Turns out it’s quite tricky – even with audio dampening, getting smooth results will take a bit more work. Until then I’ve been thinking up ways to visually highlight structural change beyond a keyboard or time based trigger.

Here is St. Anthony’s Fire (the song I will be using, right click to download) split into eight frequency bands and averaged every 60 frames (two seconds). Not exactly a sexy piece of data visualisation I know but useful enough.

(right click to see a larger version)

The song is very heavy and a total mess of frequencies but windowed averaging has made the structural changes quite clear to the trained eye. I’ve circled the points in which I could set thresholds or even better link a windowed average to a particular variable to get a nice gradual transition rather than an instant shift. Hopefully I can give these six sections a completely different look/structure and provide smooth transitions to create a clip that’s always flowing and moving. Generative animation’s beauty is its capacity to provide output, shapes and movements beyond the programmer’s initial intention – visuals that are somewhat ‘alive’. I’d like to turn the visuals into a character of sorts, a complex and wild beast that twists, turns and changes with the music. Avoiding hard cuts and instant transitions will help the visuals to live and breathe.

Offline rendering

June 14, 2007

All my processing based AV experiments to date have been in real time. In preparation for my dissertation piece I thought I should get rendering. This post at the processing forums was immensely helpful. This Davbol fella has given many budding flight 404s a nice head start.

With Davbol’s structure –


I worked in my still very rough leech code and had things working surprisingly quickly. At 320 x 240 it took over ten minutes to render about 3800 jpegs. Loaded the images into after effects, dropped in the audio and everything synced nicely.

Finally, I could see the sketch running smoothly but unfortunately after all that waiting, the problems that real time chunkiness were hiding became apparent. Everything was flashing and blisteringly fast so I had to slow down all my motion and increase the audio dampening to settle things down.

Whenever I make a programming breakthrough it seems inevitable that along with it comes a set of new tasks and a haunting and realistic vision of the workload ahead.

I’ll have to find a way to work in real time and with offline rendering. Real time for experimenting with new forms and structures and offline to fine tune values and variables. Sitting around and waiting for things to render just seems so strange and inefficient after a year and a half of playing with this stuff in real time. I guess its the price you pay for detail. Time to refine this leech code and get it reacting in a smooth and dynamic manner.