Creating Light From Music

I have long been a fan of Robot Koch - The German producer/composer who's known for creating futuristic sound with human character. His work has been featured on NBC's "The Blacklist," ABC's "How To Get Away With Murder," and Sony's new "Earth Rising" campaign for the Bravia OLED TV.

 Producer Aniela Sidorska and I approached Robot Koch about collaborating on his first single "You" from his brand new EP called Fortnite - featuring vocalist Fiora. 

The music video was done in ‘one breath’—using primarily in-camera effects with projection mapping that envelops the model in patterns of light, stars and darkness. The video contains all practical elements with no CGI. The light was created solely from the music itself, and contains every frequency of the instruments and vocals, in sync with the song.

The video was filmed with a robotic motion-control camera to appear as one single-shot to showcase the fluidity of the composition. What we attempted to capture in the relationship of the projections and performance is both the meaning of the lyrics, as well as the notion that we feel at one with the music.

On one side, we see the vocals being projected, on the other the bass. After we pass the point of looking at one’s own reflection, the projections switch, and we see the bass on the left hand side—almost acting like the heartbeat, not only of the woman but also of the song itself.



2001 Flying Thru Space 01_BB.jpg

We began by creating a series of look development tests for the video's aesthetic. Aniela and I wanted to create a visual palette with a simple and very unique look that could help us tell a story about the theme of self transformation in the lyrics. I had always wanted to figure out a way to turn sound into light - very directly - and thought this would be a good opportunity to explore that. Robot Koch provided the individual stems for "You" and we processed each separate track through a number of procedural visualization techniques - mapping frequencies to pixels - ultimately landing on spectrograms as the most visually interesting starting point. Since seeing 2001: A Space Odyssey as a kid, I had been mesmerized by Douglas Trumbull's Star Gate Sequence, which utilized slit-scan photography.  The idea in slit-scan is essentially to take a spatial axis and stretch it through time, creating a psychedelic flow of light. This turned out to be a perfect fit for this project, as it allowed us to create a sort of 'fountain of light' out of which the vocals, bass and synthesizers flowed across the model's face. After using Nuke to process the spectrograms into slit-scan videos, Aniela and I projected them using flicker-free LCD video projectors. With each test, we determined the best projector placement, color correction and speed of the light flows. Moving Star Fields - created from glitter in vegetable oil - were also added to the projections. The stars allowed us to feel the shape of the face even when there was a pause in the vocals, which would have otherwise turned pure black. Through these star elements, we were able to create the idea of 'hiding in plain sight' that was so central to the lyrics. 







After numerous rounds of testing we were finally happy with the look of the projections: a pure conversion of music into light. But at this point we realized that the video needed something else. The song to us was about 'different versions' of self and we set about representing this visually. We discovered that by projecting the vocals onto the model, and then moving her and filming her a second time with the bass, we could create a sort of conversation: two selves that would coexist in the frame and be able to appear to interact. Then as the parts of the music ebb and flow, the different 'selves' appear and disappear, representing the metamorphosis. We achieved this using a 8-axis motion control camera system, synched with the video projectors. With this system we would just press a button, and everything would start: the song played, the camera moved and the light flowed in perfect synch. Each shot was filmed in several passes with our super talented, beautiful model, Ashleigh Baugh. Tracking marker passes would later allow us to incorporate additional BG star elements on 3D cards in Nuke. We composited the passes together while shooting - to not only ensure post would be a smooth process - but to guide Ashleigh's performance, eye-lines and blocking. Director of Photography Mike Misslin and I programmed the camera movements, and Mike finessed the projector angles: the only lights used, except for the green-screen in the final shot. Aniela and I collaborated with Choreographer Melody Sample to create a performance that showed an evolution of self: from very timid and shy, to curious, to expressive and confident and finally peaceful and enlightened. We decided to project all the graphics with a red color palette - since that is more flattering on skin - and then remapped the colors to blues in post.