Unity Database

For starters, the server setup was ultimately a success, although I went down a few rabbit holes. Basically Unity has it's way of doing things and networking, a lot of this has to do with its use as a game engine (so a lot of multiplayer talk). I eventually found a solution for an independent server through Firebase, which has a lot of functionality and supports "noSQL" (they have JSON options). 

It's tough to document the process of setting up a server connection, so I'll share some error messages, hurdles and milestones along the way. The big moment was having the position values of a Unity object (sphere) update in the server live. So updating values a success - next up is figuring out how to parse incoming data so we can access video information. I'm imagining storing a bunch of videos on a seperate server (maybe squarespace) and send the url information based on the user interaction. 

Screen Shot 2018-03-28 at 9.38.32 PM.png
Screen Shot 2018-03-28 at 11.20.39 PM.png
Screen Shot 2018-03-28 at 11.57.04 PM.png
Screen Shot 2018-03-31 at 11.00.35 AM.png

Time Slices

I wanted to try something that's been on the back of my mind for some time. With AR and immersive media in general, I keep thinking what about this new medium is special and could be leveraged to show something that hasn't yet been possible. Immersive media has two things really going for it: depth (immersion/space) and time. So I started out making a story/environment that mixes time and space around a user. 

By bringing in video assets on an inverted sphere (as a panorama or skybox) and cutting that background panorama up into slices, you can create an immersive space that pays with space and time. As a simple example, imagine recording a 24 hr 360 video from the center of the Washington Square park fountain. Now take 1 minute slices of the video and evenly distribute it as a skybox around the inner circle of Washington Square Park, now you have slices of the day as a spacial installation. 

That's the idea that started this series of experiments. From there I realized if you brought in an "occlusion material" from Unity's assets, you can cut away video in interesting ways. You can make a truly experimental space when bringing in videos from other places and perspectives (as a hockney-esc collage). And finally, I looked into a Vimeo Unity SDK that allows (pro) members to stream live footage. So I have a lot of tools at my disposal and I'm seeing this as just the beginning of exploring what's possible with all of them. 

Screen Shot 2018-03-29 at 11.05.47 AM.png
Screen Shot 2018-03-29 at 11.05.57 AM.png
Screen Shot 2018-03-29 at 12.39.45 PM.png
Screen Shot 2018-03-29 at 12.43.09 PM.png

AR Sandbox


The multi-user AR experience will allow visitors to collaborate with each other in different virtual environments. Using a technique illustrated in the documentation video, Jaycee and I will create a number of portals or gateways that each lead to an entirely different environment with a unique activity. 

Currently we plan on creating three experiences/dimensions visitors walk into. The first is a simple 3D sculpture made of standard assets: cubes, spheres, and planes. Users will be able to select, move, place and delete these assets (and have the ability to change their color) to create a collaborative virtual object.

In a similar vein, we developing an environment where users can "paint" in 3D space and contribute to a larger virtual painting. Like Tilt Brush, users scribble, doodle and paint in this space and build off of one another's paintings. Since this space will likely get crowded very quickly, every stroke will be stored in an array where strokes are deleted in the order they were added when it becomes too busy. 

Inspired by the works of David Hockney, we want to develop a experience/dimension where users can capture images and save them as floating objects in space. The images can be of the room or selfies and will be stores in the position they were taken in. With enough images, an abstracted, fragmented sculpture will emerge in the same style as Hockney's collages. Hopefully, as images include the subject of other images, they will also resemble Yayoi Kusama's infinity rooms. 

To develop this experience we plan on using Unity3D with the ARKit plugin (of which we both have a solid understanding of). The final application will be a native iOS app and we plan on having at least two iOS devices (including my personal iPad). 

Screen Shot 2017-12-06 at 10.47.01 AM.png
Screen Shot 2017-12-06 at 10.47.17 AM.png
Screen Shot 2017-11-26 at 11.37.31 PM.png
Screen Shot 2017-11-26 at 11.37.54 PM.png

OpenCV as KNN


As an implementation of KNN, I wanted to set OpenCV's facial recognition coordinates as the query points. After experimenting with Sam Levine's video library VidPy, I first tried applying this to a video clip by accessing the pixel data as a nested for loop, where the KNN categories would change with every frame. I took a step back to better understand the process and settled on working with images. I'm going to open a github issue with cv2 after George Harrison wasn't recognized. 

Screen Shot 2017-11-20 at 2.19.51 PM.png

That's more like it

frame999932 copy.jpg

Music Video Battle


After seeing VidPy's "chroma" feature I knew I wanted to experiment with overlaps and color masks. This project took several detours as I tried exploring a few approaches and turned into an exercise (rather lesson) of thinking before doing. 

I first played around with openCV with the mindset that portraits would open some interesting avenues. i was excited to combine K Nearest Neighbor with OpenCV (where each face would be a point of interest) and have the chroma effects tied to each person. Sure enough simply figuring out numpy arrays was challenging enough and I settled on manually creating a chroma effect by reading and manipulating the pixels values of a video. 

I decided to tie the rewriting of pixel values with the volume of the audio from a music video. In each music video, the volume at any frame affects the amount of pixel information is masked in red. In a separate script, using VidPy's chroma feature, I then removed those masked areas and combined multiple videos in hopes there would be a push and pull between the videos as they fight over space. It ws an interesting experiment, but the resulting video falls flat. 

Screen Shot 2017-11-16 at 10.54.38 AM.png
Screen Shot 2017-11-16 at 12.54.20 PM.png
Screen Shot 2017-11-16 at 2.45.18 PM.png
frame143 copy.jpg
frame65 copy.jpg

Sol Lewitt Generator


In my first job out of college working on an online catalogue raisonne for artists represented by Pace Gallery. I was assigned to make the Sol Lewitt website and spent the better part of a summer finding every documented version of every Sol Lewitt (>1000) artworks. His work is programmatic in nature; they are essentially instructions that describe exactly how to make a drawing. For example: 

“On a wall surface, any continuous stretch of wall, using a hard pencil, place fifty points at random. The points should be evenly distributed over the area of the wall. All of the points should be connected by straight lines. “

These instructions have a formula to them and using Markov chains in rita.js I'd like to generate artificial Sol Lewitt titles. I've thought a lot about what to do with these titles, and decided to publish them as a twitter bot that called followers to draw and submit their interpretations of them. 

As a next step I'd like to create installations from these instructions. There have also been a number of people using Processing to recreate Sol Lewitt Wall Drawings - it makes sense as a procedural language. Although it's an impossible task to hard code an interpretation of each generated title. I attempted a processing sketch that adapted and changed with the parameters of the new title. 

The arguments I passed through to processing were based on a few common features throughout the titles. Using rita.js (rita.kwik) to simply check for words in the title, I looked for "color" (black, white, red) for the fill value, the "mark" (whether to use lines, points or arcs), and two booleans for "grid" and "corners" (to see if the sketch should use a grid and where to begin the marks). Needless to say the sketch is a birds nest of conflicting logic. 

Screen Shot 2017-11-30 at 6.55.48 PM.png
Screen Shot 2017-11-30 at 7.14.42 PM.png
Screen Shot 2017-12-03 at 4.57.29 PM.png
Screen Shot 2017-12-03 at 5.21.48 PM.png
Screen Shot 2017-12-03 at 5.40.14 PM.png
Screen Shot 2017-12-05 at 11.19.39 AM.png
Screen Shot 2017-12-05 at 12.08.15 PM.png
Screen Shot 2017-12-05 at 12.09.01 PM.png

Numpy Encoding


Looking through the JS implementation of deeplearn (deeplearn.js) I used a RNN/LSTM model to retrain my own data and see how it encoded the tensorflow weights for javascript. I decided to use the output from a video classifier as the input to this model, mostly because I thought it was interesting to train a ML model on another ML model. 

Screen Shot 2017-11-13 at 12.19.58 PM.png
Screen Shot 2017-11-13 at 12.19.35 PM.png
Screen Shot 2017-11-13 at 12.12.29 PM.png
Screen Shot 2017-11-13 at 12.14.57 PM.png
Screen Shot 2017-11-13 at 12.10.21 PM.png
Screen Shot 2017-11-13 at 12.11.32 PM.png
Screen Shot 2017-11-13 at 12.11.50 PM.png

ASL Translator


While playing around with OpenCV I was interested in hand gestures and looked to develop a tool that would recognize/translate American Sign Language gestures. I started with a simple script that recognized the number of digits by looking for a brightness threshold (finding the hand blob) and counted the negative spaces. 

This was a little limiting and I soon found a project by Shubham Gupta that recognized multiple hand shapes by training a ML model on a dataset of hundreds of images. It worked pretty well under the exact right lighting conditions, but the blob tracker was based on pixel color (of a skin tone) and didn't isolate the hand as well. So after updating some of the script to work with the latest version of OpenCV, I brought in the blob tracker from the digit counter so it would track based on threshold. 

Now that I have this up and running I'm looking to retrain the model on other gestures, maybe as controls for something else.

Mendelbrot 360


With the goal of creating an immersive environment, I set out to show recursion in a 360 video. I found a pretty common fractal zoom video of a mandelbrot as my source video file and began creating a function that mirrored the clip in four quadrants. The resulting file was the input for the same function so the original clip was recursively multiplied into many smaller, symmetrical copies of itself. I found the transitions between clips to be a little sudden and I began animating the clips to blend into the top left quadrant (where it's the same footage). 

Screen Shot 2017-11-02 at 10.30.40 AM.png

LSTM trained on Classifier


I was excited to start looking through deeplearing.js and refamiliarize myself with RNN/LSTM models. But first, I had an interesting detour with my old friend SketchRNN. I had the strange idea that I could repurpose the Inter.js sketch (on the new Magenta release of SketchRNN) to interpolate different fonts, instead of interpolating the original drawing models. 


Since this sketch relied on model weights to "mix" two different drawings, I thought the only way to make this work would be to train a model for each letter of each font. To interpolate a word with those fonts, I would separately draw each letter at different stages and combine them into the final word written as a mixed font. This seemed to be working, and after making each letter compatible with the updated SketchRNN model, I had one model trained and ready to go. Sampling this model took longer than expected though, so I figured it created too many "lines" from the SVG files of the letters. I decided to start from scratch - maybe I'm getting better at seeing a lost cause. (but first, I tried to manually mix them in p5). 


I headed back to deeplearning.js and decided to retrain an LSTM model with a video classifier as its input. My thinking was if a video had a predictable sequence, a user could predict future classifications of that video recording. With how seemingly random the classifier output was, I expect the LSTM model to create complete jibberish.

Screen Shot 2017-10-27 at 5.03.55 PM.png
Screen Shot 2017-10-28 at 12.48.54 PM.png
Screen Shot 2017-10-30 at 8.45.05 PM.png

Color Word2Vec


In experimenting with a JS based Word2Vec model, I had an ulterior motive to get comfortable with Three.js. The premise of this sketch was to visualize color gradients in 3D space and use a word2vec model to predict relationships between colors. The color tiles on the top bar of the screen represents a "this is to that, as this is to ___" prediction method, where a the final color is determined based on an object's position. The prediction works well - the user interface needs some work, especially in fine tuning the dragging/selection method. 

Screen Shot 2017-10-23 at 4.44.41 PM.png
Screen Shot 2017-10-23 at 8.08.22 PM.png
Screen Shot 2017-10-24 at 8.15.38 AM.png
Screen Shot 2017-10-24 at 12.20.55 AM.png
Screen Shot 2017-10-24 at 10.20.19 AM.png

Wireshark Capture


I took a deeper look into my network traffic using the network analysis tools Herbivore and Wireshark. At first glance a Wireshark capture looks a little intimidating and unfamiliar, so first task was exporting the capture data as a CSV file and looking at some of the variables in javascript. It didn't tell me too much except that the majority of my packets were using TCP. Herbivore gave a more user friendly view of the route of your internet activity. Looking into unfamiliar domains that popped up in my normal internet use, I saw a few examples of cookies, a mobile safari ad tracking software called Kochava, and most interestingly a url that pointed to a webpage with a single haiku. (Looking up this poem returns a website, Addthis.com, that wants you to download some files to help with your "security"...).   

Screen Shot 2017-10-16 at 7.50.42 PM.png
Screen Shot 2017-10-16 at 8.04.36 PM.png
Screen Shot 2017-10-16 at 8.09.39 PM.png
Screen Shot 2017-10-16 at 9.35.50 PM.png
Screen Shot 2017-10-15 at 12.58.26 PM.png
Screen Shot 2017-10-15 at 1.03.15 PM.png
Screen Shot 2017-10-16 at 9.40.56 PM.png
Screen Shot 2017-10-16 at 7.25.09 PM.png
Screen Shot 2017-10-15 at 1.14.33 PM.png
Screen Shot 2017-10-16 at 9.44.10 PM.png



Gradient Plane


For my first experiment with Unreal Engine I aimed to create a simple immersive experience based on the works of time/space artists like James Turrell or Robert Irwin and to get a handle on working with sound in a game engine. On a reflective surface, I positioned different colored lights to mimic a CYMK color wheel and added steam particle systems to give the light more body. 

Screen Shot 2017-10-04 at 3.27.05 PM.png
Screenshot (14).png
Screenshot (9).png

Caustic Waves

link for mobile

(best on mobile with headphones)


Going a little deeper with exploring 360 video, I wanted to make an immersive dreamscape based on an experience that stuck with me over the years. To bring in some effects I needed for the transitions I combined a few After Effects compositions in made with found footage and Ricoh Theta videos I took. I chose Premiere to edit these scenes together with a score made in Reaper. (If you're interested, in my dream the glowing light at the end represented all of time and space condensed into a single, floating bar of light).

Screen Shot 2017-09-27 at 9.16.32 PM.png
Screen Shot 2017-09-27 at 1.23.12 PM.png

(if you prefer YouTube).

Voyager 360 Spatial Audio with Video

link for mobile

(best on mobile with headphones)


To become more familiar with Facebook's Spacial Workstation for 360 audio, I chose to make a sound piece on the Voyager mission (with many opportunities for panning). As I was researching the mission, I came across a RadioLab interview with Ann Druyan, Carl Sagan's wife, and a hidden love story slowly emerged. Ann was directing the content for the Golden Record representing our world and culture, when she and Carl got engaged. Days afterwards, her vitals were recorded as she was thinking about Carl Sagan, and those recordings were included in the record on the Voyager spacecraft. The sound clips I used in editing this piece were all sampled from the same Golden record. For the video, I recorded a 360 video with a Theta from the inside of a globe/bar I found on the sidewalk. I edited the video in After Effects to create the illusion of looking at the stars from inside the globe (minus some masking/keying issues). 

Screen Shot 2017-09-20 at 6.17.08 PM.png
Screen Shot 2017-09-20 at 6.17.22 PM.png

Network Hops



live map

As an exercise to better understand the traceroute terminal command, I used mapbox and an IP lookup API to track the locations of the routers and servers used in my frequently visited websites. Unfortunately facebook is among them (marked as the death icon), as well as gmail (rocketship icon) and my personal website (markered with a sleeping icon since I need sleep). In the javascript console, I'm logging the complete json returned by the IP API, including the company name, city of the server, and order in which it was sent. 

Screen Shot 2017-09-17 at 12.50.09 PM.png
Screen Shot 2017-09-18 at 8.25.42 PM.png
Screen Shot 2017-09-18 at 8.53.21 PM.png
Screen Shot 2017-09-18 at 8.55.03 PM.png
Screen Shot 2017-09-18 at 11.30.59 PM.png
Screen Shot 2017-09-19 at 2.40.51 PM.png
Screen Shot 2017-09-24 at 1.35.43 PM.png
Screen Shot 2017-09-24 at 3.26.54 PM.png
Screen Shot 2017-09-25 at 11.19.28 AM.png

Regex Image


As a supplement to my experiment The Last Question, I set out to generate an image with regex characters and symbols. I kept the short story generation in the first iteration, and added a function that *attempts* to redraw an uploaded file with characters based on the image pixel value. The characters are selected based on their pixel value. 

Screen Shot 2017-09-19 at 10.24.20 AM.png
Screen Shot 2017-09-19 at 10.31.19 AM.png
Screen Shot 2017-09-19 at 10.31.33 AM.png
Screen Shot 2017-09-19 at 10.33.43 AM.png
Screen Shot 2017-09-19 at 10.37.56 AM.png
Screen Shot 2017-09-19 at 10.40.01 AM.png
Screen Shot 2017-09-19 at 2.19.56 PM.png
Screen Shot 2017-09-19 at 2.20.20 PM.png

Voyager AR with BlippAR


In this straightforward exercise with BlippAR, I used the stereo version of a 360 sound piece I recently created in the style of a Radio Lab episode. In researching the Voyager mission I stumbled upon the love story between Carl Sagan and Ann Druyan and decided that would be a more interesting focus. After some testing with BlippAR, I chose an image of the Golden Record (containing recordings of what humans are all about) as a marker for a 3D model of the Voyager and the sound piece. 

Screen Shot 2017-09-16 at 6.59.04 PM.png

Arcade Joystick Controller


With the requirements to create a controller for Left, Right, Up, Down, and Quit controls, I set up a SparkFun switch with an arduino and neopixels highlighting selected directions. The housing was a departure from the typical "ITP box" (a bamboo box from the Container Store with an acrylic, laser etched lid), instead I aimed to reuse materials from the junk shelf. 


The Last Question


live website

This website was an experiment in accessing DOM elements in p5 and generating text with Rita.js. Once an image is uploaded, the RGB pixel value of each pixel determines the generated text sourced from Isaac Asimov's short story, "The Last Question". I mapped the combined RGB value of each pixel to the order of words as they appear in the short story. The darker the pixel, the further the word lives in the text.

Screen Shot 2017-09-10 at 6.57.55 PM.png
Screen Shot 2017-09-12 at 1.02.28 AM.png
Screen Shot 2017-09-12 at 12.23.51 AM.png
Screen Shot 2017-09-12 at 2.07.44 AM.png