OpenCV as KNN

About

As an implementation of KNN, I wanted to set OpenCV's facial recognition coordinates as the query points. After experimenting with Sam Levine's video library VidPy, I first tried applying this to a video clip by accessing the pixel data as a nested for loop, where the KNN categories would change with every frame. I took a step back to better understand the process and settled on working with images. I'm going to open a github issue with cv2 after George Harrison wasn't recognized. 


frame9140.jpg
frame8690.jpg
itpknn.jpg
Screen Shot 2017-11-20 at 2.19.51 PM.png
faceoffknn.jpg
beatlesknn.jpg

That's more like it

beatlesknn2.jpg

Music Video Battle

About

After seeing VidPy's "chroma" feature I knew I wanted to experiment with overlaps and color masks. This project took several detours as I tried exploring a few approaches and turned into an exercise (rather lesson) of thinking before doing. 

I first played around with openCV with the mindset that portraits would open some interesting avenues. i was excited to combine K Nearest Neighbor with OpenCV (where each face would be a point of interest) and have the chroma effects tied to each person. Sure enough simply figuring out numpy arrays was challenging enough and I settled on manually creating a chroma effect by reading and manipulating the pixels values of a video. 

I decided to tie the rewriting of pixel values with the volume of the audio from a music video. In each music video, the volume at any frame affects the amount of pixel information is masked in red. In a separate script, using VidPy's chroma feature, I then removed those masked areas and combined multiple videos in hopes there would be a push and pull between the videos as they fight over space. It ws an interesting experiment, but the resulting video falls flat. 

Screen Shot 2017-11-16 at 10.54.38 AM.png
Screen Shot 2017-11-16 at 12.54.20 PM.png
Screen Shot 2017-11-16 at 2.45.18 PM.png
frame143 copy.jpg
frame65 copy.jpg

Sol Lewitt Generator

About

In my first job out of college working on an online catalogue raisonne for artists represented by Pace Gallery. I was assigned to make the Sol Lewitt website and spent the better part of a summer finding every documented version of every Sol Lewitt (>1000) artworks. His work is programmatic in nature; they are essentially instructions that describe exactly how to make a drawing. For example: 

“On a wall surface, any continuous stretch of wall, using a hard pencil, place fifty points at random. The points should be evenly distributed over the area of the wall. All of the points should be connected by straight lines. “

These instructions have a formula to them and using a LSTM model I'd like to generate artificial Sol Lewitt titles. Ideally I'd like this experience to be a live interaction where the title reflects something about the surrounding space. I'm planning on using a classifier included in ML core in an application that feeds the LSTM local information as seed text to generate the title. 


download.jpg
images.jpg
download.png
19110.jpg
229-Diagram.jpg
Lewitt_AM38_l.jpg
lewitt-instructions-1-571x790.jpg

Numpy Encoding

About

Looking through the JS implementation of deeplearn (deeplearn.js) I used a RNN/LSTM model to retrain my own data and see how it encoded the tensorflow weights for javascript. I decided to use the output from a video classifier as the input to this model, mostly because I thought it was interesting to train a ML model on another ML model. 

Screen+Shot+2017-10-30+at+7.51.55+PM.png
Screen+Shot+2017-10-30+at+8.45.05+PM.png
Screen Shot 2017-11-13 at 12.19.58 PM.png
Screen Shot 2017-11-13 at 12.19.35 PM.png
Screen Shot 2017-11-13 at 12.12.29 PM.png
Screen Shot 2017-11-13 at 12.14.57 PM.png
Screen Shot 2017-11-13 at 12.10.21 PM.png
Screen Shot 2017-11-13 at 12.11.32 PM.png
Screen Shot 2017-11-13 at 12.11.50 PM.png

ASL Translator

About

While playing around with OpenCV I was interested in hand gestures and looked to develop a tool that would recognize/translate American Sign Language gestures. I started with a simple script that recognized the number of digits by looking for a brightness threshold (finding the hand blob) and counted the negative spaces. 

This was a little limiting and I soon found a project by Shubham Gupta that recognized multiple hand shapes by training a ML model on a dataset of hundreds of images. It worked pretty well under the exact right lighting conditions, but the blob tracker was based on pixel color (of a skin tone) and didn't isolate the hand as well. So after updating some of the script to work with the latest version of OpenCV, I brought in the blob tracker from the digit counter so it would track based on threshold. 

Now that I have this up and running I'm looking to retrain the model on other gestures, maybe as controls for something else.

Mendelbrot 360

About

With the goal of creating an immersive environment, I set out to show recursion in a 360 video. I found a pretty common fractal zoom video of a mandelbrot as my source video file and began creating a function that mirrored the clip in four quadrants. The resulting file was the input for the same function so the original clip was recursively multiplied into many smaller, symmetrical copies of itself. I found the transitions between clips to be a little sudden and I began animating the clips to blend into the top left quadrant (where it's the same footage). 

Screen Shot 2017-11-02 at 10.30.40 AM.png

LSTM trained on Classifier

About

I was excited to start looking through deeplearing.js and refamiliarize myself with RNN/LSTM models. But first, I had an interesting detour with my old friend SketchRNN. I had the strange idea that I could repurpose the Inter.js sketch (on the new Magenta release of SketchRNN) to interpolate different fonts, instead of interpolating the original drawing models. 

 

Since this sketch relied on model weights to "mix" two different drawings, I thought the only way to make this work would be to train a model for each letter of each font. To interpolate a word with those fonts, I would separately draw each letter at different stages and combine them into the final word written as a mixed font. This seemed to be working, and after making each letter compatible with the updated SketchRNN model, I had one model trained and ready to go. Sampling this model took longer than expected though, so I figured it created too many "lines" from the SVG files of the letters. I decided to start from scratch - maybe I'm getting better at seeing a lost cause. (but first, I tried to manually mix them in p5). 

 

I headed back to deeplearning.js and decided to retrain an LSTM model with a video classifier as its input. My thinking was if a video had a predictable sequence, a user could predict future classifications of that video recording. With how seemingly random the classifier output was, I expect the LSTM model to create complete jibberish.

Screen Shot 2017-10-27 at 5.03.55 PM.png
Screen Shot 2017-10-28 at 12.48.54 PM.png
Screen Shot 2017-10-30 at 8.45.05 PM.png

Color Word2Vec

About

In experimenting with a JS based Word2Vec model, I had an ulterior motive to get comfortable with Three.js. The premise of this sketch was to visualize color gradients in 3D space and use a word2vec model to predict relationships between colors. The color tiles on the top bar of the screen represents a "this is to that, as this is to ___" prediction method, where a the final color is determined based on an object's position. The prediction works well - the user interface needs some work, especially in fine tuning the dragging/selection method. 

Screen Shot 2017-10-23 at 4.44.41 PM.png
Screen Shot 2017-10-23 at 8.08.22 PM.png
Screen Shot 2017-10-24 at 8.15.38 AM.png
Screen Shot 2017-10-24 at 12.20.55 AM.png
Screen Shot 2017-10-24 at 10.20.19 AM.png

Wireshark Capture

About

I took a deeper look into my network traffic using the network analysis tools Herbivore and Wireshark. At first glance a Wireshark capture looks a little intimidating and unfamiliar, so first task was exporting the capture data as a CSV file and looking at some of the variables in javascript. It didn't tell me too much except that the majority of my packets were using TCP. Herbivore gave a more user friendly view of the route of your internet activity. Looking into unfamiliar domains that popped up in my normal internet use, I saw a few examples of cookies, a mobile safari ad tracking software called Kochava, and most interestingly a url that pointed to a webpage with a single haiku. (Looking up this poem returns a website, Addthis.com, that wants you to download some files to help with your "security"...).   


Screen Shot 2017-10-16 at 7.50.42 PM.png
Screen Shot 2017-10-16 at 8.04.36 PM.png
Screen Shot 2017-10-16 at 8.09.39 PM.png
Screen Shot 2017-10-16 at 9.35.50 PM.png
Screen Shot 2017-10-15 at 12.58.26 PM.png
Screen Shot 2017-10-15 at 1.03.15 PM.png
Screen Shot 2017-10-16 at 9.40.56 PM.png
Screen Shot 2017-10-16 at 7.25.09 PM.png
Screen Shot 2017-10-15 at 1.14.33 PM.png
Screen Shot 2017-10-16 at 9.44.10 PM.png

 

 

Gradient Plane

About

For my first experiment with Unreal Engine I aimed to create a simple immersive experience based on the works of time/space artists like James Turrell or Robert Irwin and to get a handle on working with sound in a game engine. On a reflective surface, I positioned different colored lights to mimic a CYMK color wheel and added steam particle systems to give the light more body. 

Screen Shot 2017-10-04 at 3.27.05 PM.png
Screenshot (14).png
Screenshot (9).png

Caustic Waves

link for mobile

(best on mobile with headphones)

About

Going a little deeper with exploring 360 video, I wanted to make an immersive dreamscape based on an experience that stuck with me over the years. To bring in some effects I needed for the transitions I combined a few After Effects compositions in made with found footage and Ricoh Theta videos I took. I chose Premiere to edit these scenes together with a score made in Reaper. (If you're interested, in my dream the glowing light at the end represented all of time and space condensed into a single, floating bar of light).


Screen Shot 2017-09-27 at 9.16.32 PM.png
Screen Shot 2017-09-27 at 1.23.12 PM.png

(if you prefer YouTube).

Voyager 360 Spatial Audio with Video

link for mobile

(best on mobile with headphones)

About

To become more familiar with Facebook's Spacial Workstation for 360 audio, I chose to make a sound piece on the Voyager mission (with many opportunities for panning). As I was researching the mission, I came across a RadioLab interview with Ann Druyan, Carl Sagan's wife, and a hidden love story slowly emerged. Ann was directing the content for the Golden Record representing our world and culture, when she and Carl got engaged. Days afterwards, her vitals were recorded as she was thinking about Carl Sagan, and those recordings were included in the record on the Voyager spacecraft. The sound clips I used in editing this piece were all sampled from the same Golden record. For the video, I recorded a 360 video with a Theta from the inside of a globe/bar I found on the sidewalk. I edited the video in After Effects to create the illusion of looking at the stars from inside the globe (minus some masking/keying issues). 


Screen Shot 2017-09-20 at 6.17.08 PM.png
Screen Shot 2017-09-20 at 6.17.22 PM.png
IMG_5771.JPG
IMG_5769.JPG
unnamed.jpg
unnamed-2.jpg
IMG_5782.JPG

Network Hops

Embed

About

live map

As an exercise to better understand the traceroute terminal command, I used mapbox and an IP lookup API to track the locations of the routers and servers used in my frequently visited websites. Unfortunately facebook is among them (marked as the death icon), as well as gmail (rocketship icon) and my personal website (markered with a sleeping icon since I need sleep). In the javascript console, I'm logging the complete json returned by the IP API, including the company name, city of the server, and order in which it was sent. 


Screen Shot 2017-09-17 at 12.50.09 PM.png
Screen Shot 2017-09-18 at 8.25.42 PM.png
Screen Shot 2017-09-18 at 8.53.21 PM.png
Screen Shot 2017-09-18 at 8.55.03 PM.png
Screen Shot 2017-09-18 at 11.30.59 PM.png
Screen Shot 2017-09-19 at 2.40.51 PM.png
Screen Shot 2017-09-24 at 1.35.43 PM.png
Screen Shot 2017-09-24 at 3.26.54 PM.png
Screen Shot 2017-09-25 at 11.19.28 AM.png

Regex Image

About

As a supplement to my experiment The Last Question, I set out to generate an image with regex characters and symbols. I kept the short story generation in the first iteration, and added a function that *attempts* to redraw an uploaded file with characters based on the image pixel value. The characters are selected based on their pixel value. 


Screen Shot 2017-09-19 at 10.24.20 AM.png
Screen Shot 2017-09-19 at 10.31.19 AM.png
Screen Shot 2017-09-19 at 10.31.33 AM.png
Screen Shot 2017-09-19 at 10.33.43 AM.png
Screen Shot 2017-09-19 at 10.37.56 AM.png
Screen Shot 2017-09-19 at 10.40.01 AM.png
Screen Shot 2017-09-19 at 2.19.56 PM.png
Screen Shot 2017-09-19 at 2.20.20 PM.png

Voyager AR with BlippAR

About

In this straightforward exercise with BlippAR, I used the stereo version of a 360 sound piece I recently created in the style of a Radio Lab episode. In researching the Voyager mission I stumbled upon the love story between Carl Sagan and Ann Druyan and decided that would be a more interesting focus. After some testing with BlippAR, I chose an image of the Golden Record (containing recordings of what humans are all about) as a marker for a 3D model of the Voyager and the sound piece. 


Screen Shot 2017-09-16 at 6.59.04 PM.png
golden-record-cover-800x438.jpg

Arcade Joystick Controller

About

With the requirements to create a controller for Left, Right, Up, Down, and Quit controls, I set up a SparkFun switch with an arduino and neopixels highlighting selected directions. The housing was a departure from the typical "ITP box" (a bamboo box from the Container Store with an acrylic, laser etched lid), instead I aimed to reuse materials from the junk shelf. 


IMG_5743.jpg
IMG_5757.JPG
IMG_5760.jpg
IMG_5761.jpg
1O3A9136.JPG
1O3A9128.JPG

The Last Question

About

live website

This website was an experiment in accessing DOM elements in p5 and generating text with Rita.js. Once an image is uploaded, the RGB pixel value of each pixel determines the generated text sourced from Isaac Asimov's short story, "The Last Question". I mapped the combined RGB value of each pixel to the order of words as they appear in the short story. The darker the pixel, the further the word lives in the text.


Screen Shot 2017-09-10 at 6.57.55 PM.png
Screen Shot 2017-09-12 at 1.02.28 AM.png
Screen Shot 2017-09-12 at 12.23.51 AM.png
Screen Shot 2017-09-12 at 2.07.44 AM.png

Submarine Sound Piece

About

To get started in 360 sound design, we first took a look at Reaper and used it to construct a narrative with audio only. I wanted to play around with created different spaces with different background ambiances and saw a submarine as the perfect setting - I could travel from a very small metallic space to an underwater scene. The story itself is a bit morbid, but thanks to a Paulstrecthed Brian Wilson, the ending's a bit happier. 


Screen Shot 2017-09-17 at 11.13.11 AM.png

Color Bias Maximizer

About

The Color Bias Maximizer is a Processing sketch that exaggerates and visualizes bias as a means of demonstrating the flaws in data collection. The more my partner, Alex Fast, and I looked at the data sets we were working with (and in light of all the recent articles on AI bias), the more we wanted to focus on bias as a topic. 

We aimed to make an educational tool for people growing up with these technologies affecting them and designed an interface that visually demonstrated bias in real time. The basic premise was to scan a live video capture and determine what the strongest RGB value was for every pixel. The highest of these values then sorted those pixels into Red, Green and Blue categories. All the while we kept track of the color information that was being lost by this process.


Screen-Shot-2017-04-29-at-2.16.58-PM-600x150.png
Screen-Shot-2017-04-29-at-2.32.26-PM-768x494.png