Kandinsky Cam Paint

Painting in the air with objects and gestures, and hearing what you paint.

Role
Solo designer & developer
Context
ArtCenter College of Design
Tools
p5.js
ml5.js
Teachable Machine
Web Audio API
Claude

Kandinsky Cam Paint imagines drawing another way.

You paint without touching a surface or a screen. You show the camera an object or a gesture, and your index finger chooses where it lands. Drawing in the air changes how drawing feels, and how the marks come out.

Kandinsky described seeing color in sound. He painted music. This piece runs the other way: what you place on the canvas plays back as sound. Height becomes pitch, and color becomes the instrument.

Gesture vocabulary

Every interaction comes through the camera. The first screen teaches the vocabulary over the live view, and showing a sheet of paper begins the canvas.

Onboarding screen over the live camera: Show paper to begin, with the six objects and gestures
The first screen.

Making

From a visual effect to an instrument.

It began with an image classifier I trained in Teachable Machine on hundreds of webcam samples of each object and gesture. Two models read the camera at once: one decides what you show, the other finds your fingertip.

1 · PlacementShapes first landed at random. Then the fingertip began to guide where each mark goes.
2 · Seeing yourself paintA visible fingertip cursor and a small camera view, so you always know where you're painting.
3 · Kandinsky's paletteThe canvas fills the screen, pencil curves are smoothed, and colors come from a fixed Kandinsky-inspired palette.
4 · A tool that teaches itselfGesture hints over the camera and a progress ring for Paper explain the tool with no buttons.
5 · The painting playsA line sweeps across every 8 seconds and plays each mark it passes. Height sets pitch, color sets the instrument, black marks are percussion, and size sets how loud and how long.
6 · Every mark is a noteOnce marks made sound, every misread gesture became a wrong note. Palm and fist now need higher confidence, held about a quarter second.
7 · Holding the fingertipShapes remember the fingertip for one second, so they stay in place when a grip hides the finger.

Paintings

An early painting with overlapping circles and squares placed at random
Early · shapes placed at random
An early painting with a circle, triangles and a long pencil line
Early · the fingertip guides placement
A painting in progress with squares, circles and pencil lines
Building · circles and pencil lines
A finished painting with a vertical playhead line passing over it
Playing · the line sweeps and each mark sounds

After Wassily Kandinsky

Image classifier trained in Teachable Machine · Hand tracking with ml5.js HandPose

Play it live →