Interactive
Control.

Day 1 was how to make the machine move. Day 2 is how to make the machine see.

The job of interaction design is to build a pipeline from human experience to machine behavior. Human experience is vast and continuous — temperature, sound, light, gesture, presence. A sensor captures one narrow slice of that experience and converts it to numbers. You filter, smooth, map, and threshold those numbers until all that remains is a single X,Y coordinate pair. That pair moves the machine.

This narrowing is the funnel. Every interactive drawing system is a version of it.

• Human experience. Continuous, embodied, noisy — everything happening in and around the body.
• Sensor. A window onto one dimension of that experience: light, sound, pressure, motion.
• Raw data. Numbers over time — often a stream of values arriving many times per second.
• Processed data. Filtered, mapped, and scaled to a useful range. This is where design decisions live.
• X,Y. A single coordinate the machine can act on. The end of the funnel, the beginning of the mark.

Until now we've been queueing instructions: write the file, send the file, watch it draw. Real-time control inverts the loop — instructions arrive while the machine is still moving. The drawing is no longer authored, it's steered.

• Streaming G-code. Keep the buffer thin so new commands arrive fresh. The machine never gets too far ahead of your intent.
• WebSerial. Browser-side control of a machine via USB — no drivers, no installs, just a web page and a cable.
• OSC / MIDI. Map any controller to any axis. A knob becomes a speed dial; a slider becomes a scale factor.
• Latency budget. ~50 ms feels live. Above that and the drawing leads you instead of the other way around.

Embracing instability

Machines are built for precision and repeatability. That's their value in industry — and the first thing we want to undermine. The goal is to break open the machine's determinism and create space for improvisation, serendipity, and processes that respond to the moment rather than a predetermined plan.

Introduce latency on purpose. Feed the machine conflicting signals. Let the sensor data arrive faster than the machine can respond, so it interpolates and invents. Use the control loop not as a constraint but as a conversation — one where neither side fully controls the outcome. The best interactive drawings happen when the system surprises both you and itself.

A sensor is a translator: it reads one channel of human experience and converts it to numbers. Different sensors see different worlds. We'll look at three families, each with a pre-built example you can use in today's exercise.

Webcam

A camera captures an RGB video frame — a dense grid of color values, typically 640×480 or larger. The camera has no idea what it's looking at. It sees color at each pixel, nothing more.

To turn that grid into something a machine can draw, we run it through a vision pipeline: convert to grayscale, detect edges using a threshold or Canny filter, extract contours as ordered point lists, and convert those contours to polyline coordinates the machine can trace.

The pre-built webcam example uses the browser's getUserMedia API and runs edge detection in JavaScript. No install required.

Microphone

A microphone captures an audio signal — air pressure fluctuations over time. The raw waveform is a stream of amplitude values arriving thousands of times per second. On its own, a waveform tells you how loud but not what frequency.

To extract useful structure, we run an FFT (Fast Fourier Transform), which decomposes the signal into frequency bands and their amplitudes. Now we have two dimensions to map: frequency (pitch) and amplitude (volume). A rising note can become a rising curve. A falling note, a falling curve. Silence lifts the pen. A percussive hit becomes a sharp angle.

Direct control (HMIs)

Some interfaces skip the abstraction entirely. A human directly manipulates the machine through a physical controller. These are the simplest version of the funnel — one mapping layer, no signal processing.

• Gamepad. Joysticks map directly to X,Y. Triggers can control pen up/down or speed.
• MIDI controller. Knobs and sliders map to parameters: scale, rotation, speed, line weight.
• Tablet / iPad. Touch position becomes machine position. Draw on glass, mark on paper.

Every sensor maps to the same funnel. The creative decisions are: which sensor, which slice of its data, which mapping — and how it's embodied in its environment.

Time to make a drawing with a sensor in the pipeline. Choose one of the pre-built sensor interfaces, or wire your own. The goal: a finished piece that could not have been made without the sensor. Its presence should be legible in the final drawing.

Two approaches

The sensor generates geometry. The sensor data becomes the drawing. Edge detection turns a webcam feed into contours; audio frequencies become curves. The machine draws what the sensor sees or hears.

The sensor steers the machine. The sensor data controls the machine's position or speed while it draws existing geometry. A MIDI knob adjusts scale in real time; a gamepad nudges the pen mid-path. The base drawing is predetermined, but the sensor warps it.

Pre-built examples

• Microphone. Audio → geometry. Sound shapes the line.
• Webcam. Vision → contour. The camera draws what it sees.
• MIDI controller. Direct parameter control. Knobs map to machine variables.
• Turntable. Rotational input mapped to angular geometry.

Exercise

• Choose. Pick a sensor and an approach (generate or steer). Load the pre-built example. 10 min
• Explore the data. What do the numbers look like? Print them, watch them change. Understand the range and rhythm of your sensor. 20 min
• Map. Decide what the numbers control. Try at least two different mappings before committing. 20 min
• Draw. Queue for the machine. One drawing, then iterate your mapping if time allows. 20 min
• Review. We'll look at results as a group. Which drawings show the sensor's hand? 10 min

When you're happy with a drawing, upload your SVG and a photo of the physical output to the group folder.