Gestures on Objects

XR glasses can see the world, but they can’t really reach into it. The glasses know there’s a pair of headphones on my desk; what they can’t do is let me act on it. I keep coming back to the same question I asked with Glyph: what role do everyday physical objects play in a world that’s drifting away from the phone? This project is a different answer to it. Instead of carrying one instrumented object, what if any object could become a button?

The setup is relatively simple. A cheap NFC sticker goes on the object. It costs a few cents, has no battery, and needs no maintenance. A small antenna sits on the tip of my index finger. When my finger approaches the tagged object, the wearable knows which object it is, and how I’m touching it. Tap a pair of headphones to play or pause. Swipe across them to skip a track. Perform a double-tap to summon a voice agent that lives on that object. The object itself becomes the interface.

The interesting part is what makes the gestures possible. NFC is almost always treated as a digital event. A tag is either present or it isn’t, and you read its ID or the data stored on it. But that framing throws away a continuous signal hiding in plain sight. As the antenna gets closer to a tag, the tag draws more energy from the reader’s field, and the strength of that coupling changes smoothly with distance. The data layer in the wearable can’t see it because it only polls every 50 milliseconds and reports whether it finds a tag and what data is stored on it. So I added a second path: a tiny op-amp envelope detector that reads the raw field strength directly, giving a clean analog proximity signal updating around 100 times a second. The digital read tells you which object you’re touching. The analog read tells you how. Fuse the two and you get gestures.

A two-stage recognizer running on a microcontroller turns those traces into named gestures like tap, double-tap, swipe, hover and tap-and-hold. The ESP32 ships each one over WiFi with the tag’s ID attached. Because every event carries the object’s identity, gestures route themselves to the right thing automatically. There’s no separate “select this object” step; performing the gesture on the object is the selection.

The events are consumed by OpenObjects, a small XR platform I’ve been building, with a Lens on the Snap Spectacles. Users can describe desired interactions (visual and verbal) in plain English and an LLM handles the required wiring. Both visual overlays and voice-agent personalities are then attached to physical objects. In the demo above I tagged a pair of headphones and wired the gestures to a Spotify connector: tap to toggle playback, swipe to skip, double-tap to bring up a voice agent.

This was my final project for MAS.836, the Media Lab’s sensor technology class. Not every gesture is perfect yet. A swipe and a fast tap can look alike when the timing is tight, and the honest version is that swipe lands right about half the time. But the core idea holds up. The bigger thread is the same one Glyph pulls on. Glyph makes one carried object smart. This makes the cheapest sticker you can buy enough to fold any object into an XR experience.

The current version of the project is a prototype that pulled me into antenna design. Next, I want to shrink the antenna further and improve gesture detection.