Can a laptop hear your hand move?

A demonstration of touchless interaction left me with a question: could a laptop notice my hand moving by listening? It already has a speaker and a microphone. Perhaps those familiar parts could do something less familiar together.

I started a browser experiment around that idea, then followed it into reflected sound, frequency shifts, and echo timing. The result is a prototype with simulated signals to explore. I have not yet established reliable physical hand tracking on real hardware.

Make a sound, listen for a change

The SoundWave research project uses a device’s speaker to emit a tone and its microphone to listen for reflections. A moving hand changes the frequency of the reflected sound through the Doppler effect. Those changes provide clues about motion.

That suggests a wonderfully simple interaction: move your hand toward or away from the device to browse photographs. The difficult part is telling an intentional gesture apart from everything else the microphone receives.

An echo contains more than loudness

I wanted to see what happened if the experiment kept more of the signal. Instead of immediately reducing everything to a direction, it could display how the spectrum changes over time and explore several tones.

A spectrogram is a picture of sound energy across frequency and time. It makes changing patterns visible, but it does not retain all the information in the original signal. Following an echo’s timing and phase opens another set of questions.

FingerIO, a separate research project, explores fine-grained tracking using active sonar and multiple microphones. That work inspired the next part of my prototype: echo displays, a motion trace, and an experimental two-microphone mode. It is inspiration, not a claim that my browser experiment reproduces the researchers’ results.

A simulated hand is the easy part

The prototype’s simulated echoes let me inspect how its processing responds to a known input. That is useful for finding mistakes and understanding the display. It does not establish that a real microphone will capture the same clean signal.

Before calling it a working gesture controller, I need repeatable physical trials: leave the scene still, move a hand toward and away from the device, repeat at different distances, and record both missed movements and false triggers. Tracking a two-dimensional position also needs suitable independent microphone inputs and known geometry.

For now, I like the change in perspective. A speaker is something I usually think of as an output. Here it becomes a way to ask the space around a computer a question, while the microphone listens for an answer.

A personal observation, and a question to come back to.