Touchless Gesture Control

Problem

A kiosk in a public space is a shared touchscreen — a surface a few hundred strangers press in a day. For an exhibit you want the opposite of that: someone walks up, the thing responds, and they never touch it.

Handing out a controller solves the hygiene problem and creates three new ones, all of which involve staff walking over. Whatever replaced the touchscreen had to work for someone who has never seen it before, with no instructions and nothing in their hands.

Solution

A webcam, MediaPipe, and a bridge into the Windows input stack. The visitor’s hand becomes the cursor.

• Pose detection finds the visitor and establishes a tracking region around them, so hand tracking only listens inside that box — someone walking past in the background doesn’t steal the cursor
• MediaPipe’s gesture recognizer maps poses to actions: open palm moves the cursor, closed fist clicks, thumbs up opens the info panel
• Cursor movement runs on its own thread that continuously writes the latest position, rather than issuing one jump per video frame — the difference between a cursor that glides and one that stutters at camera framerate
• A Win32 IPC channel sends delimited command strings straight into another application’s window, so the tracker can drive an exhibit app it isn’t part of and doesn’t need to be rebuilt into
• Pose landmarks also track arm swipe direction, for navigation that doesn’t involve aiming at anything

Worth noting

Both models are read into memory and handed to MediaPipe as a buffer rather than a file path, because the path-based loader doesn’t behave on Windows. It’s two extra lines and a comment explaining why, which is the kind of thing that saves the next person a genuinely miserable afternoon.

The real lesson was that recognising a gesture is the easy half. Deciding whose gesture counts — in a room where people wander through frame — is what makes it usable in public.