How hand tracking actually works

There’s no controller, no glove and nothing to download, yet when you move your hand, a game responds. It can feel a little like magic. It isn’t, and the way it works is surprisingly easy to follow.

Step 1: The camera takes pictures, fast

A webcam doesn’t record a hand. It takes a rapid series of still pictures, typically around 30 every second. Each one is called a frame. Tracking is really the same question asked over and over: where is the hand in this frame?

Step 2: Finding the hand

Our games use MediaPipe, an open-source toolkit from Google. Its hand tracker works in two stages. First, a palm detection model scans the picture and finds where a hand is. (A palm is a simpler, more predictable shape than a hand with its fingers in every possible position, so it’s easier to spot.) Then a second hand landmark model studies that area in detail.

Step 3: Twenty-one points

The landmark model marks 21 points on the hand: the wrist, the knuckles, the joints of each finger and the fingertips. Each point gets a position across and up the picture, plus an estimate of its depth. It’s a bit like a connect-the-dots drawing of your hand, redrawn many times a second.

Step 4: Following, not searching

Once the tracker has found your hand, it doesn’t start from scratch on every frame. It follows the hand from one picture to the next, and only searches again if it loses it. That’s a big part of why tracking can run smoothly in an ordinary web browser.

Step 5: Turning points into play

From there, the game does the fun part. Depending on the game, the position of a fingertip can steer a cursor or trace a shape, and the distance between points can tell an open hand from a pinch. The game also checks whether your fingertip overlaps something on screen: a bubble to pop, a scoop to catch, a fish to feed.

Where does the video go?

Tracking happens in your browser, on your own device. byondthewords doesn’t record or store video, and the camera pictures aren’t sent to us for tracking.

The one optional exception is our Living Moments feature. When a player reaches a top score, a single still photo can be saved privately to their own account, where only they can see it. You can read the details in our Privacy Policy.

Why it sometimes struggles, and what helps

No tracker is perfect. If a game loses your hand, these simple changes help most:

• Light from the front. Face a window or a lamp, rather than sitting with a bright window behind you.
• A clear view. Keep your whole hand inside the camera’s picture, at a comfortable arm’s length from the screen.
• A calm background. Very busy or very dark backgrounds make hands harder to find.
• A steady camera. Put the laptop on a table and tilt the screen so the camera can see you comfortably.
• An up-to-date browser. If a game can’t start its tracker, update your browser, check that hardware acceleration is switched on, and reload the page.

Why this matters for accessibility

A mouse or keyboard asks for small, precise movements, which can be hard for some children. Playing with the whole hand and arm lowers that barrier for many of them, and it means the same game can be played seated or standing, with big movements or small ones.

Camera-based play has its own barriers too, such as needing a webcam and reasonable lighting. If a game isn’t tracking well for you, please tell us through our Contact page. It helps us improve.