Google AI Achieves Real-Time Hand Tracking On Smartphones
Google researchers develop a highly accurate, real-time hand tracking system that runs entirely on a smartphone, potentially paving the way for better sign language translation.
Millions of people rely on sign language to communicate, but capturing its complex gestures for digital translation remains a difficult challenge. A new real-time hand tracking advancement from Google's AI labs shows significant promise as a potential breakthrough. The system produces a highly accurate map of the hand and its fingers in real time using nothing more than a standard smartphone camera.
Robust hand perception presents a decidedly difficult computer vision task because hands frequently occlude themselves or each other and lack high contrast patterns. Hand movements are also often quick and subtle, making them incredibly hard for computers to catch accurately and quickly. Even complex multi-camera, depth-sensing rigs currently struggle to track every intricate movement required for sign language.
To achieve real-time performance on mobile devices, the researchers reduce the amount of data the algorithm processes by focusing solely on detecting the palm rather than the entire hand. Because the palm is the most distinctive part of the hand and has a square shape, it simplifies the image processing requirements. Once the system recognizes the palm, a separate algorithm assigns 21 coordinates corresponding to knuckles and fingertips, allowing for detailed finger analysis without the need for powerful desktop hardware.