3DRing: Enabling Low-Cost 3D Hand Position Tracking by Fusing Optical andInertial Sensing

July 2024 ‑ November 2024

Zhuojun Li, Chun Yu*, Chang Liu, Yuanchun Shi

Hand tracking is a fundamental task to enable natural 3D interactions. Current hand tracking systems mainly rely on high framerate (HFR, e.g. 60 FPS) optical sensors, which results in high power consumption and computational cost, thereby limiting their application in end devices. We propose 3DRing, a 3D hand position tracking method that fuses HFR inertial data from a single IMU ring with adaptive, low framerate (LFR, i.e. <10 FPS) optical data, thus maintaining high tracking accuracy and low latency. Our method incorporates two stages: (1) a Deep Extended Kalman Filter-based (DEKF) prediction stage, and (2) a Reinforcement Learning-based (RL) calibration stage. In the first stage, we use an RNN velocity prediction model and a EKF model to jointly regress HFR hand positions using inertial data. In the second stage, we use an RL-based adaptive framerate strategy to select minimal optical keyframes to calibrate the error caused in the first stage. Evaluations show that our method uses only 6.61 FPS (9.9%) optical data to achieve an average real-time tracking error of 1.75 ± 0.18𝑐𝑚, as well as an interaction speed of 86.0% in a 3D target selection task. It demonstrates a strong potential of our method to reduce the optical data reliance in mobile hand tracking devices, making the hand tracking technologies more accessible to end users.

As the second student author, I am in charge of paper writing (the abstract as well as the introduction), and algorithm design (the RL reward function).

Publication:

Submitting to IMWUT 2025