Hand tracking.. on a smartphone

[zb3]

I see that hand recognition relies on WebXR hand-tracking feature which my Pixel 7 Pro phone apparently doesn't support :(
But I know that MediaPipe has this "hand landmark tracker" model and it returns 21 landmarks and even supports 3D space.. the live demo works smoothly and accurately on my phone.

Since I know not much about these topics and nothing about these repositories, let me just ask - how possible would be to integrate these two together so that hands+gestures demos would somehow work on a smartphone?

Of course I could find it out by myself, but if this is not possible, or maybe already planned then my crappy implementation would not make sense, so here I ask.

[dli7319]

WebXR hand tracking gives us 3D hand joint poses (position + rotation) from the multiple camera feeds and the underlying hand tracking runtime on the headset. Note that the rotation is very important for mesh skinning.

It's possible to integrate hand tracking from a single phone camera but not trivial.

[ruofeidu]

This is a great proposal ;) I've thought about it but lower prioritized given our bandwidth.

Adding zhongyizhou@ to public discussion in case you are interested or have the bandwidth.

[zhongyi-zhou]

@zb3 great proposal!

As @dli7319 has mentioned, it is not trival if we want to render the webxr hand mesh based on the MediaPipe hand pipeline. @zb3 My understanding is that you are not using XR devices, and you only need the detection, not the rendering part? Do you have more specific use case description? It will be very helpful for us to understand what features will be necessary in your project.

[zb3]

Ah I see - so WebXR returns more than just positions. But still, I think it might be possible to reconstruct some of this information even without an additional trainable model, given that we can exploit anatomical constraints of those joints to infer some orientation info (but I'm not sure how accurate that could be). However I didn't yet look at this and I think I'd first need to verify whether that MediaPipe model returns reliable positions also in 3D as I've only seen the 2D positions demo.

@zhongyi-zhou That's right, I do not currently have any XR device. I remember Google mentioning that we don't need a XR device to develop for XR, so after discovering this repository I thought these demos could help me discover how hand navigation works in practice and possibly experiment with my own gestures for controlling UI - just that :)
Rendering hand meshes is not needed in my case as I'm just trying to explore simple UI interaction where the UI won't cover my hand..

[zhongyi-zhou]

@zb3 If you do not need mesh rendering, it should be much easier.

Does it work if I write you a simple demo in which some XR element will respond to some of the captured gestures (e.g., pinch and thumb up) via webcam? With this demo, I think you will be able to define your own gesture input and the UI output in your own use cases.

I will try figuring out sometime to submit pull request within two weeks. Once done, I will ping you.

[zb3]

@zhongyi-zhou Sure, that would be helpful :)

As for my experiments with full 3D positions, they're kinda noisy and the full MediaPipe model predictably isn't smooth on a more low-end hardware.