1

I'm using python's media pipe library and webcam to track pose. The obtained pose landmarks is transferred to its unity clients. The data is of JSON form and its structure is

{
        keypoint_number:{
                            "x": (some value say) 0.4134315135,
                            "y":  0.8134315135,  
                            "z": 0.3123212312
                          }
 }

totally 32 key points as per the media pipe pose estimation.

now I'm transferring these data over UDP connection.

Now the problem is when I try to map these coordinates onto unity's humanoid or robot character, the character dismantles/shatters.

What I did was I just mapped the local position of the bones of the character with the received key points coordinates.

I am stuck with this please help me.

Receiver.cs

public class Receiver : MonoBehaviour
{
    private Socket? _clientSocket;
    private EndPoint? _socketEndPoint;
    private byte[]? _buffer;
 
    public GameObject? Body;

    void Start()
    {
        _clientSocket = new  Socket (AddressFamily.InterNetwork
                , SocketType.Dgram,
                ProtocolType.Udp);
        _clientSocket.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.Broadcast, true);
        _clientSocket.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.ReuseAddress, true);
        _socketEndPoint = (EndPoint)new IPEndPoint(IPAddress.Any, 8080);
        _clientSocket.Bind(_socketEndPoint);
        _buffer = new byte[1024 * 5];

     
    }
    void Update() {

        Array.Clear(_buffer, 0, _buffer.Length);
        _clientSocket?.ReceiveFrom(_buffer
            , ref _socketEndPoint);
     
        string landmarksString = Encoding.UTF8.GetString(_buffer);
        Dictionary<int, Dictionary<string, double>> landmarks 
            = JsonConvert.DeserializeObject<Dictionary<int, Dictionary<string, double>>>(landmarksString);


        float z = (float)(landmarks[23]["z"] + landmarks[24]["z"]) / 2;


        Transform[] bodyPoints = new Transform[Body.transform.childCount];
         for (int i = 0; i < Body.transform.childCount; i++)
             bodyPoints[i] = Body.transform.GetChild(i);


          foreach(int keypoint in landmarks.Keys) { 
             float x = (float) landmarks[keypoint]["x"];
             float y = (float) landmarks[keypoint]["y"];

             bodyPoints[keypoint].localPosition = new Vector3(x, y, z);
         }
    }

}

This is the output of normal stick sphere joint figure

Banana man with rig

Aaditya Prabu K
  • 19
  • 2
  • Could you provide a full dataset and the bone tree? Just setting bone positions is probably insufficient, since it completely leaves out the rotational aspect and ignores the bone length (for example tracked arms can be shorter than the model). Dunno if a LookAt cascade would work better (underarm bone looks at hand point). It might still look warped, but at least not completely crunched. – Voidsay Apr 23 '23 at 16:32
  • @Voidsay hi thank you for the reply, im using the banana man asset from unity store, i used a bone renderer to make the bones visible and what i did was i just referenced these bones to the receiver.cs script where in that script it receives and the coordinates from the udp server through sockets and uses a for loop to map the reveived keypoints to the bones of the banana man – Aaditya Prabu K Apr 24 '23 at 08:27
  • That's all fun and good, but us insane people from Stack Overflow need something tangible to make it glitch out on our end before we can figure out what's wrong. – Voidsay Apr 24 '23 at 11:41
  • @Voidsay okay what kind of data set you want me to show ? – Aaditya Prabu K Apr 24 '23 at 14:44
  • The script that you are using and an example data set that you want to reconstruct. – Voidsay Apr 24 '23 at 15:10
  • @Voidsay i have added the script in the question please check and i want something similar to this video on youtube https://www.youtube.com/watch?v=6umg8XXWHoI – Aaditya Prabu K Apr 30 '23 at 14:24
  • The code is very rudimentary and unreliable. Rotation is completely left by the way side, so it's going to look spaghetti no matter what. The use of `transform.childCout` suggests that you are either using a scuffed rig out of the box or due to your own modification. If the root bone is `Body` I would only expect 3 children (spine, legs) on a standard mocap rig and not the full 32. You also have to rely on Unity using the same bone order as your incoming data. Are you using [this Banana man](https://assetstore.unity.com/packages/3d/characters/humanoids/banana-man-196830#content) asset? – Voidsay Apr 30 '23 at 17:41
  • Could you serialize the `_buffer` byte array into an XML and send it my way? I really want to test the integrity (and order) of the incoming pose data. See if I can manually reconstruct something vaguely human shaped. – Voidsay Apr 30 '23 at 17:45
  • @Voidsay this body consist of 32 spheres in it and it is connected by line renderer, hence it is very basic as the spheres are placed in their local positions as they are assigned by the incoming coordinates . I tried the same for banana man of unity, but there is the problem, as you said it doesn't have any rotational aspect. – Aaditya Prabu K May 01 '23 at 07:07
  • @Voidsay i have attached an output for the above explanation – Aaditya Prabu K May 01 '23 at 07:14
  • @Voidsay also the incoming data comes as json so i am mapping it in a hashmap> – Aaditya Prabu K May 01 '23 at 07:17
  • Alright, the image implies that the incoming data is good. I can also tell that mapping it on a rig is going to be a chunk of work. Do you have a failed attempt of that? What order do the `keypoint numbers` come in? Are you first constructing one leg then the other and so on or is there another silly order? – Voidsay May 01 '23 at 08:27
  • @Voidsay keypoints come in as a hashmap (no order), i can retrieve the keypoints using a hashmap get function – Aaditya Prabu K May 02 '23 at 16:50
  • I get that, but I mean that the indexes have to be associated with a body part. For example 1 is base of the leg, 2 is knee, 3 is ankle, 4 is heel, 5 is toe tip. You need a way to map these to left thigh, left leg, left foot. – Voidsay May 02 '23 at 17:03
  • if it was the stick man there directly i map them to the joint spheres but if it is a banana man i tried to map the bones to the coordinates but it shatters the figure , i tried inverse kinematics but it the target appears to be present only below the hip hence hands are not moving properly – Aaditya Prabu K May 02 '23 at 18:22

0 Answers0