5

I have a neural network trained in pytorch that I'd like to deploy into a Unity app. What's the best way to do it? I'm also interested in allowing the user to further train the neural network in the Unity app, which I guess would require to integrate some part of pytorch into Unity (maybe there's a way to integrate pytorch's C++ / torchscript API with Unity?). If anybody has experience with this, I'd like to know what the best alternatives are.

Mike Wise
  • 22,131
  • 8
  • 81
  • 104
Mei Zhang
  • 1,434
  • 13
  • 29
  • https://unity3d.com/machine-learning Unity has a machine learning toolkit you may wish to check out. – Doh09 May 26 '19 at 20:59
  • I'm familiar with ML agents, but this doesn't answer my question. ML agents is just a collection of environments that have to communicate with an external ML framework. I'm interested in embedding a ML engine into a Unity app. – Mei Zhang May 27 '19 at 00:52
  • Never claimed it would answer your question. Mentioned it in case you were unaware of its existence. Hence I made a comment not an answer. – Doh09 May 27 '19 at 05:11
  • 1
    Did you have any success training with Pytorch in a Unity app? What was your approach? – Florian Wolf Jul 06 '20 at 03:13

3 Answers3

9

Check out the new features in Unity ML Agents. There is an inference engine within Unity ML Agents (called Barracuda) that allows you to use pretrained models within your app. AFAIK, you can convert Tensorflow and ONNX models into Barracuda. It should not be a problem as Pytorch models can be converted to the ONNX format. You may need to retrain your model if it is directly affected by the app (for example, if it is an RL agent).

EDIT: To answer your second question, you can continue to train the model but not in real time. What you may be able to do is collect data from the user, and use that to further train the model (that is how TensorFlow Serving works). You can do that by converting the PyTorch model into a TensorFlow model via ONNX.

EDIT 2: Barracuda is now a standalone and production ready inference engine that runs exclusively on the ONNX format. Any framework that can be converted into the format (e.g. Keras, Pytorch, MXNet) will work as long as they contain the supported operators.

Ayaz Amin
  • 180
  • 3
  • 11
3

This is not a complete answer. But it should be able to help you progress further.

Essentially you'd as I see it just need to be able to run Python code within C# code. In this case in collaboration with Unity's framework.

I did some searching and came across 4 partial solutions:

  1. Unity python interpreter: https://forum.unity.com/threads/python-interpreter-in-unity.86461/

  2. An example of running code with IronPython in Unity: https://gamedev.stackexchange.com/questions/123526/a-python-script-controlling-a-unity-game

  3. Communication example between Unity3D C# and Python, using ZeroMQ: https://unitylist.com/p/hc8/Unity3D-Python-Communication

  4. Implementing language support for other languages yourself via an XML file: https://forum.unity.com/threads/add-multiple-language-support-to-your-unity-projects.206271/

Once you're able to run the code you'd then need to refer to the location of your python files. This way you should be able to run it if the environment you run it in also has Python installed and set up correctly.

You must ensure that the files are in your project on deployment. And that Unity can access them.

Hope this helps you.

Doh09
  • 2,324
  • 1
  • 16
  • 31
  • Have you been able to train mlagents within a build using one of these methods? Also, does this mean there is no way to somehow ship python within the build, so that the end user isn't required to install python on his computer? I am trying to create a distributable application in which the user is able to train his own agents. Currently I am facing the python roadblock and I am considering fully implementing my RL algorithm of choice in C#... – Florian Wolf Jul 07 '20 at 18:46
0

https://github.com/asus4/tf-lite-unity-sample shows how to apply tensorflow models in Android and IOS, through barracuda can also apply onnx model in mobile platform, tf-lite-unity-sample is much more faster than barracuda.

user17521188
  • 1
  • While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference. Link-only answers can become invalid if the linked page changes. - [From Review](/review/late-answers/30444926) – miken32 Nov 30 '21 at 17:29