Unity's ml-agents assets throw warnings and errors [PushBlockWithInput, Actuator, Barracuda]

Question

The Problem

I'm trying to work with Unity Machine Learning Agents and encountered problems during the setup. When I try to import the assets from Unity's ml-agents git into Unity, I get many warnings and errors inside Unity. For the purpose of context, I'm at the very beginning of learning Unity, so I don't know if the errors are due to the ml-agents package or user error from my side in how to set everything up.

The errors and warnings

Instructions to create a first test scene with assets from Unity's ml-agents git suggest making a new 3D project in Unity and drag and drop the folder projects/assets/ml-agents into the project's assets. At this point, Unity is showing many errors and warnings in the Terminal. It still has the examples in the assets but every element in the scene is full of warnings.

according to these tutorials from 2020 by dragging and dropping the assets into Unity [1] [2], I subsequently

In my case the majority of the warnings seem to come from PushBlockWithInput, PushblockActions and PushBlockWithInputPlayerController missing UnityEngine.InputSystem and Unity.MLAgents.Extensions.Input with the note "(are you missing a using directive or an assembly reference?)". This; however, did not happen in the aforementioned tutorials.

Although they make the majority of errors, they are not exclusively about assembly references. Other errors, which may or may not be about assembly references, are

• error CS0115: 'Match3Board.GetCurrentBoardSize()': no suitable method found to override
• error CS0535: 'SensorBase' does not implement interface member 'ISensor.GetCompressionType()'

The things I've tried

Python

I've followed the instructions from Unity's ml-agents git and was successful in installing a Python 3.7 environment with Anaconda, PyTorch with Cuda, and the Unity's ml-agents python package via pip. When trying to verify the ml-agents python package works with mlagents-learn --help, I first got an exception but could resolve that by updating protobuf==3.20.1 as per suggestion from a forum (just mentioning this in case it is relevant).

Unity

I downloaded the C# package from Unity's package manager and tried it for several versions (1.0.8 (Verified), 1.9.1 (Preview), 2.0.1, and 2.1.0 (Preview) -- lastest). After which I'm able to select ML-Agents from the 'Add Component' menu in the Inspector.

I've also tried to create a new Unity project with the ml-agent package 1.9.1 (Preview) with the right Barracuda version, and the release 19 branch of Unity's ml-agents git, without success (now it's 53 warnings and 70 errors). Now also the Actuators are not found, which seems to be a more common problem on its own.

VS Code

I'm using VS Code as opposed to VS as was recommended here. I downloaded .Net version 6.0.301 and checked it was installed with 'dotnet --info'. In the VS Code's extension manager, I installed the extensions C#, Code Runner, Unity Tools , Debugger for Unity.

Git-Repository

I have also switched from Unity's ml-agents git's main branch to the release 19 branch and also tried other versions of the Barracuda package, e.g. Version 3.0.0, which seems to remove the warnings, but not the errors and instead gives these notifications:

However, warnings still show up in the assets' settings:

Course: ML-Agents: Hummingbirds

I successfully completed the ML-Agents: Hummingbirds. This course doesn't use any of the assets from the ML-agents Git repository. Although, assets for this course are downloaded and added the same way, without issues. This lets me assume that the general setup for ML-agents is working but I specifically can't import the assets.

My setup

• I'm working on a machine with Windows 11
• Unity Version is 2020.3.32f1 Personal <DX11>
• The Unity ml-agent package was tried with 1.0.8 (Verified), 1.9.1 (Preview), 2.0.1, and 2.0.2 (Preview)
• The Unity ML Agents Extensions package 0.6.1 (preview)
• Python Version is, as per instructions, 3.7 with an Anaconda virtual environment
• Unity's ml-agents git is main, as well as release_19_branch
• C# editor would be Visual Studio Code 1.67.2
• DotNet Version: 6.0.301

Things I found out so far

This problem seems to be somewhat common, I've found several variations of similar problems over a couple of years, some more specific to the Actuators missing, some more general. Some are posting about problems with the InputSystem as well, but seemingly different solutions and mixed reactions to the solutions.

There are various suggestions, about version changes for Unity, the ml-agents package, and Visual Studio Code. Other solutions involve downloading additional packages in Visual Studio or Unity. Some suggest editing scripts within the cloned git repository. Although most of these threads are from the past 2 years. I've spent two days trying to set this up and fix this and am just about to give up on the ml-agents package. A shame the installation process for a seemingly great resource seems so infeasible. I'd appreciate further suggestions or directions on additional resources on how to set up this package.

Answer 1

You need to add a reference to the Input assembly definition(asmdef) Assets/Scripts is where asmdef usually is but you cant just search it if it isn't there

In the Assembly Definition References section, click the + button to add a new reference and set it to Unity.InputSystem