It is very unlikely you manage to configure your Android phone as a Jenkins slave. It's just too many libraries that you would need to build on your own so they run natively on Android. You would need to have the Jenkins slave, Appium, Robot Framework and your app running in the same phone, which isn't as powerful as any regular computer.
However, if your device is rooted, you can try something like WiFi ADB to at least get the lowest level layer to your phone available over the air. There is also a manual setup with no software available at How can I connect to Android with ADB over TCP?.
If rooting is not your option, but you can connect the device through USB just once (which I assume you can't do because probably you're using a cloud host like Azure and that's why you can't physically access the machine), do it and run adb tcip 5555 (or your desired port). This will allow you to desconnect the device and use it over WiFi with the trusted computer connected through USB (until you kill the ADB server!).
After any of both procedures, configure your project as usual and set the udid capability to the IP of your phone in the network (try to configure a static IP address) and its port, for example 192.168.0.24:5555. Before running the tests, you have to run adb connect 192.168.0.24:5555 so adb is aware of the device and Appium can find it.
You still will have some trouble getting the VM and the phone in the same network, but for that issue there are plenty of options available like configuring a VPN.
