203

I'm new to Tensorflow and would greatly benefit from some visualizations of what I'm doing. I understand that Tensorboard is a useful visualization tool, but how do I run it on my remote Ubuntu machine?

kmario23
  • 57,311
  • 13
  • 161
  • 150
user
  • 2,345
  • 3
  • 12
  • 14
  • 1
    Some possible solutions: (1) It might be the same as running on a local machine, although I haven't verified this: after you run it, it provides you a port number, with that you access it from the browser on your local machine. (2) Are you able to use remote desktop software such as RealVNC? (3) I assume you also run tensorflow computations on the same remote machine; you can also download the log files to your local machine, and run tensorboard on your local machine. – Yao Zhang Jun 23 '16 at 18:17

12 Answers12

309

Here is what I do to avoid the issues of making the remote server accept your local external IP:

  • when I ssh into the machine, I use the option -L to transfer the port 6006 of the remote server into the port 16006 of my machine (for instance): ssh -L 16006:127.0.0.1:6006 olivier@my_server_ip

What it does is that everything on the port 6006 of the server (in 127.0.0.1:6006) will be forwarded to my machine on the port 16006.

  • You can then launch tensorboard on the remote machine using a standard tensorboard --logdir log with the default 6006port
  • On your local machine, go to http://127.0.0.1:16006 and enjoy your remote TensorBoard.
Olivier Moindrot
  • 27,908
  • 11
  • 92
  • 91
  • 1
    Also had an issue with tensorboard plots being all black on Safari, it works on Chrome (related to issue [#4856](https://github.com/tensorflow/tensorflow/issues/4856)) – Olivier Moindrot Nov 30 '16 at 07:57
  • 7
    A small hint: `127.0.0.1` here is your **local** machine ip, so you should leave it as is. Do not change it to your remote ip. I wasted 10 minutes on this. How stupid I am! – DarkZero Apr 20 '17 at 12:07
  • this and enforcing IPv4 with the `-4` ssh flag solved it. Reading around similar questions, many have had issues with IPv6 addresses in their servers. – Robert Parcus Jan 05 '18 at 07:54
  • It seems like I have to stay in the ssh shell. If I close the ssh shell, there is no response from the remote tensorboard. Is there a way to be still connected to the tensorboard while shutting down the ssh shell? – Fan Mar 02 '19 at 20:50
  • @Fan: check the other answer [below](https://stackoverflow.com/a/42445070/5098368). Basically you can add the `-N` flag to avoid opening an ssh shell. Adding `-f` puts the ssh connection in the background. – Olivier Moindrot Mar 02 '19 at 23:30
  • as mentioned by @0xcaff, you should listen on your other interfaces instead – mmohaveri Mar 06 '19 at 01:44
  • @OlivierMoindrot Thank you for this answer. How do you use TensorBoard + Jupyter + SSH? Do you basically need to do the same as your answer for Jupyter as well? Does this mean you require two `-L port:ip:port` port redirections, one for Tensorboard, one for Jupyter? – Basj Jun 25 '21 at 13:52
  • @Basj: you can also add the port forwarding to your `.ssh/config`. But yes you need one port for each. – Olivier Moindrot Jun 25 '21 at 17:50
  • Hi, what should go in place of olivier? – haneulkim Aug 03 '22 at 07:35
157

You can port-forward with another ssh command that need not be tied to how you are connecting to the server (as an alternative to the other answer). Thus, the ordering of the below steps is arbitrary.

  1. from your local machine, run

    ssh -N -f -L localhost:16006:localhost:6006 <user@remote>

  2. on the remote machine, run:

    tensorboard --logdir <path> --port 6006

  3. Then, navigate to (in this example) http://localhost:16006 on your local machine.

(explanation of ssh command:

-N : no remote commands

-f : put ssh in the background

-L <machine1>:<portA>:<machine2>:<portB> :

forward <machine1>:<portA> (local scope) to <machine2>:<portB> (remote scope)

eqzx
  • 5,323
  • 4
  • 37
  • 54
136

You don't need to do anything fancy. Just run:

tensorboard --host 0.0.0.0 <other args here>

and connect with your server url and port. The --host 0.0.0.0 tells tensorflow to listen from connections on all IPv4 addresses on the local machine.

0xcaff
  • 13,085
  • 5
  • 47
  • 55
  • 8
    This should be the correct answer.. mostly likely tensorboard is just not bound to an address visible to external services. – Duane Sep 07 '18 at 04:23
  • 1
    It doesn't work for me for a remote server...What am I missing? – Idan Azuri Jan 07 '19 at 16:32
  • 1
    If you only want to listen on your public IP, you can do: tensorboard --host $(hostname -I) And then you can copy-paste the URL that tensorboard prints out: TensorBoard 1.12.2 at http://10.0.50.42:6006 (Press CTRL+C to quit) – Morten Jan 14 '19 at 09:08
  • 4
    @Idanazuri Do you have http access to your remote machine, or is it firewalled (eg: only accessible via an ssh connection)? If it's ssh only, then this answer will not work for you. – drevicko Nov 27 '19 at 00:45
  • 1
    This answer is particularly relevant to a situation where you trained on a system on your private LAN and just want to do an ad hoc examination of the training process. You will need to specify your logdir (you'll likely find that as a subdir of your checkpoint directory, with a file that looks something like "events.out.tfevents.1604862122.blah.blah.blah") – James_SO Nov 12 '20 at 20:50
  • Straightforward but works like a magic! – K. Symbol Mar 30 '22 at 06:27
28

Another option if you can't get it working for some reason is to simply mount a logdir directory on your filesystem with sshfs:

sshfs user@host:/home/user/project/summary_logs ~/summary_logs

and then run Tensorboard locally.

Scientist1642
  • 1,172
  • 1
  • 11
  • 17
20

--bind_all option is useful.

$ tensorboard --logdir runs --bind_all

The port will be automatically selected from 6006 incrementally.(6006, 6007, 6008... )

plhn
  • 5,017
  • 4
  • 47
  • 47
15

You can directly run the following command on terminal of your remote server to run tensorboard:

tensorboard --logdir {tf_log directory path} --host "0.0.0.0" --port 6006

Or you can also start the tensorboard within your ipython notebook:

%load_ext tensorboard
%tensorboard --logdir {tf_log directory path} --host "0.0.0.0" --port 6006
Devendra Soni
  • 391
  • 2
  • 11
  • what means "0.0.0.0" , do I need to replace it with something else? – zheyuanWang Jul 30 '21 at 09:35
  • Its your local machine's port, you can change it if you have different host – Devendra Soni Aug 03 '21 at 10:47
  • This worked for me. I was using VSCode to remote connect to a machine. In the VSCode terminal of the remote machine I entered the exact command in this answer. VSCode prompted me to open in a browser ( I was using Microsoft Edge), which I did, and Tensorboard showed up – user3731622 Mar 06 '23 at 20:04
12
  1. Find your local external IP by googling "whats my ip" or entering this command: wget http://ipinfo.io/ip -qO -
  2. Determine your remote external IP. This is probably what comes after your username when ssh-ing into the remote server. You can also wget http://ipinfo.io/ip -qO - again from there too.
  3. Secure your remote server traffic to just accept your local external IP address
  4. Run Tensorboard. Note the port it defaults to: 6006
  5. Enter your remote external IP address into your browser, followed by the port: 123.123.12.32:6006

If your remote server is open to traffic from your local IP address, you should be able to see your remote Tensorboard.

Warning: if all internet traffic can access your system (if you haven't specified a single IP address that can access it), anyone may be able to view your TensorBoard results and runaway with creating SkyNet themselves.

JohnAllen
  • 7,317
  • 9
  • 41
  • 65
9

You have to create a ssh connection using port forwarding:

ssh -L 16006:127.0.0.1:6006 user@host

Then you run the tensorboard command:

tensorboard --logdir=/path/to/logs

Then you can easily access the tensorboard in your browser under:

localhost:16006/
8

This is not a proper answer but a troubleshooter, hopefully helps other less seasoned networkers like me.

In my case (firefox+ubuntu16) the browser was connecting, but showing a blank page (with the tensorboard logo on the tab), and no log activity at all was shown. I still don't know what could be the reason for that (didn't look much into it but if anybody knows please let know!), but I solved it switching to ubuntu's default browser. Here the exact steps, pretty much the same as in @Olivier Moindrot's answer:

  1. On the server, start tensorboard: tensorboard --logdir=. --host=localhost --port=6006
  2. On the client, open the ssh tunnel ssh -p 23 <USER>@<SERVER> -N -f -L localhost:16006:localhost:6006
  3. Open ubuntu's Browser and visit localhost:16006. The tensorboard page should load without much delay.

To check that the SSH tunnel is effectively working, a simple echo server like this python script can help:

  1. Put the script into an <ECHO>.py file in the server and run it with python <ECHO>.py. Now the server will have the echo script listening on 0.0.0.0:5555.
  2. On the client, open the ssh tunnel ssh -p <SSH_PORT> <USER>@<SERVER> -N -f -L localhost:12345:localhost:5555
  3. On the client, in the same terminal used to open the tunnel (step 2.), issuing telnet localhost 12345 will connect to the echo script running in the server. Typing hello and pressing enter should print hello back. If that is the case, your SSH tunnel is working. This was my case, and lead me to the conclusion that the problem involved the browser. Trying to connect from a different terminal caused the terminal to freeze.

As I said, hope it helps!
Cheers,
Andres

fr_andres
  • 5,927
  • 2
  • 31
  • 44
  • I have the exact same problem on chromium+ubuntu14, even worse, it works perfectly fine for the port forwarded from one computer, but shows the same blank page (with correct favicon) for another. Firefox works just fine. Still didn't figure it out, just using firefox now. – LucasB Oct 19 '17 at 10:56
  • Yeah, many factors involved but as long as it works... another thing to note is that the TB version that comes with TF is different from the TB-standalone one that can be found here... `https://github.com/dmlc/tensorboard` just to add more alternatives to the setup. Cheers – fr_andres Oct 19 '17 at 17:17
5

Another approach is to use a reverse proxy, which allows you to view Tensorboard from any internet connected device without SSHing. This approach can make it far easier / tractable to view Tensorboard on mobile devices, for example.

Steps:

1) Download reverse proxy Ngrok on your remote machine hosting Tensorboard. See https://ngrok.com/download for instructions (~5 minute setup).

2) Run ngrok http 6006 (assuming you're hosting Tensorboard on port 6006)

3) Save the URL that ngrok outputs:

enter image description here

4) Enter that into any browser to view TensorBoard:

enter image description here

Special thanks to Sam Kirkiles

Andrew Schreiber
  • 14,344
  • 6
  • 46
  • 53
2

For anyone who must use the ssh keys (for a corporate server).

Just add -i /.ssh/id_rsa at the end.

$ ssh -N -f -L localhost:8211:localhost:6007 myname@servername -i /.ssh/id_rsa

aerin
  • 20,607
  • 28
  • 102
  • 140
1

While running the tensorboard give one more option --host=ip of your system and then you can access it from other system using http://ip of your host system:6006

user2973566
  • 11
  • 2