How to run py-faster-rcnn with X11 forwarding

Question

I'm running py-faster-rcnn with cuDNN enabled on a g2.8xlarge EC-2 instance with Ubuntu 14.04 operating system. Everything's compiled and seems to be working fine. I log in to the remote instance via:

ssh -X -i "<key.pem>" ubuntu@<IP address>

I also enter the command: export DISPLAY=:0

Running ./tools/demo.py the output looks good:

Loaded network /home/ubuntu/py-faster-rcnn/data/faster_rcnn_models/VGG16_faster_rcnn_final.caffemodel
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Demo for data/demo/000456.jpg
Detection took 0.543s for 300 object proposals 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Demo for data/demo/000542.jpg
Detection took 0.506s for 161 object proposals
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Demo for data/demo/001150.jpg
Detection took 0.507s for 194 object proposals
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Demo for data/demo/001763.jpg
Detection took 0.507s for 196 object proposals
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Demo for data/demo/004545.jpg
Detection took 0.541s for 300 object proposals

But graphical output is not rendered in my XQuartz window.

Has anyone else solved this? Need to figure out how to use AWS hardware but with local visualization capabilities. Checked here, but I wasn't able to solve my problem: BVLC/caffe#861

EDIT

Here are links to the my remote sshd_config and local ssh_config files.

Here are the results of the requested tests in the remote AWS server env:

$ echo $DISPLAY 
localhost:10.0

and

$ DISPLAY=localhost:10.0 xhost && echo success
access control enabled, only authorized clients can connect
success

UPDATE

Running the commands xeyes and xcalc on the remote machine after applying the above steps results in the expected output (eyes in the first place, a calculator in the second) on the local client. This is probably a python problem. Going to start looking there.

SOLUTION--UPDATE PYTHON LIBRARIES

After verifying that my system was setup to support X11 forwarding with the guidance of those who responded to this post, I focus on running a series of tests in python to see if matplotlib was playing well with X11. You can check this yourself by running this script interactively. If xcalc and xeyes work as expected, but this script produces an error, the problem lies with python/matplotlib.

I've since fixed the problem so I don't have the error this produced on hand, but the steps to fix on an Ubuntu 14.04, g2.8xlarge EC2 were as follows:

• Install python gobject: sudo apt-get install python-gobject-dev
• Install python-tk: sudo apt-get install python-tk
• Install pygtk:
  • wget http://ftp.gnome.org/pub/GNOME/sources/pygtk/2.24/pygtk-2.24.0.tar.gz
  • tar -xvzf pygtk-2.24.0.tar.gz
  • cd pygtk-2.24.0
  • ./configure
  • make
  • sudo make install
  • cd
• If matplotlib was installed using a package manager, e.g. pip, uninstall it and re-install from source:
  • sudo pip uninstall matplotlib
  • git clone https://github.com/matplotlib/matplotlib.git
  • cd matplotlib
  • sudo python setup.py install
  • cd
• Not sure that this is necessary, but ran sudo apt-get install xorg openbox for good measure.

After going through the above steps, python ./tools/demo.py with the py-faster-rcnn root directory returns images of bounding boxes and class probabilities as expected.

Answer 1

I assume that there is an X server running on your local host, right?

There are 2 files to consider:

• /etc/ssh/ssh_config: it is located on your local host; it is used with the client command: ssh.
• /etc/ssh/sshd_config: it is located on the remote instance; it is used with the sshd server running on the remote instance.

X11Forwarding yes and X11DisplayOffset 10 are 2 values that need to be set for sshd: the ssh daemon running on the remote instance. Edit /etc/ssh/sshd_config file on the remote instance. Maybe the sshd must be restarted on the remote instance after the changes.

Connect again with ssh -X ..., and check the value of DISPLAY variable. The example below is correct when X11DisplayOffset 10 is set:

# ssh -X -i "<key.pem>" ubuntu@<IP address>
# echo $DISPLAY
localhost:10.0

If the DISPLAY variable is not set or set to another value (something in .bashrc .profile etc changed its value at login), then this is an issue.

Another way is to test with xhost, and set the environment in the same line:

# ssh -X -i "<key.pem>" ubuntu@<IP address>
# DISPLAY=localhost:10.0 xhost && echo success

If an error message is displayed, this is an issue to fix: edit your question and add the error message. Otherwise, a status line an the success word are printed, continue with:

# DISPLAY=localhost:10.0 xcalc

A window should open on the local host. Finally continue with:

# DISPLAY=localhost:10.0 ./tools/demo.py

Answer 2

I also enter the command: export DISPLAY=:0

This step could be your problem. When you're forwarding X through SSH, it's virtually never necessary to manually set DISPLAY on the remote system.

If ssh successfully forwards X, it will set the DISPLAY environment variable in the remote session to the correct value. If DISPLAY isn't set on the remote system, it means that ssh couldn't negotiate with the server to forward X (or else that something in your shell startup is removing the variable). By setting DISPLAY yourself, you may be overwriting the correct value of the variable. At the least, the value that you're setting it to isn't likely to be correct for forwarding through SSH.

The DISPLAY value ":0" means to connect through the system's X default, which might involve connecting through a unix-domain socket or through TCP port 6000. This is intended for clients to access an X server running on the same system as the client (the EC2 server in this case).

For an SSH session, the correct DISPLAY value will normally be something like "localhost:10", meaning that clients on the remote system should connect to localhost port 6010 (6000 + 10). The number 10 may be different if port 6010 is already in use or if the SSH server was configured to use a different port range.

If you're finding that DISPLAY is not set when you log in despite running ssh with the "-X" or "-Y" option, it's possible that X11 forwarding has been disabled in the ssh server. The OpenSSH server has three relevant options which can be set in sshd_config:

X11DisplayOffset
Specifies the first display number available for sshd(8)'s X11 forwarding. This prevents sshd from interfering with real X11 servers. The default is 10.

X11Forwarding
Specifies whether X11 forwarding is permitted. The argument must be “yes” or “no”. The default is “no”.

When X11 forwarding is enabled, there may be additional exposure to the server and to client displays if the sshd(8) proxy display is configured to listen on the wildcard address (see X11UseLocalhost below), though this is not the default. Additionally, the authentication spoofing and authentication data verification and substitution occur on the client side. The security risk of using X11 forwarding is that the client's X11 display server may be exposed to attack when the SSH client requests forwarding (see the warnings for ForwardX11 in ssh_config(5)). A system administrator may have a stance in which they want to protect clients that may expose themselves to attack by unwittingly requesting X11 forwarding, which can warrant a “no” setting.

Note that disabling X11 forwarding does not prevent users from forwarding X11 traffic, as users can always install their own forwarders. X11 forwarding is automatically disabled if UseLogin is enabled.

X11UseLocalhost
Specifies whether sshd(8) should bind the X11 forwarding server to the loopback address or to the wildcard address. By default, sshd binds the forwarding server to the loopback address and sets the hostname part of the DISPLAY environment variable to “localhost”. This prevents remote hosts from connecting to the proxy display. However, some older X11 clients may not function with this configuration. X11UseLocalhost may be set to “no” to specify that the forwarding server should be bound to the wildcard address. The argument must be “yes” or “no”. The default is “yes”.

The OpenSSH ssh client will normally connect to a server even if the server refused forwarding. This can hide that fact that you requested forwarding but the server refused. ssh has an option you can set in ssh_config to control this:

ExitOnForwardFailure
Specifies whether ssh(1) should terminate the connection if it cannot set up all requested dynamic, tunnel, local, and remote port forwardings, (e.g. if either end is unable to bind and listen on a specified port). Note that ExitOnForwardFailure does not apply to connections made over port forwardings and will not, for example, cause ssh(1) to exit if TCP connections to the ultimate forwarding destination fail. The argument must be “yes” or “no”. The default is “no”.