How to setup learning environment for Udacity Deep Learning class with TensorFlow (Windows)

Question

I believe many of those interested in studying DL heard of this course:

https://www.udacity.com/course/deep-learning--ud730

I am taking the course now and would like to share step-by-step instruction on how to setup learning environment on Windows from scratch.

The first answer named SETTING UP THE ENVIRONMENT is about setting up the learning environment. You run it only once.
The second answer named AFTER LOCAL MACHINE REBOOT is about how to start the environment over after you reboot your computer.
See the third answer named HOW IT ALL WORKS to learn how all that stuff works (or you can follow the first answer blindly and check it out later).

score 5 · Accepted Answer · edited May 23 '17 at 10:31

SETTING UP THE ENVIRONMENT (run it only once!)

N.B. To start ready environment after computer reboot, use AFTER LOCAL MACHINE REBOOT instruction in the second answer.

Steps:

Download and setup Docker Toolbox:

https://www.docker.com/products/docker-toolbox

Docker is a tool to deploy preconfigured virtual learning environment on your machine. It will be running inside a virtual machine and will not mess with your computer anyhow.

(optional step) Docker will put it's files on system disk (C:) and you might want to change that if using SSD. You can do it that way:

mklink /J "C:\Users\USER\.docker" "D:\Docker"

substitute USER with your username
substitute "D:\Docker" with a path on other drive where you would like to store Docker files

More at: Change .docker directory on Windows

Open Windows CMD. Go to folder where Docker is installed. Create a new docker machine:

docker-machine create vdocker -d virtualbox

(magic step) Just run it!

FOR /f "tokens=*" %i IN ('docker-machine env --shell cmd vdocker') DO %i

More at: How do I start tensorflow docker jupyter notebook

Download and install preconfigured assignment docker image:

docker run -it -p 8888:8888 -p 6006:6006 --name tensorflow-udacity -it b.gcr.io/tensorflow-udacity/assignments:0.5.0

(important step!) Configure port forwarding:
- Run Oracle VM VirtualBox link (should be created when installing Docker):

Go to Settings... of vdocker machine:

Add port forwarding (it will forward 8888 port in virtual environment to 8810 port on your local machine):

P.S. Using :8810 port in case you already have IPython notebook installed on your local machine.

In the Settings... menu (from the previous step) allow virtual machine more memory:

N.B. The VirtualBox has to be shut down before you can make any changes to system settings. (by jlarsch)

Use the following command to stop the VM:

docker-machine stop vdocker

(optional) You can also allow it to use more cores in order to run faster:

Profit!

cool! works like a charm. I am kind of blindly following. what is the second port for?: `-p 6006:6006` — jlarsch, Sep 21 '16 at 08:40
@jlarsch That's kind of a mystery to me. It just works and I believe it's safe. — Denis Kulagin, Sep 21 '16 at 08:42
seems like by default, memory allocation to docker isn't sufficient. did you figure out how to allow docker more memory? — jlarsch, Sep 21 '16 at 09:02
@jlarsch Thanks for drawing my attention to memory issues. Added it to the instruction (step #7). — Denis Kulagin, Sep 21 '16 at 09:10
cool. It's worth pointing out that the virtualBox has to be shut down before you can make any changes to system settings. — jlarsch, Sep 21 '16 at 09:14

score 1 · Answer 2 · answered Sep 21 '16 at 08:38

1

AFTER LOCAL MACHINE REBOOT

To start learning environment after computer reboot, create .bat file (I call it udacity-tf-start.bat) with the following content:

call docker-machine start vdocker

FOR /f "tokens=*" %%i IN ('docker-machine env --shell cmd vdocker') DO %%i

call docker start -ai tensorflow-udacity

Important! %% is a kind of escaping and you only need it inside a BAT file. In case you are running the same set of commands via command-line, you should use:

FOR /f "tokens=*" %i IN ('docker-machine env --shell cmd vdocker') DO %i

answered Sep 21 '16 at 08:38

Denis Kulagin

8,472
17
60
129

are changes made to files saved in between sessions? i.e. the test data from assignment 1 for assignment 2? – jlarsch Sep 21 '16 at 08:59
@jlarsch Absolutely! That's why two separate instructions. If you installing an image downloaded from internet (or cached on your computer), it erases all your data and replaces it with source files. In case you follow *after reboot* steps, your data is persistent between sessions. – Denis Kulagin Sep 21 '16 at 09:02

Oliver Zendel · Answer 3 · 2016-11-30T02:54:58.363

Complementing to the other answers here is my starting script for creation/running/starting of a docker machine. Setup proceedure now boils down to installing the latest version of docker toolbox (this should autom. install vbox) from https://docs.docker.com/toolbox/toolbox_install_windows/ and running the script:

@echo off
set DOCKERMACHINENAME=tensorflow-udacity
set REPOSITORY=gcr.io/tensorflow/udacity-assignments:0.6.0
set "LOCALDIR0=/%SystemDrive:~0,1%/"
call :LoCase LOCALDIR0
SET "LOCALDIR=%LOCALDIR0%Users/%USERNAME%"
docker-machine.exe env %DOCKERMACHINENAME% > nul 2> nul
if "%errorlevel%"=="0" goto m_exists
::Machine has to be created
docker-machine create -d virtualbox --virtualbox-memory 8196 %DOCKERMACHINENAME%

:m_exists

::Check if machine needs to be restarted
docker-machine ip %DOCKERMACHINENAME% > nul 2>nul
if not "%errorlevel%"=="0" (docker-machine.exe restart %DOCKERMACHINENAME%)

FOR /F "tokens=*" %%i IN ('docker-machine env --shell cmd %DOCKERMACHINENAME%') DO %%i
FOR /F "tokens=*" %%F IN ('docker-machine ip %DOCKERMACHINENAME%') DO (SET DOCKERMACHINEIP=%%F)
echo Access to iPython: %DOCKERMACHINEIP%:8888

docker inspect %DOCKERMACHINENAME% > nul 2> nul
if "%errorlevel%"=="0" goto m_started
:: Machine has to be started
docker run -p 8888:8888 --name %DOCKERMACHINENAME% -v %LOCALDIR%:/mnt/hosttmp:rw -it %REPOSITORY%
goto finished

:m_started
docker start -ai %DOCKERMACHINENAME%
goto finished

:LoCase
FOR %%i IN ("A=a" "B=b" "C=c" "D=d" "E=e" "F=f" "G=g" "H=h" "I=i" "J=j" "K=k" "L=l" "M=m" "N=n" "O=o" "P=p" "Q=q" "R=r" "S=s" "T=t" "U=u" "V=v" "W=w" "X=x" "Y=y" "Z=z") DO CALL SET "%1=%%%1:%%~i%%"

:finished
::hint: to remove container use: docker rm %DOCKERMACHINENAME%

Lowercase transformation code is copied from http://www.robvanderwoude.com/battech_convertcase.php — Oliver Zendel, Nov 28 '16 at 13:16

Denis Kulagin · Answer 4 · 2016-09-21T11:39:12.510

HOW IT ALL WORKS

Disclaimer: It may somewhat resemble the plot of Christopher Nolan's Inception movie.

The overall picture

Details

Due to some limitations of Windows operating system Docker couldn't be run there natively (yet). That's why we first create a virtual box:

docker-machine create vdocker -d virtualbox

The next step (denoted as magic step) sets some environment variables for docker command to be able to connect to docker daemon running inside the virtual box:

FOR /f "tokens=*" %i IN ('docker-machine env --shell cmd vdocker') DO %i

>SET DOCKER_TLS_VERIFY=1

>SET DOCKER_HOST=tcp://192.168.99.100:2376

>SET DOCKER_CERT_PATH=C:\Users\USER\.docker\machine\machines\vdocker

>SET DOCKER_MACHINE_NAME=vdocker

Then we run:

docker run -it -p 8888:8888 -p 6006:6006 --name tensorflow-udacity -it b.gcr.io/tensorflow-udacity/assignments:0.5.0

which creates a docker container named tensorflow-udacity from an image it downloads from the specified URL. Important! That container runs inside a virtual box.

Pay attention to -p flags:

-p 8888:8888 -p 6006:6006

it tells docker daemon to forward (publish) a container's port 8888 to host (virtual box) port 8888. It's not available on the Windows machine yet!

Now we add one more port forwarding rule to the virtual box settings:

How to setup learning environment for Udacity Deep Learning class with TensorFlow (Windows)

4 Answers4

Linked