Download all files in a path on Jupyter notebook server

Question

As a user in a class that runs Jupyter notebooks for assignments, I have access to the assignments via the web interface. I assume the assignments are stored somewhere in my personal space on the server, and so I should be able to download them. How can I download all files that are in my personal user space? (e.g., wget)

Here's the path structure:

https://urltoserver/user/username

There are several directories: assignments, data, etc.

https://urltoserver/user/username/assignments

https://urltoserver/user/username/data

...

I want to download all the folders (recursively). Just enough that I can launch whatever I see online locally. If there are some forbidden folders, then ok, skip those and download the rest.

Please specify the command exactly as I couldn't figure it out myself (I tried wget)

I think it would be extremely useful if we can select multiple files and click "download" to get them all. However I think this is not supported by Jupyter notebook yet. — zyy, Jan 23 '21 at 17:00
@zyy If you install the extension `'jupyter-archive'`, you can do that directly for an individual directory now **in JupyterLab**. See [jupyter-archive](https://github.com/jupyterlab-contrib/jupyter-archive#jupyter-archive). There's an animation that shows the option that gets added to the drop-down menu for JupyterLab. — Wayne, May 16 '22 at 17:44

Serzhan Akhmetov · Answer 1 · 2018-04-14T16:04:35.557

231

Try running this as separate cell in one of your notebooks:

!tar chvfz notebook.tar.gz *

If you want to cover more folders up the tree, write ../ before the * for every step up the directory. The file notebook.tar.gz will be saved in the same folder as your notebook.

edited Apr 14 '18 at 16:04

answered Nov 17 '17 at 17:00

Serzhan Akhmetov

2,898
2
13
26

This works well. Thanks. But... my resulting tarball is sitting on my ec2 instance on AWS. What is the easiest way to get it from there to my local machine? – Kevin Glynn Jan 18 '18 at 04:21
5

From within the Jupyter notebook go to File -> Open. This will open up a new browser tab. From there click the checkbox next to your fresh tar.gz. and a 'download' button will appear at the top. Click it, specify local path and save. – wmaddox Jan 19 '18 at 03:39
7

FYI, !tar chvfz notebook.tar.hz * will pull in files that are symbolic links as well, so you won't have broken images. – Jim B. Feb 19 '18 at 04:17
The problem is that it cannot fetch files in alias. – Lei Zhang Feb 23 '18 at 08:05
2

if it's not following the link files, use `h` option of `tar` command – Abhisek Apr 14 '18 at 15:42
Probably a stupid question, but how can i open this on my local maschine again in my local juypter notebook? – PV8 Apr 30 '19 at 09:36
1

@PV8 I'm guessing you saved your notebook.tar.gz somewhere on your machine and extracted the ipynb in /some/directory. Then just follow the instructions here: https://jupyter-notebook-beginner-guide.readthedocs.io/en/latest/execute.html – Serzhan Akhmetov Apr 30 '19 at 09:46
how to handle it, if the file excee 250MB? is there a automatic splitter for it? – PV8 Apr 30 '19 at 12:12
1

@PV8 in that case I suggest you formulate your question in full and post in on SO – Serzhan Akhmetov Apr 30 '19 at 12:32
To decompress once downloaded to local: `tar -zxvf notebook.tar.gz` – swimmer Apr 01 '21 at 07:20

score 49 · Answer 2 · answered May 27 '19 at 01:58

I am taking Prof. Andrew Ng's Deeplearning.ai program via Coursera. The curriculum uses Jupyter Notebooks online. Along with the notebooks are folders with large files. Here's what I used to successfully download all assignments with the associated files and folders to my local Windows 10 PC.

Start with the following line of code as suggested in the post by Serzan Akhmetov above:

!tar cvfz allfiles.tar.gz *

This produces a tarball which, if small enough, can be downloaded from the Jupyter notebook itself and unzipped using 7-Zip. However, this course has individual files of size 100's of MB and folders with 100's of sample images. The resulting tarball is too large to download via browser.

So add one more line of code to split files into manageable chunk sizes as follows:

!split -b 50m allfiles.tar.gz allfiles.tar.gz.part.

This will split the archive into multiple parts each of size 50 Mb (or your preferred size setting). Each part will have an extension like allfiles.tar.gz.part.xx. Download each part as before.

The final task is to untar the multi-part archive. This is very simple with 7-Zip. Just select the first file in the series for extraction with 7-Zip. This is the file named allfiles.tar.gz.part.aa for the example used. It will pull all the necessary parts together as long as they are in the same folder.

Hope this helps add to Serzan's excellent answer above.

For those who don't want to use 7-Zip you can stay in unix/linux and use ```!cat allfiles* > your_file_name.gz```. This combines everything starting with allfiles in the directory into one file — Tanya Gupta, May 03 '20 at 16:40
But this seem to only create the tar on the remote server, how to download them to the local PC? — Christy Lee, Aug 17 '20 at 06:18
right-clicking on the tar file will give a set of options including download @ChristyLee — Tejas Shetty, Jun 12 '21 at 18:15

score 21 · Answer 3 · answered Feb 22 '18 at 01:43

21

You can create a new terminal from the "New" menu and call the command described on https://stackoverflow.com/a/47355754/8554972:

tar cvfz notebook.tar.gz *

The file notebook.tar.gz will be saved in the same folder as your notebook.

answered Feb 22 '18 at 01:43

Rafael Miquelino

341
2
6

1

Change `cvfz` to `chvfz` will download files that are symbolic links. – Hongbo Miao Oct 17 '20 at 19:42

Alex · Answer 4 · 2019-03-11T16:15:19.410

2

The easiest way is to archive all content using tar, but there is also an API for files downloading.

GET /files/_FILE_PATH_

To get all files in folder you can use:

GET /api/contents/work

Example:

curl https://server/api/contents?token=your_token
curl https://server/files/path/to/file.txt?token=your_token --output some.file

Source: Jupyter Docs

edited Mar 11 '19 at 16:15

answered Mar 10 '19 at 00:46

Alex

1,986
22
23

score 2 · Answer 5 · answered May 16 '22 at 12:54

2

you just need to do

zip -r filename.zip folder_name

answered May 16 '22 at 12:54

Meet Patel

21
1

1

Why do you prefer this over the `tar` based commands suggested previously, which have been validated repeatedly by the community? Can you [edit] your answer to provide an explanation of when this might be more appropriate? – Jeremy Caney May 16 '22 at 17:07
1

I always use this method, so guess therefore, it is a proven method! – Rivered Oct 07 '22 at 06:48

score 1 · Answer 6 · answered Feb 16 '18 at 12:58

1

Try first to get the directory by:

import os
os.getcwd()

And then use snipped from How to create a zip archive of a directory. You can download complete directory by zipping it. Good luck!

answered Feb 16 '18 at 12:58

Eddmik

155
7

Santiago Torres · Answer 7 · 2022-05-12T05:09:21.760

1

from google.colab import files

files.download("/content/data.txt")

These lines might work if you are working in a google colab or Jupyter notebook.

The first line imports the library files The second one, downloads your created file, example:"data.txt" (your file name) located inside content folder.

edited May 12 '22 at 05:09

answered Oct 15 '20 at 05:17

Santiago Torres

11
2

2

Although this code might solve the problem, a good answer should also explain **what** the code does and **how** it helps. – BDL Oct 15 '20 at 08:34
This made me laugh. – Greg432 Nov 23 '22 at 18:31

score 0 · Answer 8 · answered Mar 27 '17 at 10:09

I don't think this is possible with wget, even with the wget -r option. You may have to download them individually (using the Download option in the dashboard view (which is only available on single, non-directory, non-running notebook items) if that is available to you.

However, it is likely that you are not able to download them since if your teacher is using grading software like nbgrader then the students having access to the notebooks themselves is undesirable - since the notebooks can contain information about the answers as well.

score 0 · Answer 9 · edited Aug 01 '21 at 03:45

0

I've made a slightly update based on @Sun Bee's solution, and it will allow you to create multiple file backup with a timestamp subfix.

!tar cvfz allfiles-`date +"%Y%m%d-%H%M"`.tar.gz *

edited Aug 01 '21 at 03:45

Dharman

30,962
25
85
135

answered Aug 01 '21 at 03:39

Eric

2,636
21
25

score 0 · Answer 10 · answered Feb 14 '23 at 18:23

0

The above solution didn't work for me (Windows). When I extracted the tar.gz file, it produced a single file and not the folder hierarchy you see in jupyter. The following links should help:

Coursera support - link
Youtube vid - link

answered Feb 14 '23 at 18:23

Sai Siddhaarth

1
1

Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community Feb 18 '23 at 16:20

Ed Norris · Answer 11 · 2023-08-10T15:06:52.943

0

Since this question was asked, Coursera has added a Lab Files tab to some of the notebooks, which makes it trivial to download everything by clicking Download all files. Check this first to save yourself some hassle.

Instructions from Coursera:

edited Aug 10 '23 at 15:06

answered Aug 09 '23 at 20:43

Ed Norris

4,233
5
27
29

Download all files in a path on Jupyter notebook server

11 Answers11

Linked