Pyodide filesystem for NLTK resources : missing files

Question

I am trying to use NLTK in browser, thanks to pyodide. Pyodide starts well, manages to load NLTK, print its version.

Nevertheless, while the package downloading seems fine, when invoking nltk.sent_tokenize(str), NLTK raises the error that it can't find the package "punkt".

I would say the downloaded resource is lost somewhere, but I didn't understand well how Pyodide / WebAssembly manage files. Any insights ?

Simple version:

import nltk
nltk.download(pkg)
for sent in nltk.sent_tokenize("Test string"):
    print(sent)

Version with more details, specifying download directory and server url.

import nltk
pkg = "punkt"
downloader = nltk.downloader.Downloader(server_index_url="https://raw.githubusercontent.com/nltk/nltk_data/gh-pages/index.xml") 
downloader.download(pkg, download_dir='/nltk_data')
downloader.status(pkg)
for sent in nltk.sent_tokenize("Test string"):
    print(sent)

Full sample code:

<!DOCTYPE html>
<html>
  <body>
    <script type="text/javascript" src="https://cdn.jsdelivr.net/pyodide/v0.18.0/full/pyodide.js"></script>
    <script type="text/javascript">
      // init Pyodide
      async function pyodide_loader() {
        let pyodide_premise = loadPyodide({
          indexURL: "https://cdn.jsdelivr.net/pyodide/v0.18.0/full/",
        });
        let pyodide = await pyodide_premise;
        await pyodide.loadPackage("micropip");
        await pyodide.loadPackage("nltk");
        return pyodide_premise;
      }
      let pyodideReadyPromise = pyodide_loader();

      
      // run Python code and load NLTK
      async function load_packages() {
        let pyodide = await pyodideReadyPromise;
        let output = pyodide.runPython(`
print(f"*** import nltk")
import nltk
print(f"*** NLTK version {nltk.__version__=} imported, downloading resources now")

pkg = "punkt"
nltk.download(pkg)

str = "Just for testing"
for sent in nltk.sent_tokenize(str):
    print(sent)
      `);
      }
      load_packages()
    </script>
  </body>
</html>

score 3 · Accepted Answer · answered Sep 02 '21 at 14:53

Short answer is that downloading files with Python currently won't work in Pyodide because http.client, requests etc require POSIX sockets which are not supported in the browser VM.

It's curious that nltk.download doesn't error though -- it should have.

The workaround is to manually download the needed resources, for instance, using the JavaScript fetch API as illustrated in this comment;

from js import fetch

response = await fetch("<url>")
js_buffer = await response.arrayBuffer()
py_buffer = js_buffer.to_py()  # this is a memoryview
stream = py_buffer.tobytes()  # now we have a bytes object

# that we can finally write under the appropriate path
with open("<file_path>", "wb") as fh:
    fh.write(stream)

I didn't understand well how Pyodide / WebAssembly manage files.

By default it's virtual file-system (MEMFS) that gets reset at each page load. You can access it with standard python tools (open, 'os', etc). If necessary you can also mount a persistent filesystem.

I managed to solve it by downloading the files, with js.window.fetch. Thanks ! — Sylvain, Sep 03 '21 at 19:58
How do you await a function in Pyodide ? I always run into issues like `RuntimeWarning: coroutine '' was never awaited`. — Sylvain, Sep 03 '21 at 20:09
I was using `pyodide.runPython()` instead of the *async* version `pyodide.runPythonAsync()` — Sylvain, Sep 07 '21 at 09:10

score 3 · Answer 2 · answered Nov 14 '21 at 22:03

Here is a working example of loading punkt with pyodide v0.18.1. I tried to post this as a comment to @rth's accepted answer, but the number of characters exceeded the 240 chars limit.

from js import fetch
import nltk
from pathlib import Path
import os, sys, io, zipfile

response = await fetch('https://raw.githubusercontent.com/nltk/nltk_data/gh-pages/packages/tokenizers/punkt.zip')
js_buffer = await response.arrayBuffer()
py_buffer = js_buffer.to_py()  # this is a memoryview
stream = py_buffer.tobytes()  # now we have a bytes object

d = Path("/nltk_data/tokenizers")
d.mkdir(parents=True, exist_ok=True)

Path('/nltk_data/tokenizers/punkt.zip').write_bytes(stream)

# extract punkt.zip
zipfile.ZipFile('/nltk_data/tokenizers/punkt.zip').extractall(
    path='/nltk_data/tokenizers/'
)

# check file contents in /nltk_data/tokenizers/
# print(os.listdir("/nltk_data/tokenizers/punkt"))

nltk.word_tokenize("some text here")

I received a lot of help from the pyodide maintainers and other wonderful people at https://github.com/pyodide/pyodide/issues/1798 to figure this out. Thanks!

Pyodide filesystem for NLTK resources : missing files

2 Answers2

Linked