Select and read a file from user's filesystem

Question

I need to:

Enable the user to select a plain text file from the user's filesystem.
Make it available to Pyodide.
Read its contents line-by-line in Pyodide (in the Python code).

The code to read currently replaced by dummy code: inp_str = 'ACGTACGT'. The actual use case involves complicated processing of the text file, but the minimal working example simply converts the input to lowercase.

<!doctype html>
<html>
  <head>
      <script src="https://cdn.jsdelivr.net/pyodide/v0.22.1/full/pyodide.js"></script>
  </head>
  <body>
    Analyze input <br>
    <script type="text/javascript">
      async function main(){
          let pyodide = await loadPyodide();
          let txt = pyodide.runPython(`
    # Need to replace the line below with code for the user to select
    # the file from the user's filesystem and read 
    # its contents line-by-line:
    inp_str = 'ACGTACGT'

    out_str = inp_str.lower()
    with open('/out.txt', 'w') as fh:
        print(out_str, file=fh)
    with open('/out.txt', 'rt') as fh:
        out = fh.read()
    out
`);

          const blob = new Blob([txt], {type : 'application/text'});
          let url = window.URL.createObjectURL(blob);
          
          var downloadLink = document.createElement("a");
          downloadLink.href = url;
          downloadLink.text = "Download output";
          downloadLink.download = "out.txt";
          document.body.appendChild(downloadLink);
      }
      main();
    </script>
  </body>
</html>

We have external users that may not be advanced computer users. We can specify they need to use Google Chrome browser, but not specific releases like Chrome Canary. We cannot ask them to manually enable the File System API.

Based on the suggestion by TachyonicBytes, I tried the code below. Now I got the error below. I also cannot see something like select file button, or any obvious code for it:

Uncaught (in promise) DOMException: Failed to execute 'showDirectoryPicker' on 'Window': Must be handling a user gesture to show a file picker.
    at main (file:///Users/foo/bar/upload_nativefs.html:10:35)
    at file:///Users/foo/bar/upload_nativefs.html:26:7

This is line 10 referred to in the error message:

const dirHandle = await showDirectoryPicker();

And the full page is pasted below:

<!doctype html>
<html>
  <head>
      <script src="https://cdn.jsdelivr.net/pyodide/v0.22.1/full/pyodide.js"></script>
  </head>
  <body>
    Analyze input <br>
    <script type="text/javascript">
      async function main(){
          const dirHandle = await showDirectoryPicker();
          if ((await dirHandle.queryPermission({ mode: "readwrite" })) !== "granted") {
              if (
                  (await dirHandle.requestPermission({ mode: "readwrite" })) !== "granted"
              ) {
                  throw Error("Unable to read and write directory");
              }
          }    
          let pyodide = await loadPyodide();
          const nativefs = await pyodide.mountNativeFS("/mount_dir", dirHandle);
          
          pyodide.runPython(`
  import os
  print(os.listdir('/mount_dir'))
`);
      }
      main();
    </script>
  </body>
</html>

@MisterJojo Thank you for the link, but I could not figure out how to apply it in my case. If you have an answer, I would be very grateful if you post it below. Again, my main task is to enable the user to select a file from their local filesystem, and to make the file available for reading in pyodide (python) code. Thank you for the help! — Timur Shtatland, Mar 23 '23 at 18:12
I focused your question to your main ask, how to pick the file. Additionally questions like your (b) should be separate questions that reference this question — James Risner, Mar 24 '23 at 11:03
I am using Google Chrome Version 111.0.5563.110 (Official Build) (arm64), if this is relevant. — Timur Shtatland, Mar 24 '23 at 20:14

score 1 · Answer 1 · answered Mar 21 '23 at 23:51

The main documentation page for pyodide and the filesystem is here.

Assuming you are using Chrome, which has the File System Access API implemented, it seems that you can mount the filesystem with this code:

const dirHandle = await showDirectoryPicker();

if ((await dirHandle.queryPermission({ mode: "readwrite" })) !== "granted") {
  if (
    (await dirHandle.requestPermission({ mode: "readwrite" })) !== "granted"
  ) {
    throw Error("Unable to read and write directory");
  }
}

const nativefs = await pyodide.mountNativeFS("/mount_dir", dirHandle);

pyodide.runPython(`
  import os
  print(os.listdir('/mount_dir'))
`);

Afterwards, read your file and get the input string as you normally would:

with open("file") as f:
    f.read()

Now, in order to be sure you have maximum chances of this working, get the Chrome Canary, and manually enable the File System API: navigate to chrome://flags/#native-file-system-api and change the value to Enabled.

If you are not doing this for yourself, unfortunately, not all browsers can access the native filesystem, for security reasons, so you may have to go with an alternative solution, like IDBFS.

It seems that this question has a similar problem to your new problem: https://stackoverflow.com/questions/69093088 I would suggest attaching the full code to something like pastebin, I think stackoverflow truncated a large part of it (after `main()`). — TachyonicBytes, Mar 22 '23 at 22:52
I added the full code by adding an "EDIT" below my OP. Also, thank you for suggesting the question with a similar problem. Unfortunately, I could not figure out how to quickly apply it in my case. This may be due to my lack of experience with Javascript and Pyodide. Thank you for your help! — Timur Shtatland, Mar 23 '23 at 18:09

score 1 · Answer 2 · answered Mar 24 '23 at 05:36

I know you are looking for a pure pyodide solution but I would like to suggest you to have a look into pyscript solution which provides very minimal solution as per your requirement and loads the file in the virtual file system which is the file system that runs inside the browser.

Pyscript supports the Python Standard Library File and Directory APIs. These APIs access storage within the Virtual File System. The virtual file system is provided by Emscripten File System API.

Live Demo :

<html>
  <head>
    <title>Read File</title>
    <meta charset="utf-8">
    <link rel="stylesheet" href="https://pyscript.net/latest/pyscript.css" />
    <script defer src="https://pyscript.net/latest/pyscript.js"></script>
  </head>
  <body>
    <label for="myfile">Select a file:</label>
    
    <input type="file" id="fileinput" name="fileInput"/>
    <br />
    <br />
    <div id="print_output"></div>
    <br />
    <p>File Content:</p>
    <div id="content">
    </div>

    <py-script output="print_output">
      import asyncio  
      import pyodide
      import js
      
      async def get_file(e):
        files = e.target.files.to_py()
        for file in files:
          file_content = await file.text()
          js.document.getElementById("content").innerHTML = file_content

      def main():
        get_file_proxy = pyodide.ffi.create_proxy(get_file)
        js.document.getElementById("fileinput").addEventListener("change", get_file_proxy)

      main()
    </py-script>
  </body>
</html>

But if you still want to make it work in a pure pyodide way, You can modify it as per the documentation mentioned here.

TachyonicBytes · Accepted Answer · 2023-03-25T00:13:39.873

So, I am adding another answer, because I think I fully solved it.

You error comes from the fact that accessing the filesystem with await showDirectoryPicker(); has to be invoked by the user (cannot be invoked directly). So, the solution is to use a user event (such as clicking a button) for the user to trigger the function. I added the full solution here.

Basically, I replaced the main call with the button.

<!doctype html>
<html>
  <head>
      <script src="https://cdn.jsdelivr.net/pyodide/v0.22.1/full/pyodide.js"></script>
  </head>
  <body>
    <button>Analyze input</button>
    <script type="text/javascript">
      async function main(){
          const dirHandle = await showDirectoryPicker();
          if ((await dirHandle.queryPermission({ mode: "readwrite" })) !== "granted") {
              if (
                  (await dirHandle.requestPermission({ mode: "readwrite" })) !== "granted"
              ) {
                  throw Error("Unable to read and write directory");
              }
          }
          let pyodide = await loadPyodide();
          const nativefs = await pyodide.mountNativeFS("/mount_dir", dirHandle);

          pyodide.runPython(`
            import os
            print(os.listdir('/mount_dir'))
          `);
      }

      const button = document.querySelector('button');
      button.addEventListener('click', main);
    </script>
  </body>
</html>

On a fairly recent Chromium, after you click the button, you will get the system file picker in order to select a directory to mount. Afterward, you will have some pop-ups explaining what is happening. Something like this:

After you accept, the end result will look like this:

, meaning your browser has access to the filesystem, and you can use the other python functions to work with it.

Edit:

Now that I understand the problem a little better, I made the code get the file selection picker. Unfortunately, it seems that the filesystem api makes it hard to get the directory from the file, so there is still a call to the directory picker. I made did some error handling in case the file is not in the selected directory, but for now, the ergonomics are lacking in that respect.

The javascript looks like this:

      // Check if handle exists inside directory our directory handle
      async function inDirectory(directoryHandle, fileHandle) {
        return await directoryHandle.resolve(fileHandle) === null;
      }

      async function main() {
        const dirHandle = await showDirectoryPicker();
        const [fileHandle] = await showOpenFilePicker();
        const bad = await inDirectory(dirHandle, fileHandle);
        if (bad) {
          throw Error("The file is not in the mounted directory");
          return;
        }

        const fileData = await fileHandle.getFile();
        console.log(fileData.name);
          if ((await fileHandle.queryPermission({ mode: "readwrite" })) !== "granted") {
              if (
                  (await fileHandle.requestPermission({ mode: "readwrite" })) !== "granted"
              ) {
                  throw Error("Unable to read and write directory");
              }
          }
          let pyodide = await loadPyodide();
          const nativefs = await pyodide.mountNativeFS("/mount_dir", dirHandle);

          pyodide.runPython(`
            import os
            print(os.listdir('/mount_dir'))
            print(open('/mount_dir/${fileData.name}').read())
          `);
      }

      const button = document.querySelector('button');
      button.addEventListener('click', main);

I also updated the repo with it. If this is still lacking in some respect, please detail the question even further.

Edit 2:

Well, the code ended up being much simpler. I did away with the directory entirely, now it asks you to pick a file, it reads the contents into JS memory, it constructs the Python conversion function and then it just calls the function and puts the result in Blob.

The code to construct the download link appears after the Analyze button has been pressed.

This is the code, as usually, updated in the repo as well:

      async function main() {
        // Get the file contents into JS
        const [fileHandle] = await showOpenFilePicker();
        const fileData = await fileHandle.getFile();
        const contents = await fileData.text();

        // Create the Python convert toy function
        let pyodide = await loadPyodide();
        let convert = pyodide.runPython(`
from pyodide.ffi import to_js
def convert(contents):
    return to_js(contents.lower())
convert
      `);

        let result = convert(contents);
        console.log(result);

        const blob = new Blob([result], {type : 'application/text'});

        let url = window.URL.createObjectURL(blob);

        var downloadLink = document.createElement("a");
        downloadLink.href = url;
        downloadLink.text = "Download output";
        downloadLink.download = "out.txt";
        document.body.appendChild(downloadLink);

      }
      const button = document.querySelector('button');
      button.addEventListener('click', main);

The code ended up much, much cleaner.

Thank you for the detailed answer, and for the link to GitHub! I appreciate your patience in helping me with my question. I used your code, and got as far as you described in your answer. According to the pop-up messages and the console log, the code can read the contents of the *directory*. But I still cannot get to the *file selection* window (to read from the selected file). Thank you in advance for the help! — Timur Shtatland, Mar 24 '23 at 19:54
I missed that from the original question, I updated the answer now with the file picker. — TachyonicBytes, Mar 24 '23 at 21:15
Thank you so much for the update! I think I am getting much closer. (a) As you mentioned, the ergonomics are still lacking. Namely, the user has to navigate twice to the desired directory: once to select the directory, and the second time to select the file. Could this possibly be made into a single navigation sequence? (b) The "Download output" link in my original post is missing in your code. I want to convert the input to all lowercase (in this minimal simplified code - the actual case is much more complex), and enable the user to click to download the result. Thank you very much again! — Timur Shtatland, Mar 24 '23 at 21:54
I edited the answer again. The code ended up much simpler, let me know if it does what the question asks. — TachyonicBytes, Mar 25 '23 at 00:14
Thank you very much, it does most of what I need! I can take it from here. — Timur Shtatland, Mar 27 '23 at 20:05

Select and read a file from user's filesystem

3 Answers3

Linked