`pandas.read_csv` with `IPython` path problem

Question

Here is my ipython_config.py

$ cat ~/.ipython/profile_default/ipython_config.py
c = get_config()
c.InteractiveShellApp.exec_lines = [
    'import sys',
    'sys.path.append("/code")',
    'import pandas as pd',
]

Here is my file structure

# root @ 49a10d67a5d5 in /code on git:main x [15:12:53] 
$ tree
.
|-- errata
|   |-- test.ipynb
|   `-- top_funds.py
|-- skew_all.csv
`-- test.ipynb

1 directory, 4 files

Here is my ~/.profile grep

# root @ 49a10d67a5d5 in /code on git:main x [16:00:59] 
$ cat ~/.profile| grep PY
export PYTHONPATH=$PYTHONPATH:/code
export JUPYTER_PATH=$JUPYTER_PATH:/code

I want to read file /code/skew_all.csv and import modules in /code. Things are strange.

If I use ipython in the terminal with working directory /code, I can easily read it and import modules in /code.

# root @ 49a10d67a5d5 in /code on git:main x [14:42:01] 
$ ipython
Python 3.8.10 (default, Mar 13 2023, 10:26:41) 
Type 'copyright', 'credits' or 'license' for more information
IPython 8.12.2 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import sys

In [2]: sys.path
Out[2]: 
['/usr/local/bin',
 '/code',
 '/usr/lib/python38.zip',
 '/usr/lib/python3.8',
 '/usr/lib/python3.8/lib-dynload',
 '',
 '/usr/local/lib/python3.8/dist-packages',
 '/usr/lib/python3/dist-packages',
 '/code']

In [3]: pd.read_csv('skew_all.csv')
Out[3]: 
           date  skew_510300
0    2019-12-23    -0.097785
1    2019-12-24    -0.091429
2    2019-12-25    -0.111135
3    2019-12-26    -0.104044
4    2019-12-27    -0.151558
..          ...          ...
808  2023-04-26     0.035485
809  2023-04-27     0.008492
810  2023-04-28     0.024342
811  2023-05-04     0.049912
812  2023-05-05     0.030516

[813 rows x 2 columns]

In [4]: from errata.top_funds import *

In [5]:

If I use ipython in the terminal with working directory /code/errata, I can NOT read it BUT can import modules in /code.

# root @ 49a10d67a5d5 in /code/errata on git:main x [14:46:20] 
$ ipython
Python 3.8.10 (default, Mar 13 2023, 10:26:41) 
Type 'copyright', 'credits' or 'license' for more information
IPython 8.12.2 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import sys

In [2]: sys.path
Out[2]: 
['/usr/local/bin',
 '/code/errata',
 '/code',
 '/usr/lib/python38.zip',
 '/usr/lib/python3.8',
 '/usr/lib/python3.8/lib-dynload',
 '',
 '/usr/local/lib/python3.8/dist-packages',
 '/usr/lib/python3/dist-packages',
 '/code']

In [3]: pd.read_csv('skew_all.csv')
---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
Cell In[3], line 1
----> 1 pd.read_csv('skew_all.csv')
many tracebacks
FileNotFoundError: [Errno 2] No such file or directory: 'skew_all.csv'

In [4]: from errata.top_funds import *

In [5]:

If I use Jupyter notebook in VS Code with working directory /code, I can NOT read it too, BUT can import modules in /code. This is strange and unusual. But can be solved by setting Notebook File Root to ${workspaceFolder} thanks to @MingJie-MSFT

This file is /code/test.ipynb and screenshot was took with working directory of /code.

If I use Jupyter notebook in VS Code with working directory /code/errata, I can NOT read it too, BUT can import modules in /code.

And this can NOT be solved by setting Notebook File Root to ${workspaceFolder}, with working dir of /code/errata, but can be solved with working dir of /code.

This file is /code/errata/test.ipynb and screenshot was took with working directory of /code.

So the basic conclusion is path is working because of the successful module import, but read_csv not.

I guess there is something unclear with pandas.read_csv path handling. And probably some unknown difference between Jupyter notebook in VS Code and terminal ipython

My goal is to run pd.read_csv('skew_all.csv') and from errata.top_funds import * smoothly without writing any addition codes. The solution I've taken is the ipython_config.py but it has the above strange things.

An additional constrain, I don't want to write code with full path like `read_csv('/code/skew_all.csv')` — Dacian Peng, May 17 '23 at 07:00
An additional information, if I remove the `~/.ipython/profile_default/ipython_config.py`, `Jupyter notebook` in `VS Code` with working dir of `/code` can read the `.csv` as normal. — Dacian Peng, May 17 '23 at 07:04
Can you provide a simple directory structure? Would you mind using the `.env` file and using absolute paths in the env file at the same time. — MingJie-MSFT, May 17 '23 at 07:07
```# root @ 49a10d67a5d5 in /code on git:main x [15:16:16] C:1 $ env | grep PY PYTHONPATH=:/code JUPYTER_PATH=:/code``` — Dacian Peng, May 17 '23 at 07:16

score 1 · Answer 1 · answered May 17 '23 at 07:18

1

You can refer to this answer.

Open your settings and search for Notebook File Root

Then change it to ${workspaceFolder} and reload VSCode.

It seems that after updating, it runs based on the directory where the file is located as the root directory.

answered May 17 '23 at 07:18

MingJie-MSFT

5,569
1
2
13

well, I set it to `/code` or `${workspaceFolder}` both solved the condition 3&4. But disadvantages are 1. have to specify this settings in this workspace, this is an extra procedure 2. run `ipython` in `/code/errata` won't read the `.csv – Dacian Peng May 17 '23 at 07:24
I believe it is correct that `ipython` cannot read because they are not in the same directory unless you import an absolute path in the file or add the file path to the environment variable by `sys.path.append("..")`. – MingJie-MSFT May 17 '23 at 07:35
For your first point of view, I believe it is not a problem. If you do not modify certain settings, it will indeed not meet your needs. Moreover, changing it to `${workspaceFolder}` is consistent with the settings of the `.py` file. Python scripts in VSCode also use the workspace as the root directory. – MingJie-MSFT May 17 '23 at 07:38
The `~/.ipython/profile_default/ipython_config.py` have solved the path adding problem. Every time a new `ipython` process started, it executes `'sys.path.append("/code")'`. The `import sys, sys.path` has shown the correct path added. But it just doesn't work, I'm so confused. – Dacian Peng May 17 '23 at 07:38
Emmm, in fact this set of code is used by a team and everyone set it again, then explain this to them is hard. The docker method was token to make this set of code easy-to-use for everyone, and the `~/.ipython/profile_default/ipython_config.py` was set when constructing container. – Dacian Peng May 17 '23 at 07:42

`pandas.read_csv` with `IPython` path problem

1 Answers1