FileNotFoundError but the files exist in the folder

Question

I am getting this error:

FileNotFoundError: [Errno 2] File path/.csv does not exist: path/.csv

However the file actually is still there and I do not understand what it is wrong in my code for accessing it. Could you please have a look and see if you spot any error? Thank you

import pandas as pd
from os import listdir
from os.path import join, isfile
import os


def create_dataframe(paths):

    def get_files_in_path(path):
        return [f.split('.')[0] for f in listdir(path) if isfile(join(path, f))]

    dataframes = {
        (path, file): pd.read_csv(path + file + '.csv')
        for path in paths
        for file in get_files_in_path(path)
    }

    df = pd.concat(dataframes, names=['path', 'file', '_'])

paths = [f"path/My folder {f}/" for f in ['file1', 'file2', 'file3']]
data = create_dataframe(paths)

The error is in this line:

---> 18 for file in get_files_in_path(path)

The code should append in one unique dataframe all the csv files stored in file1, file2, file3 folders. The files are csv. They are called test+first.csv, another_test.csv. The path is path/My folder file1 and path/My folder file 2 and path/My folder file 3 . The expected output would be something like this (in terms of indices with path and file): path would be user_id and file would be date in the image below.

Answer 1

If you really want every csv inside every directory of path/ you could also consider going straight for a glob

That was the technique on a similar problem here https://stackoverflow.com/a/21232849/11199887

For your case it would look something like

import pandas as pd
from os.path import join
import glob

# the parent of the `/My folder` stuff
base_path = 'path/'
# the ** in combination with recursive=True descends into all directories
all_csv_filenames = glob.glob(
    join(base_path, "My folder*/**/*.csv"), recursive=True
)
df = pd.concat(
    (pd.read_csv(filename) for filename in all_csv_filenames),
    names=['path', 'file', '_'],
)

Which should match all csv files that are in folders of the variation "My folder" such as "path/My folder file1/random.csv" or "path/My folder file 2/another_test.csv" Want to give that a try?

Update:

After learning more from the screenshot and comments, it may be something like this

import pandas as pd
from pathlib import Path
from os.path import join
import glob

# the parent of the `/My folder` stuff
base_path = Path('path/')
glob_pattern = str(base_path.joinpath("My folder*/*.csv"))
all_csv_filenames = glob.glob(glob_pattern)
df = pd.concat(
    {
        (str(Path(filename).parent), str(Path(filename).stem)): pd.read_csv(
            filename
        )
        for filename in all_csv_filenames
    },
    names=['path', 'file', '_'],
)

parent gets your parent directory, stem gets your file name without the extension, and then they get turned into strings and passed as a tuple for the mapping to keys in order for pd.concat to make use of the names parameter.