How import multiple text files to one dataframe in python?

Question

I found here how to import multiple text files to one data frame. However, it gives an error. Files are with the names as footballseason1,footballseason2,footballseason3 ... (until footballseason5000)

import pandas as pd
import datetime as dt
import os, glob
os.chdir("~/Downloads/data")
filenames = [i for i in glob.glob("*.txt")]

FileNotFoundError: [Errno 2] No such file or directory: '~/Downloads/data'

However, if I try to import one file, everything is working and the directory is found

df = pd.read_csv("~/Downloads/data/footballseason1.txt", sep=",")

Could you help to fix the problem? and are there any ways to do it without changing directory and simply do all the steps using the path where all files are located?

When you read one file, you are reading a file with extension **txt** and when you read the folder you are reading extension **csv** and maybe you don't have any file — nacho, Apr 28 '21 at 07:47
what are your actual file types and where are the files stored? — imdevskp, Apr 28 '21 at 07:53
all files are of `.txt` format and located in `"~/Downloads/data/"` — Priit Mets, Apr 28 '21 at 08:06

score 2 · Accepted Answer · answered Apr 28 '21 at 08:35

2

Python's os does not understand ~ by default, so it needs to be expanded manually:

filenames = [i for i in glob.glob(os.path.expanduser("~/Downloads/data/*.txt"))]

answered Apr 28 '21 at 08:35

tdy

36,675
19
86
83

imdevskp · Answer 2 · 2021-04-28T08:11:41.940

0

You can use python's list comprehension and pd.concat like below

df = pd.concat([pd.read_csv(i, sep=',') for i in glob.glob("~/Downloads/data/*.txt", recursive=True)])

edited Apr 28 '21 at 08:11

answered Apr 28 '21 at 07:47

imdevskp

2,103
2
9
23

I have an error: `No objects to concatenate` – Priit Mets Apr 28 '21 at 08:07
try by adding recursive=True – imdevskp Apr 28 '21 at 08:13
now it says `name 'true' is not defined`. I also tried in different way to see whether functions are working using code below, but again the error that directory does not exist `import glob, os os.chdir("~/Downloads/data") for file in glob.glob("*/.txt", recursive = true): print(file) ` – Priit Mets Apr 28 '21 at 08:18
what does os.lisdir('~/Downloads/data') gives you – imdevskp Apr 28 '21 at 08:30

score 0 · Answer 3 · answered Apr 28 '21 at 08:35

Via pathlib ->

import pandas as pd
from pathlib import Path
inp_path = Path("~/Downloads/data")
df = pd.concat([
    pd.read_csv(txt_file, sep=',') for txt_file in inp_path.glob('*.txt')
])

With added check - >

import pandas as pd
from pathlib import Path
inp_path = Path("~/Downloads/data")
if inp_path.exists():
    df = pd.concat([
        pd.read_csv(txt_file, sep=',') for txt_file in inp_path.glob('*.txt')
    ])
else:
    print('input dir doesn\'t exist please check path')

score -1 · Answer 4 · answered Apr 28 '21 at 08:32

Importing Data from Multiple files Now let’s see how can we import data from multiple files from a specific directory. There are many ways to do so, but I personally believe this is an easier and simpler way to use and also to understand especially for beginners. 1)First, we are going to import the OS and glob libraries. We need them to navigate through different working directories and getting their paths. import os import glob 2) We also need to import the pandas library as we need to work with data frames. import pandas as pd 3) Let’s change our working directory to the directory where we have all the data files. os.chdir(r"C:\Users\HARISH\Path_for_our_files") 4) Now we need to create a for loop which iterates through all the .csv file in the current working directory filenames = [i for i in glob.glob("*.csv")]

This answer is completely unrelated to the OP's problem. Please read questions carefully, and note that answers on SO are supposed to really answer the question asked. — Thierry Lathuille, May 02 '21 at 16:22

How import multiple text files to one dataframe in python?

4 Answers4