How to get filename from stdin

Question

I am writing a script and i am running it from the console like this

cat source_text/* | ./mapper.py

and i would like to get the filename of each file reading at the time. Source texts folder contains a bunch of text files whose filename i need to extract as well in my mapper script.

Is that possible?

import sys
import re
import os


# re is for regular expressions
pattern = re.compile("[a-zA-Z][a-zA-Z0-9]*",
                     re.MULTILINE | re.DOTALL | re.IGNORECASE)


# Read pairs as lines of input from STDIN
for line in sys.stdin:
    ....

I don't think you can get the filename from a pipe. The only thing python sees is the stdin from the stdout of the `cat` command. — OneCricketeer, Mar 23 '16 at 15:24
That is not possible. You should modify your program to read directly from the files. — hochl, Mar 23 '16 at 15:25
Please refer to http://stackoverflow.com/questions/3207219/how-to-list-all-files-of-a-directory-in-python — OneCricketeer, Mar 23 '16 at 15:25
if you want the shell to expand the glob then just `./mapper.py source_text/*` and use `sys.argv` to iterate over the files opening them... — AChampion, Mar 23 '16 at 15:27
Thanks -- and don't forget to accept your preferred solution (whichever you like better). — hochl, Mar 23 '16 at 15:42

score 2 · Accepted Answer · answered Mar 23 '16 at 15:35

You cannot do that directly, but fileinput module can help you.

You just have to call you script that way:

./mapper.py source_text/*

And change it that way:

import fileinput
...

# Read pairs as lines of input from STDIN
for line in fileinput.input():
    ...

Then the name of the file being processed is available as fileinput.filename(), and you can also have access the the number of the line in current file as fileinput.filelineno() and still other goodies...

hochl · Answer 2 · 2016-03-23T15:34:38.127

1

That is not possible. You can modify your program to read directly from the files like this:

import sys
import re

# re is for regular expressions
pattern = re.compile("[a-zA-Z][a-zA-Z0-9]*",
                     re.MULTILINE | re.DOTALL | re.IGNORECASE)
for filename in sys.argv[1:]:
    with open(filename, "rU") as f:
        for line in f.readlines():
            if pattern.search(line) is not None:
                print filename, line,

Then you can call it with:

$ ./grep_files.py source_text/*

edited Mar 23 '16 at 15:34

answered Mar 23 '16 at 15:27

hochl

12,524
10
53
87

right, and how do i extract the filename to print it lets say in console? – Petros Kyriakou Mar 23 '16 at 15:31
@PetrosKyriakou: The filename is in the `filename` variable. – jwodder Mar 23 '16 at 15:32
ah yes got it how it works now, i am like brand new to python thank you! – Petros Kyriakou Mar 23 '16 at 15:34
I added some output :) – hochl Mar 23 '16 at 15:34

score 1 · Answer 3 · answered Mar 23 '16 at 15:47

If you use this instead of cat:

grep -r '' source_text/ | ./mapper.py

The input for mapper.py will be like:

source_text/answers.txt:42
source_text/answers.txt:42
source_text/file1.txt:Hello world

You can then retrieve the filename using:

for line in sys.stdin:
    filename, line = line.split(':', 1)
    ...

However Python is more than capable to iterate over files in a directory and reading them line-by-line, for example:

for filename in os.listdir(path):
    for line in open(filename):
        ...

How to get filename from stdin

3 Answers3

Linked