How to fix "Tail: Write Error: Broken Pipe" in running linux command in Python?

Question

I am trying to print line-by-line per file that is inside a list. At the end of each line of the file, it needs to check if the term ".sh" is in it or not.

I am getting the error

"Tail: Write error: "Broken Pipe"

Expected result:

Read each from list
Check each line of the file if the term ".sh" comes in it at the end of the line of the file.
Prints if it finds the ".sh"

This is what I have atm:

# Modules
import os
from pprint import pprint

# Files in list
dirlist = ['test.txt','test2.txt','test3.txt']

# Loop to read the file in list
for x in range (len(dirlist)):
    print ("Output of Filename: " + dirlist[x]

# Variable to save the last 3 characters of the line
last3 = os.popen ("cat " + dirlist[x] + " | tail -c 3")
print last3

# Read file
f = open(dirlist[x], "r")
# Loop to check if the keyword is the same as last3
for l in f:
    if last3 in l:
        print ("FOUND IT!")

    else:
        print ("NOT IN IT!")

Outcome:

@Nic

[![enter image description here][3]][3]

Why are you using `cat` and `tail` at all, instead of just opening the file and reading the last three lines? — Daniel Pryden, May 05 '19 at 10:54
So why are you using a Linux solution to a Python problem? Reading lines from a file is something you can do entirely in Python code, there's no need to run any other command. — Daniel Pryden, May 05 '19 at 11:01
The problem is that I have to do this in Python besides Linux. This is just a preparation for the university that I want to attend. — Xyltic, May 05 '19 at 11:03

Nic Wanavit · Answer 1 · 2019-05-05T11:33:25.880

2

I suggest that you use with environment with native python code instead of open and os.popen

Here is an example

# Files in list
dirlist = ['test.txt','test2.txt','test3.txt']

# Loop to read the file in list
for x in dirlist:
    print ("Output of Filename: " + x)
    with open(x) as f
        lines=f.readlines()
        for line in lines: #here you print each line
            print (line)

        if '.sh' in lines[-1:]: #check if .sh is in the last line
            print("found it")
        else:
            print("didnt find it")

edited May 05 '19 at 11:33

answered May 05 '19 at 11:13

Nic Wanavit

2,363
5
19
31

Ah I see. Your code is much easier comparing to mine, but it gives an I/O operation error on the [-1:] line. – Xyltic May 05 '19 at 11:19
Which python version are you using? – Nic Wanavit May 05 '19 at 11:23
At this moment 2.7.16 (but I sudo-apt installed the version 3.6). Can you explain me what the [-1:] does? – Xyltic May 05 '19 at 11:25
1

@Nic: Don't call `readlines()` twice, just call it once. Once you've read all the lines in the file there won't be anything more to read the next time around. (Also, if the file is large, you might want to use `itertools.islice` on the file object itself rather than using `readlines()` to read all the lines into memory. But that probably won't be a problem here.) – Daniel Pryden May 05 '19 at 11:31
@DanielPryden you are right. I have modified the answer accordingly. – Nic Wanavit May 05 '19 at 11:34
@MertCakir [-1:] calls the last item in the array. btw you definitely should try to switch to python 3. you can call it by running "python3 scripts.py" – Nic Wanavit May 05 '19 at 11:35
Yeah I did that @NicWanavit but that didn't work quite well. Here is my result: – Xyltic May 05 '19 at 11:36
@MertCakir the error was because i ran readlines twice. I have edited the answer, please have another go. – Nic Wanavit May 05 '19 at 11:37
I'll try it out. – Xyltic May 05 '19 at 11:40
It looks good! However, it does find the match with the '.sh' while it is inside the line. – Xyltic May 05 '19 at 11:47

score 1 · Answer 2 · answered May 05 '19 at 10:51

1

os.popen returns a file object, not a string.

See: Assign output of os.system to a variable and prevent it from being displayed on the screen

answered May 05 '19 at 10:51

cubuspl42

7,833
4
41
65

score 1 · Answer 3 · answered May 05 '19 at 11:00

tail (actually stdio) gives the "Broken Pipe" error when it tries to write output but there's nobody around to read it. (More specifically, when it receives SIGPIPE.)

If you're going to launch a child process with popen, you need to finish reading from the pipe before your program exits.

In your case, you should probably use subprocess.run rather than a bare os.popen.

Or, better yet, don't use a subprocess for simple file operations! Just do them in native Python code, and it will be much simpler.

score -1 · Answer 4 · answered May 05 '19 at 12:00

With the help Of @Nic Wanavit and Daniel Pyrden, I finally fixed it.

I've put the if/else inside the loop, otherwise it would check all the lines for the .sh instead of per line.

and I've put parenthesis inside the ".sh" section and that worked!

However, I did not do it in the last 3 characters, because the -1: wasn't working for me for some reason.

# Files in List
dirlist = ['test.txt', 'test2.txt', 'test3.txt']

# Loop to read the file in list
for x in dirlist:
 print ("Output of filename: "+  x)

 with open(x) as f:
  lines = f.readlines()
  for line lines:
   print ("Line in file: " + line)

   if (".sh" in line):
    print ("FOUND IT")

   else:
    print ("not found it \n")

How to fix "Tail: Write Error: Broken Pipe" in running linux command in Python?

@Nic

4 Answers4

Result