1

Code

import sys
import os

fp = open("/home/masi/r3.raw", "rb")

try:
    events = []
    while aBuf[:4] != b'\xFA\xFA\xFA\xFA':
        aBuf = fp.read(4)
        events.append(aBuf)         
        if aBuf == os.SEEK_END:
            # pointer cannot be outside of file so minus 144
            fileEnding = aBuf[os.SEEK_END - 144 : os.SEEK_END]
except:
    print "File end at position : ", fp.tell()
    import traceback
    traceback.print_exc()

finally:
    fp.close()

where I know that the following is never true

        if aBuf == os.SEEK_END:
            # pointer cannot be outside of file so minus 144
            fileEnding = aBuf[os.SEEK_END - 144 : os.SEEK_END]

I am comparing the pointer with the end pointer of the file, at least I am expecting so but it does not seem to correct.

Improved Code from skrrgwasme and martineau's contributions

import sys
import os
import struct
import binascii

file_name = "/home/masi/r.raw"
file_size = os.path.getsize(file_name)
print "File size is : ", file_size
read_size = 4
read_count = 0

aBuf = b'\x00\x00\x00\x00' # don't forget to create your variables before you try to read from them
fileEnding = ""
fp = open(file_name, "rb")

try:
    aBuf = fp.read(read_size)
    read_count += read_size
    event_starts = []
    event_ends = []
    event_starts.append(read_count)
    while aBuf and read_count < file_size:
        if aBuf[:read_size] == b'\xFA\xFA\xFA\xFA':
            event_ends.append(read_count)
            if read_count + 1 < file_size: event_starts.append(read_count + 1)

        aBuf = fp.read(read_size)
        read_count += read_size
        print "RC ", read_count, ", remaining: ", 1.0-float(read_count)/file_size, "%"

    if read_count >= file_size: break

except:
    print "File end at position : ", fp.tell()
    import traceback
    traceback.print_exc()

finally:
    # store to partial index of postgres database: event pointers
    fp.close()

How can you compare location of two pointers?

Léo Léopold Hertz 준영
  • 134,464
  • 179
  • 445
  • 697
  • 1
    The Python documentation for [`os.SEEK_END`](https://docs.python.org/2/library/os.html?highlight=os.seek_end#os.SEEK_END) says it's a predefined constant value that can be passed as a parameter to the [`os.lseek()`](https://docs.python.org/2/library/os.html?highlight=os.seek_end#os.lseek) function.If you want to know the current position of an open file use [`file.tell()`](https://docs.python.org/2/library/stdtypes.html?highlight=tell#file.tell). The length of a file can be obtained with [`os.path.getsize()`](https://docs.python.org/2/library/os.path.html?highlight=getsize#os.path.getsize). – martineau Jul 09 '15 at 23:36

1 Answers1

2

If you take a look at the Python source code for the os module, you'll see that os.SEEK_END isn't automatically set to the size of your file. It's just a constant that is set equal to the integer 2. It is intended to be used as a parameter for the lseek() function.

You need to get the file size in bytes first, then compare your file pointer to that. You can use os.path.getsize(path) to get your file size in bytes. Your comparison was never true because you were reading four bytes at a time, so your file pointer skipped from byte 0 to byte 4, passing over 2, which is the value of os.SEEK_END.

Suggested code:

import sys
import os

file_name = "/home/masi/r3.raw"
file_size = os.path.getsize(file_name)
read_size = 4
read_count = 0
# you could use fp.tell() in the loop instead of manually incrementing
# your own count of the file position instead, but this will avoid a lot of
# extra fp.tell() calls in the loop

aBuf = b'\x00\x00\x00\x00' # don't forget to create your variables before you try to
                           # read from them
fp = open(file_name, "rb")

try:
    events = []
    while aBuf[:read_size] != b'\xFA\xFA\xFA\xFA':
        aBuf = fp.read(read_size)
        events.append(aBuf)
        read_count += read_size         
        if read_count >= file_size:
            # pointer cannot be outside of file so minus 144
            fileEnding = aBuf[file_size - 144 : file_size]
            break
except:
    print "File end at position : ", fp.tell()
    import traceback
    traceback.print_exc()

finally:
    fp.close()

Notes:

  1. Instead of comparing for exactly the file size you expect, I suggest using a greater than or equal comparison (>=). Since you're reading four bytes at a time, if you have an odd file size, your comparison will never be true.

  2. After you get this code working, I'd suggest taking it over to Code Review Stack Exchange. As martineau has helpfully pointed out in the comments, there are a number of issues and potential pitfalls in your code that are worth correcting.

Community
  • 1
  • 1
skrrgwasme
  • 9,358
  • 11
  • 54
  • 84
  • 1
    `if aBuf >= file_size:`? How can you compare a 4 byte string to an integer? – martineau Jul 09 '15 at 17:21
  • @martineau Good catch. I forgot to add the manual bytes read count. – skrrgwasme Jul 09 '15 at 17:24
  • 1
    Makes better sense now. Suggest you also see the part about catching _any_ exception in the accepted anwser to [_Why is “except: pass” a bad programming practice?_](http://stackoverflow.com/questions/21553327/why-is-except-pass-a-bad-programming-practice). – martineau Jul 09 '15 at 18:29
  • @martineau I strongly agree that it's bad practice. I just copied the OP's code and made my changes to it. I suppose it would be good to suggest that, though. – skrrgwasme Jul 09 '15 at 18:32
  • 1
    There's still a number of issues. Here's a few. The empty `except` is potentially going to hide the fact that `aBuf = fp.read(readsize)` references an undefined variable. Even if you fix that, the `while` loop will never end if the file doesn't contain a `b'\xFA\xFA\xFA\xFA'` on a four-byte boundary before the end of the file is encountered. If the `fileEnding = aBuf[file_size - 144 : file_size]` ever executes, it will likely set `fileEnding` to `''` because `Abuf` will never have more than 4 bytes in it. Lastly, your code also assumes the `fp.read()` was always able read 4 bytes. – martineau Jul 09 '15 at 22:12
  • @martineau I agree with all of your points, but I don't really feel the need to correct all of them for this answer. I've edited it to suggest he go over to Code Review to get help cleaning it up. – skrrgwasme Jul 09 '15 at 22:17
  • Yes, the code was horrible. I changed the main loop to be like reading a book until the `read_count` reaches `file_size`. Do I skip any more critical exceptions? – Léo Léopold Hertz 준영 Jul 10 '15 at 11:29