2

Wondering if python is able to institute checks within their compile-phase of a given py to pyc to guard against a corrupt pyc due to sudden power-down of the system(and disk). When the system comes back up will the pyc that may exist be checked for integrity and if its considered suspect, regenerated?

user2471686
  • 21
  • 1

2 Answers2

0

As far as I know, the compile-to-bytecode process works the same as normal make in how it handles recompilation; it checks if the compiled files are older than the source files, and if they are it recompiles while if they aren't it leaves them be. My best suggestion would be to just clear the PYC files whenever sudden power failure is suspected and do an import to get the new ones (you could probably automate this, too, to make it simpler).

Note that I have not experimented to see if Python waits until a bytecode file is complete before writing to disk; if it does, that would reduce the possibility of the sort of corruption you talk about, as the PYC file would simply not be written to disk to begin with if the power failure occurred during actual compilation. However, corruption would still be an issue if the power loss occurs during the write, as I believe most OSes update a file's modification time when the file is opened with write access rather than when the file handle is closed.

JAB
  • 20,783
  • 6
  • 71
  • 80
  • Thanks @JAB; working with remote field systems that can have sudden power loss and i see corrupted pyc; there is also a bug thread i just saw (http://bugs.python.org/issue13146); regardless, controlling pyc generation explicitly/completely at system start sounds needed though annoying and will look to automate -- thanks – user2471686 Jun 10 '13 at 17:35
  • Apparently people also have similar issues in cases where they're updating Python modules in a non-root environment that were previously compiled by root/someone with higher access and new PYC files don't overwrite the existing ones due to access restrictions, so much so that in Python 2.6+ it is actually possible to disable the generation of PYC files, if you're okay with the performance decrease. http://stackoverflow.com/a/154617/138772 – JAB Jun 10 '13 at 17:46
0

If I interpret this message correctly, Python 3.3 will create .pyc file under a different name and rename it on successful completion. I think that's as "atomic" (their term) as it gets in this context, and it will certainly protect you from a sudden crash or loss of power. Earlier versions (including the 2.* branch) are apparently still subject to race conditions and invalid .pyc files.

Whether a file could be invalid without triggering a runtime exception is a separate question, though. The problem reports all talk about failing on import.

alexis
  • 48,685
  • 16
  • 101
  • 161
  • 1
    interesting on py3.3; thanks alexis; my pyc file corruption triggered fail on import cause the pyc was essentially a null file (similar pyc file size as if the py had been empty which it wasnt) – user2471686 Jun 11 '13 at 00:00
  • I'm not sure how you're measuring pyc size-- if the filesystem says it's empty, it's empty; but anyway sounds like your problem fits the profile. Most reports are about corrupt files being caused by race conditions, such as two python processes writing the same pyc file at the same time. – alexis Jun 11 '13 at 19:52