18

A while ago I had to upgrade some servers from Python 2.4 to Python 2.5. I found that .pyc files created under Python 2.4 would crash when Python 2.5 tried to run them.

Will this happen again when I upgrade from 2.5 to 2.6?

EDIT: Here is a bit more detail

I have a fileserver that contains the python code. This is accessed by both Ubuntu and Windows servers to run the python code. When they run the code they produce .pyc files on the fileserver.

I found that when I upgraded one of the server machines from Python 2.4 to 2.5 I had problems with .pyc files. I'm now not sure whether it was a machine running 2.5 that tried to run 2.4 bytecode or whether it was a 2.4 machine trying to run 2.5 bytecode, but if I deleted the bytecode all went well until the next bytecode clash.

I upgraded all of the machines to 2.5 and the problem went away.

pwdyson
  • 1,177
  • 7
  • 14

5 Answers5

17

In general, .pyc files are specific to one Python version (although portable across different machine architectures, as long as they're running the same version); the files carry the information about the relevant Python version in their headers -- so, if you leave the corresponding .py files next to the .pyc ones, the .pyc will be rebuilt every time a different Python version is used to import those modules. "Trying to run" wrong-version .pyc files is something I never heard about. What architectures were involved? Were the .py files around as they should be?

Edit: as the OP clarified that the crashes came when he was running both Python 2.4 and Python 2.5 programs on the same .py files (from two different servers, sharing a network drive), the explanation of the crashes becomes easy. The .py files were all the time being recompiled -- by the 2.4 Python when the 2.5 had been the one running them most recently, and vice versa -- and therefore the .pyc files were frantically busy getting rewritten all the time. Proper file locking on network drives (especially but not exclusively across different operating systems) is notoriously hard to achieve. So the following must have happened (the roles could be switched): the 2.4 server had just determined that a .pyc file was fine for it and started reading it; before it could finish reading, the 2.5 server (having previously determined that the module needed to be recompiled) wrote over it; so the 2.4 server ended up with a memory buffer that had (say) the first 4K bytes from the 2.4 version and the next 4K bytes from the 2.5 version. When it then used that mangled buffer, unsurprisingly... crash!!!

This can be a real problem if you ever find yourself continuously trying to run a single set of .py files from two or more different versions of Python (even on the same server, without the added complications of network drives). The "proper" solution would be something like virtualenv. The (simple, but dirty-ish) hack we adopted at work (many years ago, but it's still in production...!) is to patch each version of Python to produce and use a different extension for its compiled bytecode files: .pyc (or .pyo) for Python 1.5.2 (which was the most stable "system" version back when we started doing this kludge to newer versions), .pyc-2.0 for 2.0, .pyc-2.2 for 2.2, and so forth (or equivalent .pyo-X.Y of course). I hear this is soon going away at long last (thanks Thomas!-), but it did tide us semi-decently over this ticklish problem for many, many years.

A much simpler solution is to keep a single version of Python around, if that's feasible for your system; if your system has any complications that make it unfeasible to have a single Python version (as ours did, and does), then these days I'd heartily recommend virtualenv, which I've already mentioned.

With the adoption of PEP 3147 in Python 3.2, pyc files for different Python versions are distinguished automatically by filename. This should solve most problems with different Python versions overwriting each other's pyc files.

user2357112
  • 260,549
  • 28
  • 431
  • 505
Alex Martelli
  • 854,459
  • 170
  • 1,222
  • 1,395
  • Sometimes people try distributing .pyc files without .py files as a form of IP protection. – John La Rooy Feb 15 '10 at 00:04
  • @gnibbler: Yes, but that is stupid, since .pyc files are very readable and easily decompilable to a very close representation – nosklo Feb 15 '10 at 00:47
  • Thanks for that. I'm surprised by the file locking error. That means I can't upgrade from python 2.5 to 2.6 one server at a time, but have to bring them all down and upgrade them all before bringing them back up again. – pwdyson Feb 15 '10 at 04:40
  • @pwdyson right, you can't insert a skew in Python versions trying to import the _same_ `.py` modules at the same time -- unless you isolate them from each other, e.g. via `virtualenv`, duplicating the directories containing `.py` (and `.pyc`) files, and the like. – Alex Martelli Feb 15 '10 at 06:00
6

If you have the source code then it will recompile it for you. So in general you are okay.

But, this could be bad for you if users with difference versions of Python run from a central installation directory.

It could also be bad if you just have the pyc files. I just ran a quick test for you. I created two .pyc files. One in 2.5 and one in 2.6. The 2.5 won't run in 2.6 and the 2.6 won't run in 2.5. Both throw "ImportError: Bad magic number in .." error, which makes sense because the magic number has changed from 2.5 to 2.6.

If you want to determine this ahead of time you can get the magic number of your Python as follows:

$ python -V
Python 2.6.2
# python
>>> import imp
>>> imp.get_magic().encode('hex')
'd1f20d0a'

To get the magic number for a pyc file you can do the following:

>>> f = open('test25.pyc')
>>> magic = f.read(4)
>>> magic.encode('hex')
'b3f20d0a'
>>> f = open('test26.pyc')
>>> magic = f.read(4)
>>> magic.encode('hex')
'd1f20d0a'
Paul Hildebrandt
  • 2,724
  • 28
  • 26
  • Could you expand upon this? Because that was the situation: "But, this could be bad for you if users with difference versions of Python run from a central code repository." – pwdyson Feb 15 '10 at 00:26
  • Certainly. If we first assume that a person with Python 2.5 has used the repository the pycs will be 2.5 pycs. When a person with Python 2.6 uses the repository they will convert the pycs of the files they import to 2.6 pycs. They will also take the speed hit for doing it. You may also hit problems if people with different versions try to use the pycs at the same time. Just to be clear this repository of which I speak is where people are running the code from not checking code in and out. – Paul Hildebrandt Feb 15 '10 at 01:16
  • I edited my original answer and changed the term "repository" to "installation dir" to make sure I was clear. – Paul Hildebrandt Feb 15 '10 at 01:27
  • Thanks for pointing out that it would be a problem when different versions of python tried to use them at the same time. I think I can live with that. I remember problems occurring at other times too. Unfortunately this is all from memory now, as I thought I had found the source of the error (different version) and fixed it (made them all the same version). From what everyone is saying it should be safe for me to upgrade one server to 2.6 and leave the rest with 2.5 This allows me to do a gradual upgrade and check things along the way, rather than have to do them all at once. – pwdyson Feb 15 '10 at 01:41
  • 2
    @pwydson, **no** -- it's _not_ safe; read my edited answer on why this mixing of versions makes crashes likely. Use virtualenv (or otherwise segregate the directories where 2.5 and 2.6 pyc files get written), or stick with a single version, *not* multiple ones. – Alex Martelli Feb 15 '10 at 02:43
  • I agree with Alex, it's not safe. When we had this problem we made two install directories. One for each version of Python. This way you don't have the slowdowns but more importantly you avoid the crashes when the two versions access the same files at the same time. – Paul Hildebrandt Feb 15 '10 at 05:16
2

The Python version that creates the file is stored in the .pyc file itself. Usually this means that the .pyc is replaced by one with the correct Python version

some reasons this might not happen
- permissions
- .py file is not available

In the case of permission problem, Python will just use the .py and ignore the .pyc (at a cost to performance)

I think it is ok between minor versions though, eg a Python2.6.2 .pyc should work with Python2.6.4

Here is an excerpt from /usr/share/file/magic

# python:  file(1) magic for python
0   string      """ a python script text executable
0   belong      0x994e0d0a  python 1.5/1.6 byte-compiled
0   belong      0x87c60d0a  python 2.0 byte-compiled
0   belong      0x2aeb0d0a  python 2.1 byte-compiled
0   belong      0x2ded0d0a  python 2.2 byte-compiled
0   belong      0x3bf20d0a  python 2.3 byte-compiled
0   belong      0x6df20d0a  python 2.4 byte-compiled
0   belong      0xb3f20d0a  python 2.5 byte-compiled
0   belong      0xd1f20d0a  python 2.6 byte-compiled

So you can see that the correct Python version is indicated by the first 4 bytes of the .pyc file

John La Rooy
  • 295,403
  • 53
  • 369
  • 502
1

See http://www.python.org/dev/peps/pep-3149/ for a proposed fix for this (hopefully in Python 3.2)

grugnog
  • 41
  • 1
  • 3
  • That's actually a different PEP to solve a very similar but distinct problem; specifically, it's to distinguish *extension modules* compiled for incompatible ABIs. The PEP for pyc files is [PEP 3147](https://www.python.org/dev/peps/pep-3147/). – user2357112 Jul 01 '18 at 00:05
0

You will certainly need to recompile the bytecode files for them to be of any use. The bytecode magic number has changed over every major version of Python(*).

However, having non-version-matching bytecode files shouldn't ever crash Python. It will generally just ignore any bytecode that doesn't have the correct version number, so there shouldn't be an error, it'll just be slower the first time as it recompiles (or slower every time if the user running the scripts doesn't have write permission to update the bytecode).

(*: and often during the development phases, plus in earlier versions it sometimes changed over minor versions too. See import.c for a full list of magic numbers and their corresponding Python versions.)

bobince
  • 528,062
  • 107
  • 651
  • 834