5

pyc files have been the cause of grief in the past and I have just recently seen a few posts about preventing python from generating them. Currently we just run a script to clean them up after any code changes to ensure fresh ones get generated but it would be much easier to just disable them in general. Is there any side effects to disabling them in our production environment that maybe I am not aware of? What are the downsides of doing this?

The only real issue we experience is occasionally the files get out of date causing import errors and is difficult to debug after a massive refactor. Once realizing it is a pyc problem it is an easy enough fix, just run the script but that realization could come 30 minutes down the road of debugging.

Jared Mackey
  • 3,998
  • 4
  • 31
  • 50
  • Can you describe the issues you're having with them in a bit more detail? There may be a better option that just disabling them. – skrrgwasme Jun 07 '16 at 03:46

2 Answers2

4

Disabling the writing and use of compiled bytecode comes with a performance cost at startup -- your Python code is loaded into memory when the process spawns, and non-.pyc/.pyo files means the interpreter is forced to parse every imported file on startup. If you're currently removing all .pyc and .pyo files from your code directory and your imports, you're forcing a version of this already.

I'm curious as to where they've caused grief in the past (are you disabling modified times on the filesystem?) as if the .py file is newer than the .pyc file, the Python interpreter (at least in common implementations, including CPython), will re-compile the source, but that's not very important for the scope of this question.

If you're importing behind a function, e.g.: 

def my_database_call(budget_id):
    import some_expensive_module_to_import
    some_orm(...).get(budget_id)

that import cost won't be incurred until that function is called (and perhaps, every subsequent time as well). This is not markedly different than if you were to allow bytecode (.pyc or "optimized" .pyo files), but at least in that case, you're paying to that import/parse/compile price only once.

In the case you brought up in the comments, if you're "shelling out" to the OS to call a Python script, and are not allowing bytecode to be written in that call, every call incurs the compilation price.

  • 1
    Sometimes they do not get recompiled after checking out new code with Git. It can sometimes be a major headache debugging that the `pyc` files is what is causing these import errors. Especially after massive refactors. – Jared Mackey Jun 07 '16 at 03:45
  • Ah, okay, I can see that happening. The primary change if you don't allow bytecode files to be written will be increased startup-time. This can be very important if your application throws an exception in production and you are relying on it to restart. –  Jun 07 '16 at 03:48
  • How would this effect using `popen` to run a python script? We do this very frequently. – Jared Mackey Jun 07 '16 at 03:49
  • Same thing -- `Popen(['python', 'my_script.py'])` follows the same rules as the call of `python my_script.py` loads `my_script` and its imports into memory. Without allowing saving bytecode, it has to parse/"compile" every source file on each call -- just longer startup times. –  Jun 07 '16 at 03:52
  • 1
    Ok, so this all could be "slower", are you aware of any documentation on how much slower? I tried googling but came up empty handed. I realize this would vary wildly based on the app and script. – Jared Mackey Jun 07 '16 at 03:53
  • You nailed it. Depends _entirely_ based on the app, script, and number of imports, complexity of the file contents. And to be careful, _slower_ on startup and initial import. If you're importing behind a function, that import cost won't be incurred until that function is called, and I believe every subsequent time as well. –  Jun 07 '16 at 03:56
  • Could you edit your answer to include that last piece? That is really relevant and is probably going to deter us from using it. – Jared Mackey Jun 07 '16 at 03:57
  • Done :) I'm going to head out for the night, but I'd be happy to help you out tomorrow if you still have questions that don't go answered. I hope the answer was helpful and good luck! –  Jun 07 '16 at 04:06
3

You can try adding the following code to your main. Any module imported after will not create a .pyc:

import sys
sys.dont_write_bytecode=True

Or setting the enviroment variable:

PYTHONDONTWRITEBYTECODE

You could also edit or create your usercustomize.py found in /site-packages to contain:

import sys
sys.dont_write_bytecode=True

Can also pass the "-B" option. As far as deleting or disabling .pyc generation, the repercussions seem small or indistinguishable. The .pyc is generated to improve speed. Not the speed of a program but the amount of time it takes to load. If you are constantly importing a module then a .pyc is recommended because it reduces the load time of said modules. Upon researching there seems to be no other benefit of using/generating .pyc's.

TheLazyScripter
  • 2,541
  • 1
  • 10
  • 19