Using PyInstaller with Bokeh and CustomJS

Question

I'm developing a python application which uses Bokeh and PubSub and other python modules. I'm almost done with it and it's running fine. But when I tried to make an executable out of it by using PyInstaller, I ran into a lot of problems. I solved the Jinja2 TemplateNotFound and the unregistered loader type issue by using @dzman's post from this thread. I solved the PubSub import problem by using @Stefano's post from this thread. Now the problem is, when I am trying to run the .exe file it throws error as shown below

Traceback (most recent call last):
File "module1.py", line 191, in <lambda>
File "module1.py", line 274, in getQueryItems
File "module1.py", line 353, in queryTheDatabaseToGetResult
File "module1.py", line 399, in createDataframeFromTheResult
File "module2.py", line 17, in __init__
File "module2.py", line 75, in plotFunction
File "site-packages\bokeh\models\callbacks.py", line 68, in 
from_coffeescript
File "site-packages\bokeh\util\compiler.py", line 190, in nodejs_compile
File "site-packages\bokeh\util\compiler.py", line 169, in _run_nodejs
File "site-packages\bokeh\util\compiler.py", line 164, in _run
RuntimeError: module.js:538
throw err;
^

Error: Cannot find module 'C:\Users\user_name\Desktop\PYTHON~1\dist\PATHWA~1\bokeh\server\static\js\compiler.js'
at Function.Module._resolveFilename (module.js:536:15)
at Function.Module._load (module.js:466:25)
at Function.Module.runMain (module.js:676:10)
at startup (bootstrap_node.js:187:16)
at bootstrap_node.js:608:3

In the code the error occurs right at the very last line of the following code snippet,

    checkbox.callback = CustomJS.from_coffeescript(args = dict(plot = fig, checkbox = checkbox), code=""" 

    rends = plot.select("hideable");
    rends[i].visible = i in checkbox.active for i in [0...rends.length];

    """)    #This is line 68 as shown in the error message

So, it's the CustomJS where the code is failing. I could not find a single post that describes how to use PyInstaller with Custom JS in Bokeh. But I really need to do this as I have to distribute the executable. Any help is highly appreciated and thanks in advance !!

I'm using Pyinstaller 3.3.1, Python 2.7 and Bokeh 0.12.11

neo · Answer 1 · 2018-04-03T21:43:32.183

So, the issue is the program could not find the compiler.js file. I had to made few changes to incorporate this file in the PyInstaller executable. Under ..\bokeh\util the compiler.py file is present. In this file under the nodejs_compile function,

def nodejs_compile(code, lang="javascript", file=None):
    compilejs_script = join(bokehjs_dir, "js", "compiler.js")
    ...
    ...

it is using bokehjs_dir, which is already defined in the file as

bokehjs_dir = settings.bokehjsdir()

bokehjs_dir is a variable which is eventually appended by other variables and makes the path for the program to reach compiler.js file )

so the value of bokehjs_dir is set as an absolute path to the bokeh directory, which caused the problem as the generated executable could not access that.

so just comment out that line and add the code snippet as shown below,

import bokeh

#bokehjs_dir = settings.bokehjsdir()

if getattr(sys, 'frozen', False):
   # we are running in a bundle        
   temp_dir = sys._MEIPASS
else:
   # we are running in a normal Python environment
   temp_dir = os.path.dirname(bokeh.__file__)

bokehjs_dir = temp_dir + '\\server\\static'

As mentioned in the already referenced post, what it does is when the code is frozen, it redirects the program to look into sys._MEIPASS (temporary folder created by PyInstaller to unpack the bundle).

Now all that needs to be done is the inclusion of ..\bokeh\server\static\ folder in the bundle with the same name as static. This can be done by editing the pyinstaller .spec file. The edited .spec file looks like the follwing,

a = Analysis(['module1.py'],
         pathex=['C:\\Users\\user_name\\Desktop\\PythonFiles'],
         binaries=[],
         datas=[(r'C:\Python27\Lib\site-packages\bokeh\core\_templates', '_templates'), (r'C:\Python27\Lib\site-packages\bokeh\server', 'server')],
         hiddenimports=[],
         hookspath=[],
         runtime_hooks=[],
         excludes=[],
         win_no_prefer_redirects=False,
         win_private_assemblies=False,
         cipher=block_cipher)

By editing this we made sure that the required compiler.js got bundled in the PyInstaller executable. And we have already made sure that when the program runs the compiler.py file will find it.

Hopefully this will be helpful to someone in the future if he/she will face the same issue !

In `bokeh==0.13` you "only" need to edit the `.spec` file as suggested. — norok2, Jul 18 '18 at 18:44

score 0 · Answer 2 · answered Apr 02 '18 at 22:35

0

I'm not sure this will be workable or not. Compiling from CoffeeScript requires the coffee compiler, which is a separate external executable, to be installed and runnable by Bokeh at runtime. You might have success rewriting the CustomJS code in pure JavaScript, but even then I think you would need the node executable to be installed and runnable as well. (But I am not 100% certain on that.) If there is some way with pyinstaller to package and bundle these additional programs to that Bokeh can use them, you may be able to make it work.

Edit: I think node is only required for full custom extensions, and not CustomJS, so I'd suggest that converting your coffeescript to pure javascript is definitely worth a shot.

Otherwise, I know the idea of pre-pre-compiling JS code so that it can be bundled statically, without any need for any external runtime executables, was floated at one point in a different GH issue, but I don't think it was followed up with its own issue. Please feel free to submit a GitHub feature request issue, but it might not be able to be addressed for some time.

answered Apr 02 '18 at 22:35

bigreddot

33,642
5
69
122

bigreddot, thanks for your suggestion. The functionalities that I can achieve by using CoffeeScript, I could not do the same by using JavaScript. So discarding CoffeeScript is not an option for me. However, thanks for suggesting the Coffee compiler thing, after so many trials and error I could solve this issue by bundling coffee compiler with an approach similar to what @dzman has mentioned in his post [Link is give in the question] – neo Apr 03 '18 at 20:55
I don't understand this comment: "The functionalities that I can achieve by using CoffeeScript, I could not do the same by using JavaScript. " CoffeeScript is compiled *to JavaScript*, so by definition there is nothing at all the CS can do that JS cannot. If nothing else, you could run the `coffee` compiler yourself, at the command line, on your CS code to get the resulting JS. – bigreddot Apr 03 '18 at 21:11
Well I'm not sue. Maybe it is possible and I just did not know how to do it by using JS. Please look into the following post, this will clarify my issue with JS . https://stackoverflow.com/questions/49203844/using-checkbox-widget-in-bokeh-to-hide-or-show-lines-for-dynamic-number-of-lines – neo Apr 03 '18 at 21:36

score 0 · Answer 3 · edited Jun 07 '18 at 05:00

For me, the following solution worked:

Replace the following lines in bokeh.core.templates (located in "bokeh\core\templates.py"):

from jinja2 import Environment, PackageLoader, Markup

_env = Environment(loader=PackageLoader('bokeh.core', '_templates'))
_env.filters['json'] = lambda obj: Markup(json.dumps(obj))

by:

    import sys
    import os
    directory = os.path.dirname(sys.executable)

    _env = Environment(loader=FileSystemLoader(os.path.join(directory, '_templates')))
    _env.filters['json'] = lambda obj: Markup(json.dumps(obj))

Copy the folder bokeh\core_templates in the output folder, where your frozen .exe application lies.

In summary, the solution to the problem is just to explicitly copy the templates to a location and tell the frozen application where to find these templates.

score 0 · Answer 4 · answered Oct 05 '18 at 04:09

I think this topic can be closed. In Bokeh 0.13 the "core.templates.py" file has been modified and now includes the templates in the correct way when the script is frozen:

def get_env():
    ''' Get the correct Jinja2 Environment, also for frozen scripts.
    '''
    if getattr(sys, 'frozen', False):
        templates_path = os.path.join(sys._MEIPASS, '_templates')
        return Environment(loader=FileSystemLoader(templates_path))
    else:
        return Environment(loader=PackageLoader('bokeh.core', '_templates'))


_env = get_env()
_env.filters['json'] = lambda obj: Markup(json.dumps(obj))

Using PyInstaller with Bokeh and CustomJS

4 Answers4