Install a python package in Grumpy

Question

I am using Fedora 25 and installed Grumpy in the following way:

git clone https://github.com/google/grumpy.git  
cd grumpy  
make  
export GOPATH=$PWD/build  
export PYTHONPATH=$PWD/build/lib/python2.7/site-packages 

also was able to create Go binary from a simple python file without any problem:

$tools/grumpc hello.py > hello.go
$ go build -o hello hello.go
$ ./hello
hello, world 

However I do not have any clue about how to install a python package! Simple pip install didn't work out.

I tried virtualenv but looks like that also didn't work. I checked the site-packages dir inside grumpy build which had the following only:

ll build/lib/python2.7/site-packages/
> drwxr-xr-x. 4 root root 4096 Jul  4 12:13 grumpy

How should a python package be install in Grumpy then?

Answer 1

Well... unlike what the readme says, there's a good amount of the standard library missing. So more than likely you cannot.

Here's my process for trying to turn pygments into an executable with go. It failed by the way. It is basically a manual process to take a package and make it build with go using Grumpy.

I went to pypi and got the Pygments tar.

I followed the readme's:

make
export PATH=$PWD/build/bin:$PATH
export GOPATH=$PWD/build
export PYTHONPATH=$PWD/build/lib/python2.7/site-packages

I tried to run something like:

echo 'import sys; print sys.version' | grumprun

Which was working.

I untarred the pypi file, I read the setup.py etc and found I was primarily interested in the pygments.cmdline. So I copied the pygments directory into $GOPATH/src/__python__/.

Then I tried to build the cmdline as a module so that I could make it an executable by wrapping it with go the same way it'd been wrapped in a bash script before.

grumpc -modname=pygments.cmdline \
  $GOPATH/src/__python__/pygments/cmdline.py \
> $GOPATH/src/__python__/pygments/cmdline/module.go

This did not work out due to locale missing.

I changed into $GOPATH/src/__python__/pygments and looked for files without imports:

cd $GOPATH/src/__python__/pygments
grep -cr '^ *\(from\|import\)' . >imports

I identified the filter module had none so:

grumpc -modname=pygments.filter filter.py >filter/module.go

Since this worked, I thought I was on to something, but I wasn't. Leading me to think if I just built things with few dependencies first, I could build up to the whole package (note locale isn't part of the package, so this was a bad plan):

# Oh, this doesn't really handle the base __init__.py file well.
for f in *.py */*.py; do F="${f%%.py}";
if [ "${f: -11}" != "__init__.py" ];
then
  mkdir -p "$F";
  [ ! -s $F/module.go ] && grumpc -modname=pygments.${F/\//.} $f > $F/module.go;
else
  F="${F%%__init__}";
  [ ! -s ${F}module.go ] && grumpc -modname=pygments.${F/\//.} $f > ./$F/module.go;
fi
done

Here's a short list of the initial modules I would have needed to write a native implementation thereof.

colorama.initialise
codecs
chardet
pkg_resources
docutils
unicodedata
locale
ctags
subprocess
gzip
urllib
shutil
gzip

I also got several other errors:

<unknown>:118:2235-118:2245: error: unicode character out of range
line 101: node not yet implemented: Exec
line 12: future feature division not yet implemented by grumpy
RuntimeError: maximum recursion depth exceeded while calling a Python object

To be a little fair, there were a number of files that seemed to present no trouble (the ones that have a size) but there were 20 that did have issues, and they seem to include the cmdline, the core formatter, lexer, plugin, and main utils (the terminal, terminal256, bbcode, irc, mapping, rtf and svg formatters seemed okay…):

$ wc `find . -name module.go`
    2023     7797    63728 ./token/module.go
       0        0        0 ./unistring/module.go
    4510    16633   149451 ./filters/module.go
    2221     7035    94627 ./lexers/idl/module.go
       0        0        0 ./lexers/_lua_builtins/module.go
     493     1941    18572 ./lexers/capnproto/module.go
     788     3165    28867 ./lexers/hexdump/module.go
    6437    22288   242059 ./lexers/pascal/module.go
     572     2208    21237 ./lexers/inferno/module.go
    1460     5853    60953 ./lexers/prolog/module.go
    1610     5918    62664 ./lexers/esoteric/module.go
     731     2700    27881 ./lexers/supercollider/module.go
    5950    23129   218602 ./lexers/parsers/module.go
    2579     7756   104501 ./lexers/_csound_builtins/module.go
   14141    49681   644896 ./lexers/lisp/module.go
    6336    23011   236369 ./lexers/modula2/module.go
    5287    19964   200365 ./lexers/asm/module.go
   10004    37737   340097 ./lexers/robotframework/module.go
       0        0        0 ./lexers/_sourcemod_builtins/module.go
    1621     6234    57562 ./lexers/textedit/module.go
     644     2399    24139 ./lexers/go/module.go
     350     1371    13898 ./lexers/apl/module.go
    1325     4984    46972 ./lexers/nix/module.go
    9008    34169   375371 ./lexers/scripting/module.go
    1684     6706    66923 ./lexers/archetype/module.go
     432     1690    16511 ./lexers/parasail/module.go
    5941    18213   301062 ./lexers/ncl/module.go
    4338    15189   259395 ./lexers/_cocoa_builtins/module.go
    2433     9545    90020 ./lexers/textfmts/module.go
    8056    29732   307645 ./lexers/python/module.go
    5293    19344   210921 ./lexers/css/module.go
    4641    16372   178991 ./lexers/c_like/module.go
    1731     6533    67410 ./lexers/felix/module.go
       0        0        0 ./lexers/_php_builtins/module.go
    1966     7720    72987 ./lexers/fantom/module.go
     311     1197    11568 ./lexers/trafficscript/module.go
     595     2238    22795 ./lexers/chapel/module.go
    7294    28597   273136 ./lexers/shell/module.go
     285     1138    10140 ./lexers/web/module.go
    1147     4478    43247 ./lexers/grammar_notation/module.go
    4999    19994   205998 ./lexers/haxe/module.go
    3382    12053   137873 ./lexers/theorem/module.go
     818     3115    30130 ./lexers/dalvik/module.go
    6599    24623   259510 ./lexers/perl/module.go
    1415     5699    54536 ./lexers/smalltalk/module.go
    2286     8370    84331 ./lexers/dylan/module.go
    1524     5900    59673 ./lexers/pawn/module.go
    6150    20492   311029 ./lexers/_vim_builtins/module.go
    1325     5164    52099 ./lexers/rust/module.go
     556     2137    20104 ./lexers/graph/module.go
     524     2098    19189 ./lexers/other/module.go
    2988    10857   123993 ./lexers/business/module.go
    5703    21448   206789 ./lexers/erlang/module.go
    2957     9869   126529 ./lexers/r/module.go
    3273     9848   139052 ./lexers/_asy_builtins/module.go
    4162    16299   158236 ./lexers/basic/module.go
    9823    38671   401800 ./lexers/jvm/module.go
     552     2175    21298 ./lexers/ambient/module.go
    1169     4612    42833 ./lexers/diff/module.go
    4003    14888   160168 ./lexers/objective/module.go
     638     2463    24234 ./lexers/elm/module.go
    1764     5371    71961 ./lexers/_tsql_builtins/module.go
    4680    18630   176165 ./lexers/html/module.go
     397     1581    14231 ./lexers/compiled/module.go
    3834    14988   143117 ./lexers/crystal/module.go
    1218     4584    44945 ./lexers/nimrod/module.go
    2411     7267   117452 ./lexers/_mql_builtins/module.go
    6289    24800   238668 ./lexers/haskell/module.go
    2682     9870   111225 ./lexers/factor/module.go
    2186     8504    81021 ./lexers/php/module.go
    1403     5406    53632 ./lexers/varnish/module.go
     364     1432    13878 ./lexers/rnc/module.go
     612     2355    23021 ./lexers/ooc/module.go
     188      740     6580 ./lexers/math/module.go
    2235     8347    82881 ./lexers/modeling/module.go
    2151     7910    87783 ./lexers/praat/module.go
       0        0        0 ./lexers/special/module.go
     251      993     8934 ./lexers/agile/module.go
    5161    19940   196827 ./lexers/dotnet/module.go
    1133     4277    43582 ./lexers/monte/module.go
    1324     5218    50039 ./lexers/typoscript/module.go
    1202     5339    60462 ./lexers/testing/module.go
     740     2725    27378 ./lexers/oberon/module.go
     796     2951    29702 ./lexers/verification/module.go
    3054    12135   114894 ./lexers/clean/module.go
    1171     4376    44599 ./lexers/tcl/module.go
    5643    21530   202343 ./lexers/markup/module.go
     865     3144    32057 ./lexers/ampl/module.go
    1289     5014    48411 ./lexers/bibtex/module.go
     989     3742    37888 ./lexers/qvt/module.go
   11296    44649   438089 ./lexers/int_fiction/module.go
     355     1392    13373 ./lexers/iolang/module.go
    4353    17287   170343 ./lexers/ml/module.go
    4231    12712   172022 ./lexers/_stata_builtins/module.go
     632     2272    23038 ./lexers/smv/module.go
    1393     4261    63346 ./lexers/_cl_builtins/module.go
    1545     5885    55706 ./lexers/make/module.go
    4918    16606   192919 ./lexers/matlab/module.go
    1645     5625    63597 ./lexers/sas/module.go
       0        0        0 ./lexers/_scilab_builtins/module.go
    5438    20671   209628 ./lexers/configs/module.go
     532     1948    19215 ./lexers/x10/module.go
    3570    13980   137719 ./lexers/rebol/module.go
    1862     6405    71722 ./lexers/fortran/module.go
   19878    76674   719037 ./lexers/templates/module.go
    3279     9851   165624 ./lexers/_openedge_builtins/module.go
    1069     3241    48798 ./lexers/_stan_builtins/module.go
     492     1819    18748 ./lexers/nit/module.go
     406     1591    15567 ./lexers/snobol/module.go
    2537     9157    99186 ./lexers/julia/module.go
    9952    29904   499664 ./lexers/_lasso_builtins/module.go
    2613     8174   119072 ./lexers/igor/module.go
    1190     4456    45298 ./lexers/urbi/module.go
    3312    12007   129704 ./lexers/hdl/module.go
     421     1666    34239 ./lexers/foxpro/module.go
    1227     5383    58804 ./lexers/automation/module.go
       0        0        0 ./lexers/_postgres_builtins/module.go
    6791    25704   260009 ./lexers/dsls/module.go
     291     1156    10516 ./lexers/text/module.go
    9626    38665   383525 ./lexers/javascript/module.go
    2551    10063    99066 ./lexers/csound/module.go
     483     1819    17444 ./lexers/roboconf/module.go
     577     2206    24072 ./lexers/ezhil/module.go
     771     3078    28434 ./lexers/resource/module.go
     731     2804    27243 ./lexers/whiley/module.go
       0        0        0 ./lexers/graphics/module.go
     521     1925    19467 ./lexers/eiffel/module.go
    4179    16387   156894 ./lexers/data/module.go
     856     3182    32625 ./lexers/j/module.go
     546     2165    22731 ./lexers/forth/module.go
   10209    41331   393029 ./lexers/webmisc/module.go
    1393     5279    56499 ./lexers/d/module.go
    2480     9288    91522 ./lexers/c_cpp/module.go
    1578     6503    62492 ./lexers/rdf/module.go
    5282    20171   299945 ./lexers/_mapping/module.go
     186      734     6614 ./lexers/functional/module.go
    1128     3921    43138 ./lexers/ecl/module.go
    5148    19594   192146 ./lexers/ruby/module.go
    2040     7593    81180 ./lexers/actionscript/module.go
    1974     7796    78115 ./lexers/installers/module.go
    1226     4725    48193 ./lexers/algebra/module.go
     590     2309    23425 ./lexers/console/module.go
       0        0        0 ./lexers/module.go
     639     2525    24688 ./lexers/stata/module.go
    7116    25919   255452 ./lexers/sql/module.go
       0        0        0 ./sphinxext/module.go
       0        0        0 ./util/module.go
    1482     5456    46562 ./regexopt/module.go
       0        0        0 ./lexer/module.go
       0        0        0 ./plugin/module.go
    1580     5904    52942 ./formatters/svg/module.go
    4294    16118   142678 ./formatters/terminal256/module.go
    1641     5938    53535 ./formatters/rtf/module.go
       0        0        0 ./formatters/latex/module.go
    2149     8026    70726 ./formatters/irc/module.go
       0        0        0 ./formatters/other/module.go
    1561     5851    52413 ./formatters/terminal/module.go
       0        0        0 ./formatters/html/module.go
       0        0        0 ./formatters/img/module.go
    1064     4241    38824 ./formatters/_mapping/module.go
    1091     3974    35174 ./formatters/bbcode/module.go
       0        0        0 ./formatters/module.go
       0        0        0 ./cmdline/module.go
    2164     7934    70484 ./style/module.go
     339     1364    11693 ./styles/xcode/module.go
     471     1922    16586 ./styles/vim/module.go
     290     1151     9947 ./styles/vs/module.go
     582     2426    20747 ./styles/pastie/module.go
     362     1458    12394 ./styles/bw/module.go
     493     2029    17533 ./styles/native/module.go
    1076     4521    37325 ./styles/paraiso_light/module.go
     566     2347    20186 ./styles/friendly/module.go
     483     1988    17060 ./styles/autumn/module.go
     647     2714    23016 ./styles/murphy/module.go
     330     1331    11549 ./styles/fruity/module.go
     383     1556    13367 ./styles/borland/module.go
     305     1237    10664 ./styles/algol_nu/module.go
     469     1928    16516 ./styles/trac/module.go
     556     2302    19786 ./styles/default/module.go
     887     3608    30851 ./styles/rainbow_dash/module.go
     224      875     7492 ./styles/abap/module.go
     855     3618    30861 ./styles/tango/module.go
     567     2347    20158 ./styles/emacs/module.go
     804     3371    27927 ./styles/arduino/module.go
     508     2093    18035 ./styles/perldoc/module.go
     559     2321    19905 ./styles/manni/module.go
    1076     4521    37315 ./styles/paraiso_dark/module.go
     304     1216    10415 ./styles/sas/module.go
     219      851     7439 ./styles/rrt/module.go
     647     2714    23036 ./styles/colorful/module.go
     194      747     6446 ./styles/igor/module.go
     871     3651    30445 ./styles/monokai/module.go
     818     3418    28958 ./styles/lovelace/module.go
     305     1238    10656 ./styles/algol/module.go
     702     2507    22631 ./styles/module.go
     263     1042     8927 ./styles/stata/module.go
       0        0        0 ./formatter/module.go
     785     2943    25598 ./filter/module.go
     461     1656    13491 ./modeline/module.go
    1023     3955    33738 ./scanner/module.go
     962     3670    30137 ./console/module.go
     761     2770    23417 ./module.go
  440346  1641915 17381149 total

I moved on to making util.py not complain by removing the uses of locale and io as they were in try blocks or python 3 checks and had other paths to take.

Then I got colorama from the pypi and moved the colorama dir from the tar into the __python__ directory.

This needed me to remove some uses of ctypes and basically gut the win32.py file. But it did respond well after that to:

for f in $GOPATH/build/src/__python__/colorama/*.py;
do
  F="${f%%.py}";
  if [ "${f: -11}" != "__init__.py" ];
  then
    mkdir -p "$F";
    [ ! -s $F/module.go ] && grumpc -modname=${F/\//.} $f > $F/module.go;
  else
    F="${F%%/__init__}";
    [ ! -s $F/module.go ] && grumpc -modname=${F/\//.} $f > $F/module.go;
   fi
done

Now I thought I was at the least set to run a demo in the tar file's demo directory.

$ cat demo01.py |grumprun
…
grumpy.compiler.util.ImportError: line 12: no such module: fixpath
# removed the import of fixpath
$ cat demo01.py |grumprun
        Traceback (most recent call last):
  File "/var/folders/gd/q65byyhs54g3qknbyv5r7q8r0000gp/T/tmpVK_oyV/src/__python__/module.py", line 33, in <module>
    sys.stdout.write('%s%-7s' % (foreground, NAMES[foreground]))
NotImplementedError: conversion flags not yet supported
exit status 1

Okay great, so even when something is in the standard library, parts of it are not implemented. This makes sense. But now I'm too frustrated to continue.

Okay well, demos 2, 3, 4, 6, 7, and 8 seemed to work. Demo 5 failed on an import collision, probably because I'm on MacOS and it might work on linux. But all the demos showed that the removal of atexit (and putting the reset_all into deinit) was affecting the behavior such that it does not set colors back to the original state.