5

I have a strange behaviour for pyunpack, a package for unpacking, inside an executable.

I want to do the following thing:

I have a .7z type of file whose ending is not in .7z but in .sent.

First I try to unzip it the direct way, which leads to an expected error that is caught.

Inside this error catching, I am first adding the .7z extension, then I am unzipping the file properly into a folder called "grog", then I give the zipped file its original name back.

Here is the code below:

# test.py
from os.path import abspath, join, exists, dirname
from os import rename, mkdir
from shutil import copy
import multiprocessing
import pyunpack

multiprocessing.freeze_support()
print(0)
name = "file_to_be_unzipped.sent"
print("a")
path = "C:\\Users\\myname\\eclipse-workspace-tms\\test_unzip_exe"
print(abspath("."))
print("b")
unzip_dest = join(path, "grog")
if not exists(unzip_dest):
    mkdir(unzip_dest)
print("c")
name = join(path, name)
print("d")
print("e")
try:
    print(1)
    pyunpack.Archive(name).extractall(unzip_dest)
    print(2)
except pyunpack.PatoolError as pterr:
    print(3)
    temp_f_name = name + ".7z"
    print(4)
    rename(name, temp_f_name)
    try:
        print(5)
        pyunpack.Archive(temp_f_name).extractall(unzip_dest)
        print(6)
        rename(temp_f_name, name)
        print(7)
    except pyunpack.PatoolError as pterr2:
        # removing useless 7z extension
        print(8)
        rename(temp_f_name, name)
        print(9)
        # Case when the file is already unzipped
        if str(pterr2).find("Is not archive"):
            print(10)
            copy(name, unzip_dest)
            print(11)
        print(12)
except ValueError as v:
    print(13)
    print(v)
    print(14)

When I launch the script test.py, I get the expected behaviour:

0
a
C:\Users\myname\eclipse-workspace-tms\test_unzip_exe
b
c
d
e
1
3
4
5
6
7

then I build the executable with the following command line:

pyinstaller --log-level=DEBUG test.spec

and the following spec file:

# -*- mode: python ; coding: utf-8 -*-

block_cipher = None

import pyunpack
import patoolib
from pyunpack import Archive, PatoolError
from patoolib.programs import ar
from patoolib.programs import  arc
from patoolib.programs import  archmage
from patoolib.programs import  arj
from patoolib.programs import  bsdcpio
from patoolib.programs import  bsdtar
from patoolib.programs import  bzip2
from patoolib.programs import  cabextract
from patoolib.programs import  chmlib
from patoolib.programs import  clzip
from patoolib.programs import  compress
from patoolib.programs import  cpio
from patoolib.programs import  dpkg
from patoolib.programs import  flac
from patoolib.programs import  genisoimage
from patoolib.programs import  gzip
from patoolib.programs import  isoinfo
from patoolib.programs import  lbzip2
from patoolib.programs import  lcab
from patoolib.programs import  lha
from patoolib.programs import  lhasa
from patoolib.programs import  lrzip
from patoolib.programs import  lzip
from patoolib.programs import  lzma
from patoolib.programs import  lzop
from patoolib.programs import  mac
from patoolib.programs import  nomarch
from patoolib.programs import  p7azip
from patoolib.programs import  p7rzip
from patoolib.programs import  p7zip
from patoolib.programs import  pbzip2
from patoolib.programs import  pdlzip
from patoolib.programs import  pigz
from patoolib.programs import  plzip
from patoolib.programs import  py_bz2
from patoolib.programs import  py_echo
from patoolib.programs import  py_gzip
from patoolib.programs import  py_lzma
from patoolib.programs import  py_tarfile
from patoolib.programs import  py_zipfile
from patoolib.programs import  rar
from patoolib.programs import  rpm
from patoolib.programs import  rpm2cpio
from patoolib.programs import  rzip
from patoolib.programs import  shar
from patoolib.programs import  shorten
from patoolib.programs import  star
from patoolib.programs import  tar
from patoolib.programs import  unace
from patoolib.programs import  unadf
from patoolib.programs import  unalz
from patoolib.programs import  uncompress
from patoolib.programs import  unrar
from patoolib.programs import  unshar
from patoolib.programs import  unzip
from patoolib.programs import  xdms
from patoolib.programs import  xz
from patoolib.programs import  zip
from patoolib.programs import  zoo
from patoolib.programs import  zopfli
from patoolib.programs import  zpaq


# from pyunpack import Archive, PatoolError

a = Analysis(['test.py'],
             pathex=['C:\\Users\\myname\\eclipse-workspace-tms\\test_unzip_exe'],
             binaries=[],
             datas=[],
             hiddenimports=['pyunpack', 'patoolib',
                             'patoolib.programs.ar',
                             'patoolib.programs.arc',
                             'patoolib.programs.archmage',
                             'patoolib.programs.arj',
                             'patoolib.programs.bsdcpio',
                             'patoolib.programs.bsdtar',
                             'patoolib.programs.bzip2',
                             'patoolib.programs.cabextract',
                             'patoolib.programs.chmlib',
                             'patoolib.programs.clzip',
                             'patoolib.programs.compress',
                             'patoolib.programs.cpio',
                             'patoolib.programs.dpkg',
                             'patoolib.programs.flac',
                             'patoolib.programs.genisoimage',
                             'patoolib.programs.gzip',
                             'patoolib.programs.isoinfo',
                             'patoolib.programs.lbzip2',
                             'patoolib.programs.lcab',
                             'patoolib.programs.lha',
                             'patoolib.programs.lhasa',
                             'patoolib.programs.lrzip',
                             'patoolib.programs.lzip',
                             'patoolib.programs.lzma',
                             'patoolib.programs.lzop',
                             'patoolib.programs.mac',
                             'patoolib.programs.nomarch',
                             'patoolib.programs.p7azip',
                             'patoolib.programs.p7rzip',
                             'patoolib.programs.p7zip',
                             'patoolib.programs.pbzip2',
                             'patoolib.programs.pdlzip',
                             'patoolib.programs.pigz',
                             'patoolib.programs.plzip',
                             'patoolib.programs.py_bz2',
                             'patoolib.programs.py_echo',
                             'patoolib.programs.py_gzip',
                             'patoolib.programs.py_lzma',
                             'patoolib.programs.py_tarfile',
                             'patoolib.programs.py_zipfile',
                             'patoolib.programs.rar',
                             'patoolib.programs.rpm',
                             'patoolib.programs.rpm2cpio',
                             'patoolib.programs.rzip',
                             'patoolib.programs.shar',
                             'patoolib.programs.shorten',
                             'patoolib.programs.star',
                             'patoolib.programs.tar',
                             'patoolib.programs.unace',
                             'patoolib.programs.unadf',
                             'patoolib.programs.unalz',
                             'patoolib.programs.uncompress',
                             'patoolib.programs.unrar',
                             'patoolib.programs.unshar',
                             'patoolib.programs.unzip',
                             'patoolib.programs.xdms',
                             'patoolib.programs.xz',
                             'patoolib.programs.zip',
                             'patoolib.programs.zoo',
                             'patoolib.programs.zopfli',
                             'patoolib.programs.zpaq'],
             # hiddenimports=['Archive', 'PatoolError'],
             hookspath=[],
             runtime_hooks=[],
             excludes=[],
             win_no_prefer_redirects=False,
             win_private_assemblies=False,
             cipher=block_cipher,
             noarchive=False)
pyz = PYZ(a.pure, a.zipped_data,
             cipher=block_cipher)
exe = EXE(pyz,
          a.scripts,
          [],
          exclude_binaries=True,
          name='test',
          debug=False,
          bootloader_ignore_signals=False,
          strip=False,
          upx=True,
          console=True )
coll = COLLECT(exe,
               a.binaries,
               a.zipfiles,
               a.datas,
               strip=False,
               upx=True,
               upx_exclude=[],
               name='test')

and then after an unexpected long time, I get the following:

0
a
C:\Users\myname\eclipse-workspace-tms\test_unzip_exe\dist\test
b
c
d
e
1
2

where the file in the destination ("grog") is not an unzipped file as wanted but simply a copy.

Does anybody have an idea of what is going wrong?

Thanks a lot

EDIT:

I have made some progress: In the script, after print(1), if I add:

sys.executable = "C:\\Users\\myname\\AppData\\Local\\Continuum\\anaconda3\\python.exe"

Then it works again. However I have a non portable .exe

Since importing only the python.exe file via the spec file is not enough, the other solution would be to add into the spec file

("C:\\Users\\myname\\AppData\\Local\\Continuum\\anaconda3\*,'.'")

which is going to make a huge executable, or pick one by one only the necessary files through trial and error, which takes ages. Any elegant solution is welcome.

EDIT 2: For more insight, the part from the pyunpack where the bug occurs is at the init file located in:

"C:\\Users\\myname\\AppData\\Local\\Continuum\\anaconda3\\Lib\\site-packages\\pyunpack\\__init__.py"

in the function extract_all_patool:

   `p = EasyProcess([
        sys.executable,
        patool_path,
        '--non-interactive',
        'extract',
        self.filename,
        '--outdir=' + directory
        # '--verbose'
    ]).call(timeout=self.timeout) `

the problem being that sys.executable is set to the test.exe file instead of the python.exe executable itself

EDIT3: I found a semi-portable solution, not ideal but I found nothing better yet:

The end-user is expected to install anaconda in the default path, then pip install patool and pip install pyunpack, and then copy the exe file anywhere in his username folder.

On my side: I am adding in the spec file import pathlib, from pathlib import Path In the hidden_mports list of the spec file, I am adding: 'pathlib', 'pathlib.Path',

Then in the code, after print(1), I am adding:

abspath = abspath(".")
user_path = Path(abspath).parts
user_path = join(user_path[0], user_path[1], user_path[2], user_path[3] )
conda_path = join("AppData", "Local", "Continuum", "anaconda3", "python.exe")
sys.executable = join(user_path, conda_path)
Matt Dnv
  • 1,620
  • 13
  • 23
  • @Alejandro Fernandez – Matt Dnv Mar 12 '20 at 14:15
  • A neighbouring question was here: https://stackoverflow.com/questions/57388924/how-to-use-patool-in-a-pyinstaller-exe – Matt Dnv Mar 12 '20 at 14:16
  • @legorooj maybe you can help on this one, I set a bounty – Matt Dnv Mar 14 '20 at 21:37
  • NB: adding `import patoolib` in the code has not changed anything either – Matt Dnv Mar 15 '20 at 08:39
  • In spec file, I have also added: `from patoolib import configuration from patoolib import util from patoolib import programs` and in hidden_imports: `'patoolib.util', 'patoolib.configuration', 'patoolib.programs'` but still no effect – Matt Dnv Mar 15 '20 at 09:44
  • nothing changed by adding `'pyunpack.cli'` and `'pyunpack.about'` either – Matt Dnv Mar 15 '20 at 10:23
  • 2
    I'm closing the issue you posted on GitHub; SO is the place for this. Give me a day or two to figure it out. – Legorooj Mar 15 '20 at 23:14
  • @Legorooj, thanks a lot, I have added a second EDIT locating the problem on the pyunpack side, hoping it can help you – Matt Dnv Mar 16 '20 at 07:58
  • Have you considered unpacking the files in "file_to_be_unzipped.sent" in pyunpack spec script and setting `datas` option to `Analysis` to include this file? Not sure I see a reason to perform the procedure during runtime in the packed executable. – Oluwafemi Sule Mar 16 '20 at 14:12

2 Answers2

1

The problem is, as you found in your 2nd edit, that pyunpack needs patool installed on the target system - which, in turn, requires python.

To fix this you need to:

  • Use tarballs or zipfiles; shutil supports theses.

  • Or require the end user to have python installed.

Another way of putting it is that to use pyunpack with pyinstaller, you need python on the target system.

Legorooj
  • 2,646
  • 2
  • 15
  • 35
  • Hey @legorooj: sorry for the late answer, I've been sick from the famous covid for the last 12 days. I'd like more precision: your solution 2 seems to be what I have suggested in my edit 3 if i understand well (please deny otherwise). Could you please give more precision regarding your solution number 1? Thanks and regards – Matt Dnv Mar 28 '20 at 11:01
  • I have also tried to import the whole of anaconda, to spare time having to select the anaconda files one by one; but it still doesn't work: aparat from making a uge cumbersome executable file, it makes new errors in script/patool – Matt Dnv Mar 28 '20 at 11:05
  • 1
    @MattDnv at least you're immune as far as we know now... https://docs.python.org/3/library/shutil.html#shutil.unpack_archive is the function you want, or look into `tarfile` and `zipfile` modules - both stdlib. Yes solution 2 == edit 3. – Legorooj Mar 28 '20 at 11:10
-2

The output of print(abspath(".")) when running directly in python is:

C:\Users\myname\eclipse-workspace-tms\test_unzip_exe

While running with pyinstaller is:

C:\Users\myname\eclipse-workspace-tms\test_unzip_exe\dist\test

Thus, the archive is not extracted in the dist\test folder.

Fix this by setting path to return value of abspath(".") instead of hard-coding it to C:\Users\myname\eclipse-workspace-tms\test_unzip_exe

Oluwafemi Sule
  • 36,144
  • 1
  • 56
  • 81
  • Thanks for answering Oluwafemi, however, in my opinion, it doesn't solve the problem: I really do want to extract into the folder: `path = "C:\\Users\\myname\\eclipse-workspace-tms\\test_unzip_exe\\grog"` the existence of which is guaranteed, for instance by the mkdir function – Matt Dnv Mar 15 '20 at 08:07