UserWarning: Could not import the lzma module. Your installed Python is incomplete

Question

After Installing Google Cloud Bigquery Module, if I import the module into python code. I see this warning message. Happening to me in python 3.7.3 Virtualenv.

Tried to reinstall GCP bigquery module Expectation-in python code if we write" from google.cloud import bigquery ".Should not result in any error or messege.

import os
import sys
import logging
from datetime import datetime
from google.cloud import bigquery

/home/informatica/.local/lib/python3.7/site-packages/pandas/compat/__init__.py:84: UserWarning: Could not import the lzma module. Your installed Python is incomplete. Attempting to use lzma compression will result in a RuntimeError.
  warnings.warn(msg)
 exit()

>>> import os >>> import sys >>> import logging >>> from datetime import datetime >>> from google.cloud import bigquery /home/informatica/.local/lib/python3.7/site-packages/pandas/compat/__init__.py:84: UserWarning: Could not import the lzma module. Your installed Python is incomplete. Attempting to use lzma compression will result in a RuntimeError. warnings.warn(msg) >>> exit() — Sreekanth, Sep 01 '19 at 04:39
The `Lzma` module is part of the *python standard library*, which means something is wrong with your python download. I would try uninstalling and reinstalling python. — Lord Elrond, Sep 01 '19 at 07:04
3.8.3 installed using pyenv on CentOS and same issue. None of the fixes below worked. — TheProletariat, Jul 10 '20 at 17:48
Performing steps in this [site](https://tecadmin.net/install-python-3-8-ubuntu/) worked for me. You can do this even you already have installed python, there is no need to uninstall. — highlytrainedbadger, Feb 15 '21 at 07:16

score 136 · Answer 1 · answered Sep 03 '19 at 14:22

136

If you compile Python from source, you must have the lzma-dev package installed, or it will not be built into python.

For ubuntu: sudo apt-get install liblzma-dev

For centos: yum install -y xz-devel

Then configure && make && make install

answered Sep 03 '19 at 14:22

Chenglu

1,757
2
14
23

17

Where do I do that configure and make? – Darwin Harianto Oct 28 '19 at 07:21
1

I ran into this issue from compiling the python from the souce code. So the configure and make is do in the directory of the source code. – Chenglu Oct 28 '19 at 09:13
1

@matt525252 The source code needs to be downloaded from python.org https://www.python.org/downloads/ – Phani Rithvij May 13 '20 at 15:10
10

@DarwinHarianto if you install Python from source go to the python source code folder and run commands `./configure && make && make altinstall` here. Also I recommend using `sudo altinstall` instead to not to rewtite default python version. – Mikhail_Sam Jun 16 '20 at 10:46
8

FOR CENTOS: Firstly ```sudo yum install -y xz-devel``` Then recompile python from source code ```cd Python-3.8*/``` then ```./configure --enable-optimizations``` then ```sudo make altinstall``` – Kyrylo Malakhov Nov 13 '20 at 17:07
1

`./configure` can be found where your python is installed. In my case, it was under `/usr/src/python-3.x.x/` – Ashish Kumar Jun 30 '21 at 10:38
Note that if you've compiled you're own Python distribution (eg through ```pyenv install```), you'll need to re-compile the distribution after installing the package. – Panagiss Nov 11 '22 at 08:13
If you use pyenv, you might also run into this issue. Installing lzma-dev as described and then `pyenv install x.y.z` and overwriting the existing installation, where x.y.z is the version you are using, resolved the issue. – Fabian Hertwig Jan 10 '23 at 18:23

score 60 · Answer 2 · answered Jun 16 '20 at 11:11

60

I used other good answers from here and didn't solve the problem (Ubuntu 18.04, Python3.8), still get this warning. Actually there is one more package is needed to be installed to solve the problem:

sudo apt-get install lzma

So the whole pipeline (run in the python source code folder):

sudo apt-get install liblzma-dev
sudo apt-get install lzma
./configure --enable-optimizations
sudo make
sudo make altinstall

answered Jun 16 '20 at 11:11

Mikhail_Sam

10,602
11
66
102

1

This is exactly what helped me, thank you =) (Ubuntu 18.04, Python3.8) – Ersain Jul 09 '20 at 10:42
1

This helped me on Ubuntu 20.04, Python 3.8.5. – Henrik Jul 28 '20 at 11:18
@Li-PinJuan I'm not sure we talking about the same thing, but I thought using `altinstall` prevents from overwriting default Python. – Mikhail_Sam Aug 17 '20 at 06:43
3

my soultion: sudo apt-get install liblzma-dev sudo apt-get install lzma pyenv install 3.6.6 (reinstall) – walkman Sep 09 '20 at 08:31
2

For Ubuntu 18.04 this is the answer that helps. It is not enough to install `liblzma-dev` – Corel Jan 16 '21 at 16:40
code still seems to run fine even though error statement, what are some consequences of ignoring warnings like this? – haneulkim May 28 '21 at 01:25
@haneulkim at the moment I can't remember for sure, but because of this some of the python libraries (like pandas) did not work for me – Mikhail_Sam Oct 08 '21 at 08:32
It worked for me. 'run in the python source code folder' is important. – Sherman Chen Oct 11 '21 at 15:12

score 59 · Answer 3 · answered Oct 23 '19 at 08:18

59

On MacOS and pyenv (https://realpython.com/intro-to-pyenv/), I was able to have this warning go away by having xz installed with homebrew. Using version python 3.6.9 as an example

brew install xz && pyenv install 3.6.9

To use installed python, one needs to add this into .bash_profile

eval "$(pyenv init -)"

and start using it by running

pyenv global 3.6.9

answered Oct 23 '19 at 08:18

Mika Riekkinen

730
4
9

7

for mac OS: ```brew install readline xz```
https://github.com/pyenv/pyenv/wiki/Common-build-problems – Sean Apr 17 '20 at 05:20
1

I needed to run two more commands before I finally got rid of the lzma error: `set -gx CPPFLAGS "-I/usr/local/opt/readline/include"` `set -gx LDFLAGS "-L/usr/local/opt/readline/lib"` After that, I re-installed python versions via pyenv, and no longer saw the lzma error. Hope this helps! – Swaraj Oct 02 '20 at 19:24
1

I tried the comments above and now I have a bunch of printouts in my terminal. Can you tell me how to undo the effects of these commands? Thanks! – Nicole Goebel Oct 05 '20 at 19:04
@NicoleGoebel Did you figure out how to get rid of the extra printouts? – Abhishek Kasireddy Oct 15 '20 at 03:25
I ended up re-installing python and requirements and it seemed to fix things ... – Nicole Goebel Oct 16 '20 at 04:14

score 53 · Answer 4 · answered Apr 30 '20 at 19:48

53

On macOS, if you manage your python with pyenv and package with homebrew, you need to install "xz" first:

brew install xz

After installing xz, you can install python 3.8 by (I'm using 3.8.2 as an example:

pyenv install 3.8.2

Above will fix the problem.

answered Apr 30 '20 at 19:48

Yingbo Miao

882
9
12

2

This fixed it for me! – Chris Abbott Jun 15 '20 at 05:11
1

How is this different from Mika Riekkinen's answer? – Joshua Schlichting Jul 08 '21 at 14:07
1

I was about to post what worked for me, but saw that the same solution has been posted here. :) – Johnson Nov 04 '21 at 06:32
@JoshuaSchlichting There were differences. I assume the someone updated the answer. – Yingbo Miao Feb 28 '22 at 02:44

dwolfeu · Answer 5 · 2023-02-08T21:29:03.420

25

This solution* worked on my setup (Apple M1 with pyenv):

CFLAGS="-I$(brew --prefix xz)/include" LDFLAGS="-L$(brew --prefix xz)/lib" pyenv install 3.9.4

*Edit (2023-02-08): Link removed following the helpful comment from Martin Delille.

edited Feb 08 '23 at 21:29

answered May 18 '21 at 18:15

dwolfeu

1,103
2
14
21

Same issue on apple M1, but I get "command not found: brew" when defining both brew (/opt/homebrew/bin) and these flags in my .zprofile. Any thoughts? – Entropy Jun 26 '21 at 02:01
Might this help? https://stackoverflow.com/questions/18428374/commands-not-found-on-zsh – dwolfeu Jun 28 '21 at 07:57
What do you do with this code? Can you elaborate? – Ege Can Jul 06 '21 at 08:55
Super! Thanks! Fixed the message on my Apple M1 chip laptop as well. In python version 3.9.6 for me -- as a python reinstall alone did nothing. – warrens Jul 14 '21 at 12:07
2

This worked with my already installed python installation (pyenv). E.g. no need to install python again if you are mid project. @EgeCan I think what this does is tell the OS where to look for the binaries. I.e. when referring to xz the OS needs to know about xz. – Ole Henrik Skogstrøm Oct 08 '21 at 15:16
1

@EgeCan note the `pyenv install 3.9.4` at the end of the solution, so when you use `pyenv` to install a version of python use this whole line in terminal and change the version from `3.9.4` to the one you want. Note, unclear to me if this is a once off or if you need all of the code before `pyenv` for every install. – qwertytam Oct 22 '21 at 18:13
1

Your link seem to be broken and unsafe... – Martin Delille Feb 07 '23 at 10:37

score 20 · Answer 6 · answered Mar 27 '20 at 16:49

20

If you are using centos and compile python from source, you can install from following commands

For centos: sudo yum install -y xz-devel

Recompile python from source code

cd Python-3.8*/
./configure --enable-optimizations
sudo make altinstall

answered Mar 27 '20 at 16:49

Rafayet Ullah

1,108
4
14
27

2

Exactly what I needed. – berkoab Jun 10 '20 at 13:13
Exactly what I needed for centos! Thanks, Bro! – Kyrylo Malakhov Nov 13 '20 at 17:06

Ion Toloaca · Answer 7 · 2021-04-04T02:03:11.213

11

See last comment on https://github.com/pandas-dev/pandas/issues/27532

brew install xz # To pick up liblzma
prefix=$(brew --prefix)
export LDFLAGS="-L$prefix/opt/xz/lib $LDFLAGS"
export CPPFLAGS="-I$prefix/opt/xz/include $CPPFLAGS"
export PKG_CONFIG_PATH="$prefix/opt/xz/lib/pkgconfig:$PKG_CONFIG_PATH"
# YOU CANNOT HAVE THE GNUBINS in your PATH when you run this
PYTHON_CONFIGURE_OPTS="--enable-shared" pyenv install 3.9.2
python3 -c "import lzma" # should work and not throw "cannot import _lzma"

edited Apr 04 '21 at 02:03

answered Apr 04 '21 at 01:46

Ion Toloaca

111
1
4

reinstalling python with `pyenv install 3.7.9` solve for me, not sure why ‍♂️ – shlomiLan Sep 30 '21 at 10:50

score 8 · Answer 8 · answered Apr 18 '21 at 21:20

What solved for me:

sudo apt-get install libbz2-dev
sudo cp /usr/lib/python3.8/lib-dynload/_bz2.cpython-38-x86_64-linux-gnu.so/usr/local/lib/python3.8/
sudo cp /usr/lib/python3.8/lib-dynload/_lzma.cpython-38-x86_64-linux-gnu.so /usr/local/lib/python3.8/

Naveen Reddy Marthala · Answer 9 · 2022-03-14T16:35:44.143

5

I had built Python 3.8 from source on Debian 10 and some times couldn't start mlflow server at all and sometimes got warnings albeit successful launches and also every pandas import gave me this warning.

Here's what worked for me:

purged the existing installation.
did sudo apt install libncurses-dev libgdbm-dev libz-dev tk-dev libsqlite3-dev libreadline-dev liblzma-dev libffi-dev libssl-dev
built python from source again.

I never got the warning again and had no problems whatsoever.

edited Mar 14 '22 at 16:35

answered Jun 05 '21 at 12:25

Naveen Reddy Marthala

2,622
4
35
67

You need to install these packages and then resinstall the Python binaries (run `(sudo) apt install ...` before running `(sudo) make`). – Zhanwen Chen Jun 09 '23 at 23:54

score 4 · Answer 10 · answered Dec 07 '21 at 19:24

4

I found the solution from: https://github.com/pandas-dev/pandas/issues/28219

I just ran: CPPFLAGS="-I$(brew --prefix xz)/include" pyenv install 3.10.0

OS: Monterey
M1 chip
pyenv
python 3.10.0

answered Dec 07 '21 at 19:24

dami.max

377
3
17

score 3 · Answer 11 · answered Feb 10 '21 at 09:49

I did brew install xz and reinstalled everything, but that didn't do it for me.

What helped me was to add correct linkage for xz as well:


    export LDFLAGS="-L$(brew --prefix xz)/lib -L$(brew --prefix readline)/lib -L$(brew --prefix zlib)/lib"
    export CPPFLAGS="-I$(brew --prefix xz)/include  -I$(brew --prefix readline)/include -I$(brew --prefix zlib)/include -I$(xcrun --show-sdk-path)/usr/include"

score 2 · Answer 12 · answered Sep 29 '21 at 18:49

2

I see that

yum install -y lzma

Also runs without errors.

answered Sep 29 '21 at 18:49

AnupamB

31
4

I was having this problem with pyenv install 3.9.2, 3.9.9 on Amazon Linux 2. Installing lzma modue just fixed that issue. – Tyn Jan 21 '22 at 12:31

score 1 · Answer 13 · answered Oct 08 '21 at 11:30

1

My OS is : CentOS 8.X

Step to step run command below, then fix the problem:

sudo yum install -y xz-devel
cd Python-3.8.5
sudo ./configure --prefix=/usr/local/python3.8.5 --enable-optimizations --with-ssl
sudo make
sudo make install

answered Oct 08 '21 at 11:30

bluetata

577
7
12

in step 3, you should modify the path to your python installed folder if it's necessary. – bluetata Nov 02 '21 at 03:14

sanjarcode · Answer 14 · 2023-06-09T12:47:08.787

1

note: For Linux/Ubuntu

Step 1: sudo apt-get install liblzma-dev

Step 2: pyenv install <python_version>. This will prompt that the version is already installed, say yes. This will cause a reinstall, this time "completely".

edited Jun 09 '23 at 12:47

answered Apr 03 '23 at 18:59

sanjarcode

437
5
15

score 0 · Answer 15 · answered Aug 15 '22 at 09:23

I'd like to provide some info on what "xz" and "lzma" are.

man xz:

xz, unxz, xzcat, lzma, unlzma, lzcat - Compress or decompress .xz and .lzma files

https://www.nongnu.org/lzip/xz_inadequate.html :

xz is a container format which currently contains another container format (LZMA2), which in turn contains a mix of LZMA data and uncompressed data.

Well, as far as I understand, lzma is a compression algorithm and xz is the file format.

For example, on https://www.python.org/downloads/release/python-3106/ , there's XZ compressed source tarball compared to Gzipped source tarball .

So if you don't need to deal with the .xz files with pandas, you could just ignore the warning and keep going. Otherwise you need to reinstall python and you alreay have lzma libraries installed in the system. The lzma library's package name varies in different distros/OS. For example, on mac it's xz and ubuntu it's liblzma-dev.

Also see:

https://pandas.pydata.org/pandas-docs/dev/whatsnew/v0.25.1.html#io-and-lzma

https://github.com/pandas-dev/pandas/blob/b5958ee1999e9aead1938c0bba2b674378807b3d/pandas/_testing.py#L245

lzma vs zlib:

https://joblib.readthedocs.io/en/latest/auto_examples/compressors_comparison.html

UserWarning: Could not import the lzma module. Your installed Python is incomplete

15 Answers15

Linked