2

I am trying to install delta-lake-reader[aws] on my MacBook Pro with MacOS Mojave and it is failing. I have Python 3.9.1 installed on my laptop.

$ pip3 install delta-lake-reader[aws]


Collecting delta-lake-reader[aws]
  Using cached delta_lake_reader-0.2.2-py3-none-any.whl (7.4 kB)
Collecting pyarrow<3.0.0,>=2.0.0
  Using cached pyarrow-2.0.0.tar.gz (58.9 MB)
  Installing build dependencies ... |

After that it hangs for a while, then fails with a lot of errors, and finally, it gives the following output before ending:

ERROR: Cannot install delta-lake-reader[aws]==0.1.0, delta-lake-reader[aws]==0.1.1, delta-lake-reader[aws]==0.2.1 and delta-lake-reader[aws]==0.2.2 because these package versions have conflicting dependencies.

The conflict is caused by:
    delta-lake-reader[aws] 0.2.2 depends on pyarrow<3.0.0 and >=2.0.0
    delta-lake-reader[aws] 0.2.1 depends on pyarrow<3.0.0 and >=2.0.0
    delta-lake-reader[aws] 0.1.1 depends on pyarrow<3.0.0 and >=2.0.0
    delta-lake-reader[aws] 0.1.0 depends on pyarrow<3.0.0 and >=2.0.0

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies
$

I tried uninstalling pyarrow but it installs 3.0.0 again and fails the same way.

How can I overcome this?

Rafiq
  • 1,380
  • 4
  • 16
  • 31

1 Answers1

2

The problem lays in the explicit dependency, that doesn't include pyarrow 3.0.0, but requires 2.x.x version. And pyarrow 2.x.x is not compatible with Python 3.9, that's why your build fails, but it works with Python 3.8, I was able to install it with pip3.8 install 'delta-lake-reader[aws]==0.2.2'. So you have following choice:

  • Use Python 3.8
  • Download repository, and make change in the pyproject.toml to use pyarrow 3.0.0, and after that use poetry tool to build wheel
  • File issue in the project, and wait until author fixes it.
  • If you don't need additional functionality, then you can use official Delta Lake python package that could be installed with pip install deltalake command
Alex Ott
  • 80,552
  • 8
  • 87
  • 132
  • Thank you @Alex Ott for the prompt and detailed response. I'd like to Python 3.8 for a different number of reasons including this one. Is uninstalling 3.9 and installing 3.8 on MacOS a complex job? I have other libraries like Boto3 and so on, and VS Code tied up with this version. If I can uninstall 3.9 and install 3.8 those I can re-install. Python folders on my mac `/usr/bin/python` and `/usr/local/bin/python3` for Python 2.7 and 2.9 respectively. – Rafiq Apr 10 '21 at 17:04
  • It depends on how it’s installed. I think that you can select specific version as default in homebrew – Alex Ott Apr 10 '21 at 17:05
  • ah, ok - 3.9 is the system python. I think that you can install python 3.8 via homebrew, and just make it default instead of 3.9 - see answer here: https://stackoverflow.com/questions/60453261/how-to-default-python3-8-on-my-mac-using-homebrew – Alex Ott Apr 10 '21 at 17:18
  • I just reviewed all the solutions in the the link you gave me in last comment, but not sure which will be a safer option for me. The accepted solution there, by the question poster, is from here https://discourse.brew.sh/t/how-to-default-python-3-8-on-my-mac-using-homebrew/7050/4. I reviewed that discussion as well. `pyenv` is another option. I know, this is a different question, but I feel you can suggest me a safer option. – Rafiq Apr 10 '21 at 18:46
  • It’s really depends on your workflow. I personally would be just follow homebrew instructions – Alex Ott Apr 10 '21 at 18:47