0

I am trying to install the required Python packages for a Python project in Python 3.11 (for Windows), using pip install -r requirements.txt. My libraries that I need to download are:

    numpy
transformers==v3.1.0
tqdm
torch
scikit-learn
spacy
torchtext
pandas
nltk
sentence_transformers

tokenizers are needed for one of my packages to run, however my Anaconda failed to build wheel for this package. At first it was caused by my lack of Rust compiler, so I install them like in this question: Could not build wheels for tokenizers, which is required to install pyproject.toml-based projects. However, when I reinstall my tokenizers I got this problem:

      running build_ext
  running build_rust
  cargo rustc --lib --message-format=json-render-diagnostics --manifest-path Cargo.toml --release -v --features pyo3/extension-module -- --crate-type cdylib
  warning: unused manifest key: target.x86_64-apple-darwin.rustflags
      Updating crates.io index
      Updating git repository `https://github.com/n1t0/rayon-cond`
   Downloading crates ...
  error: failed to download `once_cell v1.17.1`

  Caused by:
    unable to get packages from source

  Caused by:
    failed to parse manifest at `C:\Users\hoang\.cargo\registry\src\github.com-1ecc6299db9ec823\once_cell-1.17.1\Cargo.toml`

  Caused by:
    failed to parse the `edition` key

  Caused by:
    this version of Cargo is older than the `2021` edition, and only supports `2015` and `2018` editions.
  error: `cargo rustc --lib --message-format=json-render-diagnostics --manifest-path Cargo.toml --release -v --features pyo3/extension-module -- --crate-type cdylib` failed with code 101
  [end of output]



 note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for tokenizers
Failed to build tokenizers
ERROR: Could not build wheels for tokenizers, which is required to install pyproject.toml-based projects

As I check my Cargo and Rust version, both of them are from version 1.68.2 (in 2023), so I am not sure what has gone wrong with my installation. So what methods could I use to fix this problem?

Update: Here is the result when I look into Cargo.toml file above:

[package]
edition = "2021"
rust-version = "1.56"
name = "once_cell"
version = "1.17.1"
authors = ["Aleksey Kladov <aleksey.kladov@gmail.com>"]
exclude = [
    "*.png",
    "*.svg",
    "/Cargo.lock.msrv",
    "rustfmt.toml",
]

Another detail that I would like to add is that my packages needed tokenizers to run, so it requires tokenizers-0.8.1; and when I installed tokenizers directly from pip, pip would only download the wheel file and not building any wheels.

Hoang Cuong Nguyen
  • 329
  • 2
  • 11
  • 2
    Same as [this question](https://stackoverflow.com/q/75838056/5397009), you probably have an older version of Rust installed somewhere and `pip` is using that older version instead of the one you checked. – Jmb Apr 12 '23 at 06:19
  • Yeah I see, but how do I know that which version of Rust is being used by pip? And I did reinstall Rust in my laptop quite a bit earlier, and still got the same problem. – Hoang Cuong Nguyen Apr 12 '23 at 06:26
  • 2
    You can try adding the `--verbose` and/or `--debug` options to `pip` and see if that prints the path of the `cargo` command it's running. – Jmb Apr 12 '23 at 06:34
  • I have updated my question above for `--verbose` option, and I hope you could help a look into it because I am not sure about the `cargo` command. – Hoang Cuong Nguyen Apr 12 '23 at 06:48
  • Unfortunately the verbose trace you have removed from the question don't give the path to the `cargo` version used by `pip`. – Jmb Apr 12 '23 at 07:11
  • Yes the `Cargo.toml` file is saying that it _requires_ edition 2021, but for some reason `pip` is running an older version of the `cargo` executable. – Jmb Apr 12 '23 at 07:13
  • What command do you run to install `tokenizers`? I just tried to `pip install tokenizers` on Python 3.1 Windows 64 bits and it downloaded a pre-compiled wheel instead of attempting to build it. – Jmb Apr 12 '23 at 07:17
  • Oh, I am sorry for deleting the `verbose` description! I have updated my questions already, and I think that could help you better for this case. – Hoang Cuong Nguyen Apr 12 '23 at 09:09

0 Answers0