Could not build wheels for tokenizers, which is required to install pyproject.toml-based projects

Question

I am trying to install the ernie and tokenizers on Windows, but for both cases it is showing the following error:

ERROR: Command errored out with exit status 1:
   command: 'c:\users\my pc\appdata\local\programs\python\python39\python.exe' 'c:\users\my pc\appdata\local\programs\python\python39\lib\site-packages\pip\_vendor\pep517\in_process\_in_process.py' build_wheel 'C:\Users\MYPC~1\AppData\Local\Temp\tmpyfqm86h3'
       cwd: C:\Users\my pc\AppData\Local\Temp\pip-install-504a8mvv\tokenizers_a45d458877334719a3bf67c944dce6d7
  Complete output (38 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build\lib
  creating build\lib\tokenizers
  copying tokenizers\__init__.py -> build\lib\tokenizers
  running build_ext
  running build_rust
      Updating crates.io index
  cargo rustc --lib --manifest-path Cargo.toml --features pyo3/extension-module --target x86_64-pc-windows-msvc --release --verbose -- --crate-type cdylib
  warning: unused manifest key: target.x86_64-apple-darwin.rustflags
     Compiling proc-macro2 v1.0.30
     Compiling unicode-xid v0.2.2
     Compiling syn v1.0.80
     Compiling autocfg v1.0.1
       Running `rustc --crate-name build_script_build --edition=2018 "C:\Users\my pc\.cargo\registry\src\github.com-1ecc6299db9ec823\proc-macro2-1.0.30\build.rs" --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debug-assertions=off --cfg "feature=\"default\"" --cfg "feature=\"proc-macro\"" -C metadata=0b72d9facdf9b740 -C extra-filename=-0b72d9facdf9b740 --out-dir "C:\Users\my pc\AppData\Local\Temp\pip-install-504a8mvv\tokenizers_a45d458877334719a3bf67c944dce6d7\target\release\build\proc-macro2-0b72d9facdf9b740" -L "dependency=C:\Users\my pc\AppData\Local\Temp\pip-install-504a8mvv\tokenizers_a45d458877334719a3bf67c944dce6d7\target\release\deps" --cap-lints allow`
       Running `rustc --crate-name unicode_xid "C:\Users\my pc\.cargo\registry\src\github.com-1ecc6299db9ec823\unicode-xid-0.2.2\src\lib.rs" --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debug-assertions=off --cfg "feature=\"default\"" -C metadata=c5404e573edb1c5f -C extra-filename=-c5404e573edb1c5f --out-dir "C:\Users\my pc\AppData\Local\Temp\pip-install-504a8mvv\tokenizers_a45d458877334719a3bf67c944dce6d7\target\release\deps" -L "dependency=C:\Users\my pc\AppData\Local\Temp\pip-install-504a8mvv\tokenizers_a45d458877334719a3bf67c944dce6d7\target\release\deps" --cap-lints allow`
       Running `rustc --crate-name build_script_build --edition=2018 "C:\Users\my pc\.cargo\registry\src\github.com-1ecc6299db9ec823\syn-1.0.80\build.rs" --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debug-assertions=off --cfg "feature=\"clone-impls\"" --cfg "feature=\"default\"" --cfg "feature=\"derive\"" --cfg "feature=\"extra-traits\"" --cfg "feature=\"full\"" --cfg "feature=\"parsing\"" --cfg "feature=\"printing\"" --cfg "feature=\"proc-macro\"" --cfg "feature=\"quote\"" -C metadata=b3126876d28730d4 -C extra-filename=-b3126876d28730d4 --out-dir "C:\Users\my pc\AppData\Local\Temp\pip-install-504a8mvv\tokenizers_a45d458877334719a3bf67c944dce6d7\target\release\build\syn-b3126876d28730d4" -L "dependency=C:\Users\my pc\AppData\Local\Temp\pip-install-504a8mvv\tokenizers_a45d458877334719a3bf67c944dce6d7\target\release\deps" --cap-lints allow`
       Running `rustc --crate-name autocfg "C:\Users\my pc\.cargo\registry\src\github.com-1ecc6299db9ec823\autocfg-1.0.1\src\lib.rs" --error-format=json --json=diagnostic-rendered-ansi --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debug-assertions=off -C metadata=077a4bc4a690ef7e -C extra-filename=-077a4bc4a690ef7e --out-dir "C:\Users\my pc\AppData\Local\Temp\pip-install-504a8mvv\tokenizers_a45d458877334719a3bf67c944dce6d7\target\release\deps" -L "dependency=C:\Users\my pc\AppData\Local\Temp\pip-install-504a8mvv\tokenizers_a45d458877334719a3bf67c944dce6d7\target\release\deps" --cap-lints allow`
     Compiling memchr v2.4.1
       Running `rustc --crate-name build_script_build --edition=2018 "C:\Users\my pc\.cargo\registry\src\github.com-1ecc6299db9ec823\memchr-2.4.1\build.rs" --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debug-assertions=off --cfg "feature=\"default\"" --cfg "feature=\"std\"" -C metadata=420338dd5ea363a5 -C extra-filename=-420338dd5ea363a5 --out-dir "C:\Users\my pc\AppData\Local\Temp\pip-install-504a8mvv\tokenizers_a45d458877334719a3bf67c944dce6d7\target\release\build\memchr-420338dd5ea363a5" -L "dependency=C:\Users\my pc\AppData\Local\Temp\pip-install-504a8mvv\tokenizers_a45d458877334719a3bf67c944dce6d7\target\release\deps" --cap-lints allow`
  error: linker `link.exe` not found
    |
    = note: The system cannot find the file specified. (os error 2)

  note: the msvc targets depend on the msvc linker but `link.exe` was not found

  note: please ensure that VS 2013, VS 2015, VS 2017 or VS 2019 was installed with the Visual C++ option

  error: could not compile `syn` due to previous error

  Caused by:
    process didn't exit successfully: `rustc --crate-name build_script_build --edition=2018 "C:\Users\my pc\.cargo\registry\src\github.com-1ecc6299db9ec823\syn-1.0.80\build.rs" --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debug-assertions=off --cfg "feature=\"clone-impls\"" --cfg "feature=\"default\"" --cfg "feature=\"derive\"" --cfg "feature=\"extra-traits\"" --cfg "feature=\"full\"" --cfg "feature=\"parsing\"" --cfg "feature=\"printing\"" --cfg "feature=\"proc-macro\"" --cfg "feature=\"quote\"" -C metadata=b3126876d28730d4 -C extra-filename=-b3126876d28730d4 --out-dir "C:\Users\my pc\AppData\Local\Temp\pip-install-504a8mvv\tokenizers_a45d458877334719a3bf67c944dce6d7\target\release\build\syn-b3126876d28730d4" -L "dependency=C:\Users\my pc\AppData\Local\Temp\pip-install-504a8mvv\tokenizers_a45d458877334719a3bf67c944dce6d7\target\release\deps" --cap-lints allow` (exit code: 1)
  warning: build failed, waiting for other jobs to finish...
  error: build failed
  error: cargo failed with code: 101

  ----------------------------------------
  ERROR: Failed building wheel for tokenizers
Failed to build tokenizers
ERROR: Could not build wheels for tokenizers, which is required to install pyproject.toml-based projects

Yous should copy and paste all lines of the error. Also, please include the OS information. — umitu, Oct 17 '21 at 00:52
I fixed the code formatting for you, but this is not the full error message. Please copy-paste the actual error message you get into the question, at least the part from "Building wheel for tokenizers" until the end! This part only contains the "summary" which is that the command failed, however, it does not include the relevant information about what failed. This makes it difficult to answer the question. — wovano, Oct 18 '21 at 18:22
Now that you've included the relevant information, the cause of the error is pretty clear, right? **_error: linker `link.exe` not found_** And the solution is also mentioned in the error message: ***please ensure that VS 2013, VS 2015, VS 2017 or VS 2019 was installed with the Visual C++ option*** — wovano, Oct 19 '21 at 11:03
Sorry the solution was not clear, could you please elaborate. I am using the visual studio code and i installed an extension called visual c++ option in visual studio code but what is vs 2013 and how do i ensure it. — Jaswanth Ch, Oct 20 '21 at 16:14
You need Visual Studio (the compiler) version 2013 or higher to build this Python library. Visual Studio Code is an IDE. Although the name is similar it's something completely different! You can download Visual Studio [here](https://visualstudio.microsoft.com/visual-cpp-build-tools/). I recently did it to install another Python library. [This answer](https://stackoverflow.com/a/69565927/10669875) might be helpful as well. — wovano, Oct 20 '21 at 16:20

score 38 · Answer 1 · answered Apr 07 '22 at 09:21

38

It need to install rust compiler in your environment, you can use the command:

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

and then you can do it successfully.

answered Apr 07 '22 at 09:21

AlanFok

381
3
3

7

this almost worked for me except that I had to take the extra step of executing `source $HOME/.carge/env` after the rust compiler was installed. – doij790 Jun 26 '22 at 23:38
3

There's a small typo -- it should be `source $HOME/.cargo/env ` but yes, this worked for me too. – nim.py Sep 27 '22 at 14:44
where do we run this command? – Furkan Gözükara Oct 18 '22 at 21:51
@FurkanGözükara in the terminal once the Rust install is complete. It should tell you to do it as soon as it's done. It even gives you the command. Restarting the shell has the same effect. – BagMan Mar 01 '23 at 16:52
It works! But you need to log out and log in back then reactivate the virtualenv. – CK.Nguyen Apr 14 '23 at 07:47
If this does not work, try downgrading from Python 3.10 to Python 3.8 – TSR May 05 '23 at 19:22

score 4 · Answer 2 · answered May 31 '22 at 15:54

4

So what worked for me is twofold: -1. Install Rust -2. Install Visual Studio (latest version, 2022) and make sure to install BOTH Python workloads and Desktop Development C++ Workloads.

answered May 31 '22 at 15:54

saladin1991

142
9

score 3 · Answer 3 · answered Jul 24 '22 at 16:06

3

It could be because of the version of python. I got the same error when I tried to install the transformers library and I was using python 3.6 but when I upgraded it to python 3.8, the library got installed successfully.

answered Jul 24 '22 at 16:06

Nesrine Mejbri

31
1

score 3 · Answer 4 · answered May 02 '23 at 11:11

3

After exploring rust, cargo, the wheels, and a lot of stuff I realized that my python version (3.10) was the problem. I created another environment with the 3.8 version while this rust bug is fixed.

answered May 02 '23 at 11:11

Cristian Rojas

31
2

user67275 · Answer 5 · 2021-12-12T09:33:52.070

I solved this problem by installing Visual Studio 2019 Community Edition with Python development workloads. I am not sure whether the Python development workloads were essential. Rust compiler has already been installed. (rustup-init.exe from www.rust-lang.org)

Edit: When I uninstalled Python 3.10, Visual Studio, and Rust completely and tried again with only Python 3.10 and Visual Studio Community edition with Python development workloads, it didn't work. But it succeeded after I installed Desktop Development with C++ workloads.

Could not build wheels for tokenizers, which is required to install pyproject.toml-based projects

5 Answers5

Linked