I have created a web scraping project using the requests-html library (in a venv). It was all working well until I stupidly downloaded python again in the normal terminal (instead of the vscode terminal). Since then the code doesn't work anymore and I cannot figure out what the problem is.
So I have opened a new file to test my code iteratively without a venv and I cannot even install the requests-html library at all: ModuleNotFoundError: No module named 'requests_html
.
So I run pip install requests-html
and it gives me the following error:
Collecting requests-html
Using cached requests_html-0.10.0-py3-none-any.whl (13 kB)
Collecting pyquery
Using cached pyquery-1.4.3-py3-none-any.whl (22 kB)
Collecting pyppeteer>=0.0.14
Using cached pyppeteer-1.0.2-py3-none-any.whl (83 kB)
Collecting bs4
Using cached bs4-0.0.1-py3-none-any.whl
Requirement already satisfied: w3lib in /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages (from requests-html) (2.0.1)
Requirement already satisfied: parse in /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages (from requests-html) (1.19.0)
Requirement already satisfied: requests in /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages (from requests-html) (2.28.1)
Requirement already satisfied: fake-useragent in /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages (from requests-html) (0.1.11)
Collecting importlib-metadata>=1.4
Using cached importlib_metadata-5.0.0-py3-none-any.whl (21 kB)
Requirement already satisfied: certifi>=2021 in /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages (from pyppeteer>=0.0.14->requests-html) (2022.9.24)
Requirement already satisfied: appdirs<2.0.0,>=1.4.3 in /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages (from pyppeteer>=0.0.14->requests-html) (1.4.4)
Requirement already satisfied: pyee<9.0.0,>=8.1.0 in /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages (from pyppeteer>=0.0.14->requests-html) (8.2.2)
Requirement already satisfied: tqdm<5.0.0,>=4.42.1 in /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages (from pyppeteer>=0.0.14->requests-html) (4.64.1)
Requirement already satisfied: urllib3<2.0.0,>=1.25.8 in /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages (from pyppeteer>=0.0.14->requests-html) (1.26.12)
Requirement already satisfied: websockets<11.0,>=10.0 in /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages (from pyppeteer>=0.0.14->requests-html) (10.3)
Collecting beautifulsoup4
Using cached beautifulsoup4-4.11.1-py3-none-any.whl (128 kB)
Collecting cssselect>0.7.9
Using cached cssselect-1.1.0-py2.py3-none-any.whl (16 kB)
Collecting lxml>=2.1
Using cached lxml-4.9.1.tar.gz (3.4 MB)
Preparing metadata (setup.py) ... done
Requirement already satisfied: idna<4,>=2.5 in /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages (from requests->requests-html) (3.4)
Requirement already satisfied: charset-normalizer<3,>=2 in /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages (from requests->requests-html) (2.1.1)
Requirement already satisfied: zipp>=0.5 in /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages (from importlib-metadata>=1.4->pyppeteer>=0.0.14->requests-html) (3.9.0)
Requirement already satisfied: soupsieve>1.2 in /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages (from beautifulsoup4->bs4->requests-html) (2.3.2.post1)
Building wheels for collected packages: lxml
Building wheel for lxml (setup.py) ... error
error: subprocess-exited-with-error
× python setup.py bdist_wheel did not run successfully.
│ exit code: 1
╰─> [184 lines of output]
Building lxml version 4.9.1.
Building without Cython.
Building against libxml2 2.9.4 and libxslt 1.1.29
running bdist_wheel
running build
running build_py
creating build
creating build/lib.macosx-10.9-universal2-cpython-310
creating build/lib.macosx-10.9-universal2-cpython-310/lxml
copying src/lxml/_elementpath.py -> build/lib.macosx-10.9-universal2-cpython-310/lxml
////// I CROPPED THE MESSAGE HERE FOR SPACE REASONS //////
running build_ext
building 'lxml.etree' extension
creating build/temp.macosx-10.9-universal2-cpython-310
creating build/temp.macosx-10.9-universal2-cpython-310/src
creating build/temp.macosx-10.9-universal2-cpython-310/src/lxml
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -arch arm64 -arch x86_64 -g -DCYTHON_CLINE_IN_TRACEBACK=0 -Isrc -Isrc/lxml/includes -I/Library/Frameworks/Python.framework/Versions/3.10/include/python3.10 -c src/lxml/etree.c -o build/temp.macosx-10.9-universal2-cpython-310/src/lxml/etree.o -w -flat_namespace
In file included from src/lxml/etree.c:96:
In file included from /Library/Frameworks/Python.framework/Versions/3.10/include/python3.10/Python.h:11:
In file included from /Library/Developer/CommandLineTools/usr/lib/clang/11.0.3/include/limits.h:21:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/limits.h:63:
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/cdefs.h:807:2: error: Unsupported architecture
#error Unsupported architecture
^
In file included from src/lxml/etree.c:96:
In file included from /Library/Frameworks/Python.framework/Versions/3.10/include/python3.10/Python.h:11:
In file included from /Library/Developer/CommandLineTools/usr/lib/clang/11.0.3/include/limits.h:21:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/limits.h:64:
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/machine/limits.h:8:2: error: architecture not supported
#error architecture not supported
^
In file included from src/lxml/etree.c:96:
In file included from /Library/Frameworks/Python.framework/Versions/3.10/include/python3.10/Python.h:25:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/stdio.h:64:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/_stdio.h:71:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/_types.h:27:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types.h:33:
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/machine/_types.h:34:2: error: architecture not supported
#error architecture not supported
^
In file included from src/lxml/etree.c:96:
In file included from /Library/Frameworks/Python.framework/Versions/3.10/include/python3.10/Python.h:25:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/stdio.h:64:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/_stdio.h:71:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/_types.h:27:
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types.h:55:9: error: unknown type name '__int64_t'
typedef __int64_t __darwin_blkcnt_t; /* total blocks */
^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types.h:56:9: error: unknown type name '__int32_t'; did you mean '__int128_t'?
typedef __int32_t __darwin_blksize_t; /* preferred block size */
^
note: '__int128_t' declared here
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types.h:57:9: error: unknown type name '__int32_t'; did you mean '__int128_t'?
typedef __int32_t __darwin_dev_t; /* dev_t */
^
note: '__int128_t' declared here
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types.h:60:9: error: unknown type name '__uint32_t'; did you mean '__uint128_t'?
typedef __uint32_t __darwin_gid_t; /* [???] process and group IDs */
^
note: '__uint128_t' declared here
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types.h:61:9: error: unknown type name '__uint32_t'; did you mean '__uint128_t'?
typedef __uint32_t __darwin_id_t; /* [XSI] pid_t, uid_t, or gid_t*/
^
note: '__uint128_t' declared here
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types.h:62:9: error: unknown type name '__uint64_t'
typedef __uint64_t __darwin_ino64_t; /* [???] Used for 64 bit inodes */
^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types.h:68:9: error: unknown type name '__darwin_natural_t'
typedef __darwin_natural_t __darwin_mach_port_name_t; /* Used by mach */
^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types.h:70:9: error: unknown type name '__uint16_t'; did you mean '__uint128_t'?
typedef __uint16_t __darwin_mode_t; /* [???] Some file attributes */
^
note: '__uint128_t' declared here
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types.h:71:9: error: unknown type name '__int64_t'
typedef __int64_t __darwin_off_t; /* [???] Used for file sizes */
^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types.h:72:9: error: unknown type name '__int32_t'; did you mean '__int128_t'?
typedef __int32_t __darwin_pid_t; /* [???] process and group IDs */
^
note: '__int128_t' declared here
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types.h:73:9: error: unknown type name '__uint32_t'; did you mean '__uint128_t'?
typedef __uint32_t __darwin_sigset_t; /* [???] signal set */
^
note: '__uint128_t' declared here
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types.h:74:9: error: unknown type name '__int32_t'; did you mean '__int128_t'?
typedef __int32_t __darwin_suseconds_t; /* [???] microseconds */
^
note: '__int128_t' declared here
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types.h:75:9: error: unknown type name '__uint32_t'; did you mean '__uint128_t'?
typedef __uint32_t __darwin_uid_t; /* [???] user IDs */
^
note: '__uint128_t' declared here
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types.h:76:9: error: unknown type name '__uint32_t'; did you mean '__uint128_t'?
typedef __uint32_t __darwin_useconds_t; /* [???] microseconds */
^
note: '__uint128_t' declared here
In file included from src/lxml/etree.c:96:
In file included from /Library/Frameworks/Python.framework/Versions/3.10/include/python3.10/Python.h:25:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/stdio.h:64:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/_stdio.h:71:
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/_types.h:43:9: error: unknown type name '__uint32_t'; did you mean '__uint128_t'?
typedef __uint32_t __darwin_wctype_t;
^
note: '__uint128_t' declared here
In file included from src/lxml/etree.c:96:
In file included from /Library/Frameworks/Python.framework/Versions/3.10/include/python3.10/Python.h:25:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/stdio.h:64:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/_stdio.h:75:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types/_va_list.h:31:
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/machine/types.h:37:2: error: architecture not supported
#error architecture not supported
^
fatal error: too many errors emitted, stopping now [-ferror-limit=]
20 errors generated.
Compile failed: command '/usr/bin/clang' failed with exit code 1
creating var
creating var/folders
creating var/folders/dz
creating var/folders/dz/n8vf6x950yjdbzwftt_1vd1m0000gn
creating var/folders/dz/n8vf6x950yjdbzwftt_1vd1m0000gn/T
cc -I/usr/include/libxml2 -c /var/folders/dz/n8vf6x950yjdbzwftt_1vd1m0000gn/T/xmlXPathInit046146bm.c -o var/folders/dz/n8vf6x950yjdbzwftt_1vd1m0000gn/T/xmlXPathInit046146bm.o
cc var/folders/dz/n8vf6x950yjdbzwftt_1vd1m0000gn/T/xmlXPathInit046146bm.o -lxml2 -o a.out
error: command '/usr/bin/clang' failed with exit code 1
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for lxml
Running setup.py clean for lxml
Failed to build lxml
Installing collected packages: lxml, importlib-metadata, cssselect, beautifulsoup4, pyquery, pyppeteer, bs4, requests-html
Running setup.py install for lxml ... error
error: subprocess-exited-with-error
× Running setup.py install for lxml did not run successfully.
│ exit code: 1
╰─> [181 lines of output]
Building lxml version 4.9.1.
Building without Cython.
Building against libxml2 2.9.4 and libxslt 1.1.29
running install
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
warnings.warn(
running build
running build_py
creating build
creating build/lib.macosx-10.9-universal2-cpython-310
creating build/lib.macosx-10.9-universal2-cpython-310/lxml
////// I CROPPED THE MESSAGE HERE FOR SPACE REASONS //////
running build_ext
building 'lxml.etree' extension
creating build/temp.macosx-10.9-universal2-cpython-310
creating build/temp.macosx-10.9-universal2-cpython-310/src
creating build/temp.macosx-10.9-universal2-cpython-310/src/lxml
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -arch arm64 -arch x86_64 -g -DCYTHON_CLINE_IN_TRACEBACK=0 -Isrc -Isrc/lxml/includes -I/Library/Frameworks/Python.framework/Versions/3.10/include/python3.10 -c src/lxml/etree.c -o build/temp.macosx-10.9-universal2-cpython-310/src/lxml/etree.o -w -flat_namespace
In file included from src/lxml/etree.c:96:
In file included from /Library/Frameworks/Python.framework/Versions/3.10/include/python3.10/Python.h:11:
In file included from /Library/Developer/CommandLineTools/usr/lib/clang/11.0.3/include/limits.h:21:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/limits.h:63:
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/cdefs.h:807:2: error: Unsupported architecture
#error Unsupported architecture
^
In file included from src/lxml/etree.c:96:
In file included from /Library/Frameworks/Python.framework/Versions/3.10/include/python3.10/Python.h:11:
In file included from /Library/Developer/CommandLineTools/usr/lib/clang/11.0.3/include/limits.h:21:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/limits.h:64:
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/machine/limits.h:8:2: error: architecture not supported
#error architecture not supported
^
In file included from src/lxml/etree.c:96:
In file included from /Library/Frameworks/Python.framework/Versions/3.10/include/python3.10/Python.h:25:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/stdio.h:64:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/_stdio.h:71:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/_types.h:27:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types.h:33:
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/machine/_types.h:34:2: error: architecture not supported
#error architecture not supported
^
In file included from src/lxml/etree.c:96:
In file included from /Library/Frameworks/Python.framework/Versions/3.10/include/python3.10/Python.h:25:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/stdio.h:64:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/_stdio.h:71:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/_types.h:27:
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types.h:55:9: error: unknown type name '__int64_t'
typedef __int64_t __darwin_blkcnt_t; /* total blocks */
^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types.h:56:9: error: unknown type name '__int32_t'; did you mean '__int128_t'?
typedef __int32_t __darwin_blksize_t; /* preferred block size */
^
note: '__int128_t' declared here
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types.h:57:9: error: unknown type name '__int32_t'; did you mean '__int128_t'?
typedef __int32_t __darwin_dev_t; /* dev_t */
^
note: '__int128_t' declared here
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types.h:60:9: error: unknown type name '__uint32_t'; did you mean '__uint128_t'?
typedef __uint32_t __darwin_gid_t; /* [???] process and group IDs */
^
note: '__uint128_t' declared here
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types.h:61:9: error: unknown type name '__uint32_t'; did you mean '__uint128_t'?
typedef __uint32_t __darwin_id_t; /* [XSI] pid_t, uid_t, or gid_t*/
^
note: '__uint128_t' declared here
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types.h:62:9: error: unknown type name '__uint64_t'
typedef __uint64_t __darwin_ino64_t; /* [???] Used for 64 bit inodes */
^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types.h:68:9: error: unknown type name '__darwin_natural_t'
typedef __darwin_natural_t __darwin_mach_port_name_t; /* Used by mach */
^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types.h:70:9: error: unknown type name '__uint16_t'; did you mean '__uint128_t'?
typedef __uint16_t __darwin_mode_t; /* [???] Some file attributes */
^
note: '__uint128_t' declared here
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types.h:71:9: error: unknown type name '__int64_t'
typedef __int64_t __darwin_off_t; /* [???] Used for file sizes */
^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types.h:72:9: error: unknown type name '__int32_t'; did you mean '__int128_t'?
typedef __int32_t __darwin_pid_t; /* [???] process and group IDs */
^
note: '__int128_t' declared here
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types.h:73:9: error: unknown type name '__uint32_t'; did you mean '__uint128_t'?
typedef __uint32_t __darwin_sigset_t; /* [???] signal set */
^
note: '__uint128_t' declared here
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types.h:74:9: error: unknown type name '__int32_t'; did you mean '__int128_t'?
typedef __int32_t __darwin_suseconds_t; /* [???] microseconds */
^
note: '__int128_t' declared here
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types.h:75:9: error: unknown type name '__uint32_t'; did you mean '__uint128_t'?
typedef __uint32_t __darwin_uid_t; /* [???] user IDs */
^
note: '__uint128_t' declared here
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types.h:76:9: error: unknown type name '__uint32_t'; did you mean '__uint128_t'?
typedef __uint32_t __darwin_useconds_t; /* [???] microseconds */
^
note: '__uint128_t' declared here
In file included from src/lxml/etree.c:96:
In file included from /Library/Frameworks/Python.framework/Versions/3.10/include/python3.10/Python.h:25:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/stdio.h:64:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/_stdio.h:71:
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/_types.h:43:9: error: unknown type name '__uint32_t'; did you mean '__uint128_t'?
typedef __uint32_t __darwin_wctype_t;
^
note: '__uint128_t' declared here
In file included from src/lxml/etree.c:96:
In file included from /Library/Frameworks/Python.framework/Versions/3.10/include/python3.10/Python.h:25:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/stdio.h:64:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/_stdio.h:75:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types/_va_list.h:31:
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/machine/types.h:37:2: error: architecture not supported
#error architecture not supported
^
fatal error: too many errors emitted, stopping now [-ferror-limit=]
20 errors generated.
Compile failed: command '/usr/bin/clang' failed with exit code 1
cc -I/usr/include/libxml2 -c /var/folders/dz/n8vf6x950yjdbzwftt_1vd1m0000gn/T/xmlXPathInitq1lwalnt.c -o var/folders/dz/n8vf6x950yjdbzwftt_1vd1m0000gn/T/xmlXPathInitq1lwalnt.o
cc var/folders/dz/n8vf6x950yjdbzwftt_1vd1m0000gn/T/xmlXPathInitq1lwalnt.o -lxml2 -o a.out
error: command '/usr/bin/clang' failed with exit code 1
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure
× Encountered error while trying to install package.
╰─> lxml
note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.
Essentially it "fails building the wheel for lxml". How can I solve this issue? It is weird that everything was running fine until I tinkered with the terminal. My system is MacOS Monterey V12.5.1