Trouble importing libraries

Question

I am having issues when I try to import requests and beautifulsoup into my script. I have created a virtual environment in this path: C:\Users\eldet\AppData\Local\Programs\Python\Python311\venv and have installed the two libraries here: C:\Users\eldet\AppData\Local\Programs\Python\Python311\venv\Lib\site-packages while my script is here: C:\Users\eldet\AppData\Local\Programs\Python\Python311\venv\Scripts. The IDE I am using is VSCode. As far as I have understood (I’m new to Python), the libraries should import, since they are in the same virtual environment. The Python interpreter is in the sys.path: C:\Users\eldet\AppData\Local\Programs\Python\Python311, the requests version is 2.31.0, bs4 is 0.0.1 and beautigulsoup4, 4.12.2. When debugging, I get the Module not found error.

This is the script:

def main():
    """This is the main function of the program.

    It imports the `requests` and `bs4` modules and then uses them to scrape data from Google Maps.

    The `get_restaurants()` function is used to get a list of restaurants from Google Maps.
    The `print()` function is used to print the list of restaurants.
    """

    from bs4 import BeautifulSoup
    import requests

    def get_restaurants(query):
        """Gets a list of restaurants from Google Maps.

        Args:
            query: The query to search for.

        Returns:
            A list of restaurant objects. Each restaurant object has the following properties:
                name: The name of the restaurant.
                address: The address of the restaurant.
                phone_number: The phone number of the restaurant.
                website: The website of the restaurant.
                rating: The rating of the restaurant.
                reviews: The number of reviews for the restaurant.
        """

        url = "https://www.google.com/maps/search/{}/@37.9716578,23.7168783,12z".format(query)
        response = requests.get(url)
        soup = BeautifulSoup(response.content, "html.parser")

        restaurants = []
        for restaurant in soup.find_all("div", class_="section-result"):
            try:
                name = restaurant.find("div", class_="section-result__title").text.strip()
            except AttributeError:
                name = None
            try:
                address = restaurant.find("div", class_="section-result__address").text.strip()
            except AttributeError:
                address = None
            try:
                phone_number = restaurant.find("div", class_="section-result__phone").text.strip()
            except AttributeError:
                phone_number = None
            try:
                website = restaurant.find("div", class_="section-result__website").text.strip()
            except AttributeError:
                website = None
            try:
                rating = restaurant.find("div", class_="section-result__rating").text.strip()
            except AttributeError:
                rating = None
            try:
                reviews = restaurant.find("div", class_="section-result__reviews").text.strip()
            except AttributeError:
                reviews = None

            restaurants.append({
                "name": name,
                "address": address,
                "phone_number": phone_number,
                "website": website,
                "rating": rating,
                "reviews": reviews,
            })

        return restaurants

    query = "Mexican restaurants in Athens, Greece"
    restaurants = get_restaurants(query)

    for restaurant in restaurants:
        print(restaurant)

if __name__ == "__main__":
    main()

I have uninstalled and reinstalled the libraries as well as started all over with the installation of Python.
I have set the path to include ther intrpreter running the following:

set PATH=%PATH%;C:\Python311.

I have made sure that the directory is in the path by running:

echo %PATH%

Paths found:

C:\Users\eldet\AppData\Local\Programs\Python\Python311\venv\Lib\site-packages>echo %PATH%
C:\Users\eldet\AppData\Local\Programs\Python\Python311\venv\Scripts;C:\Users\eldet\AppData\Local\Programs\Python\Python311\Scripts\;C:\Users\eldet\AppData\Local\Programs\Python\Python311\;C:\Users\eldet\mongosh-1.6.0-win32-x64\mongosh-1.6.0-win32-x64\bin;C:\Python310\Scripts\;C:\Python310\;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\WINDOWS\System32\OpenSSH\;C:\Program Files\Docker\Docker\resources\bin;C:\ProgramData\DockerDesktop\version-bin;C:\Program Files\nodejs\;C:\ProgramData\chocolatey\bin;C:\Program Files (x86)\Google\Cloud SDK\google-cloud-sdk\bin;C:\Program Files\PuTTY\;C:\Users\eldet\AppData\Local\Programs\Python\Python311\Scripts\pip.exe;C:\users\eldet\appdata\local\programs\python\python311\lib\site-packages;C:\Users\eldet\AppData\Local\Microsoft\WindowsApps;C:\Users\eldet\AppData\Roaming\npm;C:\Users\eldet\AppData\Local\Google\Cloud SDK\google-cloud-sdk\bin;C:\Program Files (x86)\MongoDB Atlas CLI\;C:\Users\eldet\AppData\Local\Programs\Microsoft VS Code\bin;C:\Users\eldet\Documents\Selenium Scraper

I have activated the virtual environment and made surre that all installations are within it:

(venv) C:\Users\eldet\AppData\Local\Programs\Python\Python311\venv\Lib\site-packages

This is the traceback:

Traceback (most recent call last):
  File "c:\Users\eldet\AppData\Local\Programs\Python\Python311\venv\Scripts\google_maps_scraper.py", line 78, in <module>
    main()
  File "c:\Users\eldet\AppData\Local\Programs\Python\Python311\venv\Scripts\google_maps_scraper.py", line 10, in main
    import requests
ModuleNotFoundError: No module named 'requests'

Last but not least, I have restarted my computer.

So, this is how things stand at this moment. I don't understand:

(maps_scraper) C:\Users\eldet\maps_scraper\Lib\site-packages>dir
 Volume in drive C is Windows-SSD
 Volume Serial Number is D2DD-4186

 Directory of C:\Users\eldet\maps_scraper\Lib\site-packages

25/06/2023  09:37    <DIR>          .
22/06/2023  08:32    <DIR>          ..
22/06/2023  08:36    <DIR>          beautifulsoup4-4.12.2.dist-info
22/06/2023  08:36    <DIR>          bs4
24/06/2023  12:27    <DIR>          bs4-0.0.1.dist-info
22/06/2023  08:34    <DIR>          certifi
22/06/2023  08:34    <DIR>          certifi-2023.5.7.dist-info
24/06/2023  22:33    <DIR>          cffi
24/06/2023  22:33    <DIR>          cffi-1.15.1.dist-info
22/06/2023  08:34    <DIR>          charset_normalizer
22/06/2023  08:34    <DIR>          charset_normalizer-3.1.0.dist-info
22/06/2023  08:32               151 distutils-precedence.pth
24/06/2023  22:33    <DIR>          file
24/06/2023  22:33    <DIR>          file-0.3.0.dist-info
24/06/2023  21:51             2,596 google_maps_scraper.py
22/06/2023  08:34    <DIR>          idna
22/06/2023  08:34    <DIR>          idna-3.4.dist-info
22/06/2023  08:32    <DIR>          pip
22/06/2023  08:32    <DIR>          pip-23.1.2.dist-info
22/06/2023  08:32    <DIR>          pkg_resources
24/06/2023  22:33    <DIR>          pycparser
24/06/2023  22:33    <DIR>          pycparser-2.21.dist-info
24/06/2023  23:14    <DIR>          pyflakes
24/06/2023  23:14    <DIR>          pyflakes-3.0.1.dist-info
24/06/2023  12:26    <DIR>          requests
24/06/2023  12:26    <DIR>          requests-2.31.0.dist-info
22/06/2023  08:32    <DIR>          setuptools
22/06/2023  08:32    <DIR>          setuptools-65.5.0.dist-info
25/06/2023  09:52               311 set_pythonpath.py
22/06/2023  08:36    <DIR>          soupsieve
22/06/2023  08:36    <DIR>          soupsieve-2.4.1.dist-info
22/06/2023  08:34    <DIR>          urllib3
22/06/2023  08:34    <DIR>          urllib3-2.0.3.dist-info
24/06/2023  22:33           181,760 _cffi_backend.cp311-win_amd64.pyd
22/06/2023  08:32    <DIR>          _distutils_hack
24/06/2023  22:07    <DIR>          __pycache__
               4 File(s)        184,818 bytes
              32 Dir(s)  844,945,043,456 bytes free

score 2 · Answer 1 · answered Jun 21 '23 at 15:05

2

You have created your app within a virtual environment and installed the relevant modules (here: requests, bs4) within that environment.

You'll also have to run the script from within the activated virtual environment, because the modules are not present in your global python packages. (unless you install them there or add to pythonpath, which defeats the purpose of your virtual environment).

The VSCode documentation on virtual environments says you'll can select the correct interpreter (the python of your virtual environment, that is) by using Python: Select Interpreter or Ctrl+Shift+P to bring up the available/defined options.

answered Jun 21 '23 at 15:05

srn

614
3
15

Thanks! I have been running the script from within the virtual environment and have correctly selected the interpreter. – Yannis Volos Jun 21 '23 at 17:27
Here's what I have done, but to no avail: I have left the python interpreter in C:\Users\eldet\AppData\Local\Programs\Python\Python311 and have deleted venv to get out of the virtual environment? I have moved my script to C:\Users\eldet\Documents\Python Scripts\Web Scraping I have created a new virtual environment here: C:\Users\eldet\Documents\python_virtual_environments and have installed the libraries. I have activated the virtual environment and then I have run the script but I still get a ModuleNotFoundError. – Yannis Volos Jun 21 '23 at 17:31
Try this. Activate your virtual environment, then find out your pythonpath from within that, using the answers here: https://stackoverflow.com/questions/1489599/how-do-i-find-out-my-pythonpath-using-python -- then confirm the modules are actually in there. (You could also temporarily install bs4/requests for your system-wide python and try if your script runs there). – srn Jun 21 '23 at 20:24

Trouble importing libraries

1 Answers1