I have a selenium script that downloads a series of CSV files , then goes into the downloads folder, reads in the CSVs as DataFrames (and deletes them as it goes to avoid mess) and does some analysis on them.
Locally this all works fine, however I plan to deploy this to an environment and don't know how to configure the scraper downloads.
In this question: Selenium Webdriver in Python - files download directory change in Chrome preferences
the second answer works for me to change my download folder to a downloads directory within my venv in pycharm. The only issue is I have to use the full path, i.e. C:\Users\user\PycharmProjects\Scraping_Project\venv\Downloads\
I am concerned that deploying to an environment this will not work. Whenever I try to just use the relative path (venv\Downloads) it does not work.
My questions then are:
- How to get this to work with the relative path
- Is this necessary on an environment like AWS Lambda / Elastic beanstalk
