How to I automate creating a script with an input parameter?

Question

I have a folder of similar-looking scripts which scrape google alerts from their RSS feeds.

All the files are exactly the same except the variable uniqueurl at the end of url

url = 'https://www.google.co.in/alerts/feeds/*uniqueurl*'
resp = requests.get(url)
soup = BeautifulSoup(resp.text, 'html.parser')

output = []
for entry in soup.find_all('entry'):

    item = {
        'Title': entry.find('title', {'type': 'html'}).text,
        'Pubdate': entry.find('published').text,
        'Content': entry.find('content').text,
        'Link': entry.find('link')['href']
    }

    output.append(item)

df = pd.DataFrame(output)
df.to_csv('google_alert.csv',index=False)

How do I run a command like python create.py uniqueurl which generates the above file with just the url variable updated with what is passed in the command?

Do you really need a separate script for each URL? Why not just use an argument provided at runtime? — matszwecja, Feb 28 '22 at 12:20
Because the results returned are added over existing CSVs and there are various categories that I've separated them for NLP later. — Parzival, Feb 28 '22 at 12:25
That still does not mean you cannot use a runtime argument and decide a category based on that during runtime. — matszwecja, Feb 28 '22 at 12:27
How can I add a list of URLs and set their destinations too? — Parzival, Feb 28 '22 at 12:33

score 0 · Answer 1 · answered Feb 28 '22 at 13:26

Use sys.argv to capture any variables you want to get at runtime each time the script is executed.

import sys

uniqueUrl = sys.argv[1]

url = f'https://www.google.co.in/alerts/feeds/{uniqueUrl}'
print(uniqueUrl)
destination = sys.argv[2]
print(destination)

resp = requests.get(url)

Then when running a script you can pass a value that will be assigned to those variables like so: python script.py uniqueUrl Category - that way you don't need to store multiple scripts and regenerate them every time you want to make some small difference in the code.

How to I automate creating a script with an input parameter?

1 Answers1