How can I read a list of urls from a text file and save all the responses into one text file?

Question

I have a list of urls inside a text file. I would like to feed the urls from the text file into my script of code instead of pasting each url into the script but unsure how to do so. I also would like to save all the responses into one text file. Thank you

import requests
from datetime import datetime

def run():

    url = "https://en.wikipedia.org/wiki/NBA"

    payload = {}
    headers= {}

    response = requests.request("POST", url, headers=headers, data = payload)

    print(response.text)

startTime = datetime.now()
run()
print(datetime.now() - startTime)

File: Wiki.text

https://en.wikipedia.org/wiki/NBA
https://en.wikipedia.org/wiki/NFL
https://en.wikipedia.org/wiki/MLB
https://en.wikipedia.org/wiki/NHL

Does this answer your question? [Copying from one text file to another using Python](https://stackoverflow.com/questions/15343743/copying-from-one-text-file-to-another-using-python) — MendelG, Aug 17 '20 at 23:17
Where are you stuck on? you need to open the file, loop through the links and write the output to another file — MendelG, Aug 17 '20 at 23:20
@MendelG I am trying to print the responses of the all urls into a text file. Not copy links. — techsandspecs, Aug 17 '20 at 23:21

score 1 · Answer 1 · edited Aug 18 '20 at 00:16

1

you can do something like this:

import requests
list_of_pages = open('list_of_links.txt', 'r+')
save = open('output.txt', 'a+')
save.truncate(0)
for page_link in list_of_pages:
    url = page_link
    payload = {}
    headers = {}
    response = requests.request("POST", url, headers=headers, data=payload)
    save.write(str(response) + "\n")

edited Aug 18 '20 at 00:16

hedy

1,160
5
23

answered Aug 17 '20 at 23:54

icecube

111
2
16

3

output in the file: – icecube Aug 17 '20 at 23:59

F.M · Accepted Answer · 2020-08-18T00:41:27.277

Try this, it returns the code of each page in Wiki.txt and also the time it took to get it:

import requests
from datetime import datetime
def run():
    links_file = open('Wiki.txt', 'r')
    outputs_file = open('outputs.txt', 'a')
    for link in links_file.readlines():  # run for ever line in file
            payload = {}
            headers = {}

            response = requests.request("POST", link, headers=headers, data=payload)

            outputs_file.write(response.text + '\n')
    links_file.close()
    outputs_file.close()

startTime = datetime.now()
run()
print(datetime.now() - startTime)

Basically all you needed to do was add a loop that runs for every line in the file, and add the resulting code to another text file.

How can I read a list of urls from a text file and save all the responses into one text file?

2 Answers2