0

I have this piece of code which web scrapes a certain site and prints out what it finds on that certain webpage.

I'm pretty new to this, how can I just collect the data from just the console, like what is seen in the picture.

inspect console

Here is the code so far, thanks for the help

import requests

url = 'url goes here'
r = requests.get(url)
print(r.text)
Nguyễn Vũ Thiên
  • 761
  • 1
  • 9
  • 19
Trainer Scrap
  • 1
  • 1
  • You can't get data directly from the console as far as I know, but whatever you want to do in the console (in JavaScript) you can also do in Python. Please, correct me if I missunderstood your question. – Y. Georgiev Dec 22 '21 at 08:42

1 Answers1

0

Here are some ways to collect the output:

  1. if the data is pretty small and well-formatted, like just 1 line for each URL, you can just copy the output from the console prints.

  2. if the data is very big, I assume this is your situation, you can write the output into files.

    import requests
    
    url = 'url goes here'

    r = requests.get(url)
    
    print(r.text)  
    
    with open('/path/to/file.txt', 'w', encoding='utf-8') as f:
    
        f.write('r.text')
  1. if you have thousands of URL, and need to write into thousand files, just add a for loop for each url and write the output to different files.

above example are using a txt file, you can also write the output into a .xml file or .html file, any format that is more convenient to re-use for you, like docx, excel, csv, json, etc.

Shawn Deng
  • 1
  • The data set is rather small, as you just said only like one line, so I think that your first option is the way to go , although I'm not quite sure how to do that. Sorry I this doesn't make any sense – Trainer Scrap Dec 22 '21 at 08:26
  • the picture in your post is the browser console, I thought you mean the IDE console output. You can try this one use selenium: https://stackoverflow.com/questions/20907180/getting-console-log-output-from-chrome-with-selenium-python-api-bindings – Shawn Deng Dec 29 '21 at 05:26