0

I need to dump the contents (text only) of a webpage every 1 minute to a text file (append it all to one text file) The webpage is a web log which is updated on the fly automatically. Could I use a short python script to accomplish this simple copy/paste automation?Or do you know of any ready made program that does the same?

I have checked the answers to this: Python: saving large web page to file and this: Dumping a dynamic web page to file? But I am new to python so I cannot build my script on the info provided

It seems that this may be a solution: https://www.seleniumhq.org/projects/webdriver/

But could you give me a small working example

Mariyah
  • 13
  • 5

1 Answers1

0

You could absolutely use Python for this task, but without further details of the site you're trying to save, it's impossible to say how difficult it will be.

A very simple example could look like this (using the popular requests library).

import requests

response = requests.get('http://example.com')

with open('output.txt', 'w+') as handle:
    handle.write(response.text)

Of course, Python is overkill for such a simple example. You could accomplish this using curl:

curl http://example.com >> output.txt

Then all you need to do is set up some kind of task runner (cron maybe) to run this every minute or as often as you'd like.

If the website has JavaScript that is necessary for page rendering, you'll need to use a more advanced solution, but like I said you haven't given enough details for a more detailed answer.

Tim Shaffer
  • 166
  • 2
  • 7