Fetch data in background - maybe using threads

Question

I want to use Python 3 to fetch some data from a website periodically. At the same time I'd like to process the data from the last request.

While there is no response (or if an error occurs) I'd like to work on with the previous (valid) value.

Here is an example:

import threading
import urllib.request
import urllib.error

import time

start_time = time.time()

req = urllib.request.Request("http://fake-response.appspot.com/")

text = "Before response"


def fetch():
    try:
        response = urllib.request.urlopen(req)
        return response.read().decode('latin1')
    except (urllib.error.HTTPError, urllib.error.URLError) as e:
        print(e)

print("Initial text is \"%s\"" % text)

text = threading.Thread(target=fetch)

while time.time() - start_time < 15:
    print("%s - text: \"%s\"" % (time.strftime("%H:%M:%S"), text))
    # This should print '<TIME> - text: "Before response" for ten seconds
    # After that, '<TIME> - text: {"response":"This request has finished sleeping for 10 seconds"}' should be displayed

    time.sleep(0.5)

I found how to get the return value from a thread in python?, but I don't know if this applies well.

I would prefer a solution that is easy but doesn't involve additional libraries (I try to keep the memory footprint low as this is only a sideline of my project).

With 3.4+, I would try using the new asyncio module to periodically fetch new data. I would have to reread the doc, though, to suggest code. — Terry Jan Reedy, Nov 17 '14 at 23:28
@TerryJanReedy good point but I use 3.2 and I am a bit conservative with using another module... — speendo, Nov 18 '14 at 10:12

speendo · Accepted Answer · 2014-11-18T10:15:05.160

Don't know if this is good practise, but I found a solution with a more elaborate class based design:

import threading
import urllib.request
import urllib.error

import time

start_time = time.time()

req = urllib.request.Request("http://fake-response.appspot.com/")


class Fetcher(threading.Thread):
    def __init__(self):
        self.text = "No response yet!"
        super().__init__()

    def run(self):
        try:
            response = urllib.request.urlopen(req)
            self.text = response.read().decode('latin1')
        except (urllib.error.HTTPError, urllib.error.URLError) as e:
            print(e)

fetcher = Fetcher()

print("Initial text is \"%s\"" % fetcher.text)

fetcher.start()

text = fetcher.text

while time.time() - start_time < 15:
    print("%s - text: \"%s\"" % (time.strftime("%H:%M:%S"), fetcher.text))
    # This should print '<TIME> - text: "No response yet!"' for ten seconds
    # After that, '<TIME> - text: {"response":"This request has finished sleeping for 10 seconds"}' should be displayed

    time.sleep(0.5)

Fetch data in background - maybe using threads

1 Answers1