Caching online prices fetched via API unless they change

Question

I'm currently working on a site that makes several calls to big name online sellers like eBay and Amazon to fetch prices for certain items. The issue is, currently it takes a few seconds (as far as I can tell, this time is from making the calls) to load the results, which I'd like to be more instant (~10 seconds is too much in my opinion).

I've already cached other information that I need to fetch, but that information is static. Is there a way that I can cache the prices but update them only when needed? The code is in Python and I store info in a mySQL database.

I was thinking of somehow using chron or something along that lines to update it every so often, but it would be nice if there was a simpler and less intense approach to this problem.

Thanks!

Answer 1

You can use memcache to do the caching. The first request will be slow, but the remaining requests should be instant. You'll want a cron job to keep it up to date though. More caching info here: Good examples of python-memcache (memcached) being used in Python?

Answer 2

Have you thought about displaying the cached data, then updating the prices via an ajax callback? You could notify the user if the price changed with a SO type notification bar or similar.

This way the users get results immediately, and updated prices when they are available.

Edit

You can use jquery:

Assume you have a script names getPrices.php that returns a json array of the id of the item, and it's price.

No error handling etc here, just to give you an idea

 My necklace: <div id='1'> $123.50 </div><br>
 My bracelet: <div id='1'> $13.50 </div><br>
 ...

 <script>
 $(document).ready(function() {
   $.ajax({ url: "getPrices.php", context: document.body, success: function(data){
      for (var price in data)
      {
          $(price.id).html(price.price);
      }
    }}));
 </script>

Answer 3

You need to handle the following in your application:

1. get the price
2. determine if the price has changed
3. cache the price information

For step 1, you need to consider how often the item prices will change. I would go with your instinct to set a Cron job for a process which will check for new prices on the items (or on sets of items) at set intervals. This is trivial at small scale, but when you have tens of thousands of items the architecture of this system will become very important.

To avoid delays in page load, try to get the information ahead of time as much as possible. I don't know your use case, but it'd be good to prefetch the information as much as possible and avoid having the user wait for an asynchronous JavaScript call to complete.

For step 2, if it's a new item or if the price has changed, update the information in your caching system.

For step 3, you need to determine the best caching system for your needs. As others have suggested, memcached is a possible solution. There are a variety of "NoSQL" databases you could check out, or even cache the results in MySQL.

Answer 4

How are you getting the price? If you are scrapping the data from the normal HTML page using a tool such as BeautifulSoup, that may be slowing down the round-trip time. In this case, it might help to compute a fast checksum (such as MD5) from the page to see if it has changed, before parsing it. If you are using a API which gives a short XML version of the price, this is probably not an issue.