0

I want to make a web page that generates a uniform random number between 0 and 99 every 10 seconds, and displays a list of the 100 most recent numbers (which are the same for everyone visiting the site). It should update live.

My design is the following:

  • A long-running Python process (e.g. using supervisord) that runs in an eternal loop, generating numbers at 10-second intervals, and writing the numbers to a file or SQL database, and pruning the old numbers since they are no longer needed.
  • Then the web server process simply reads the file and displays to the user (either on initial load, or from an Ajax call to get the most recent numbers)

I don't feel great about this solution. It's pretty heavy on file system I/O, which is not really a bottleneck or anything, but I just wonder if there's a smarter way that is still simple. If I could store the list as an in-memory data structure shared between processes, I could have one process push and pop values every 10 seconds, and then the web server processes could just read that data structure. I read a bit about Unix domain sockets, but it wasn't clear that this was a great fit to my problem

Is there a more efficient approach that is still simple?

EDIT: the approach suggested by Martijn Peters in his answer (don't generate anything until someone visits) is sensible and I am considering it too, since the website doesn't get very heavy traffic. The problem I see is with race conditions, since you then have multiple processes trying to write to the same file/DB. If the values in the file/DB are stale, we need to generate new ones, but one process might read the old values before another process has had the chance to update them. File locking as described in this question is a possibility, but many people in the answers warn about having multiple processes write to the same file.

Community
  • 1
  • 1
RexE
  • 17,085
  • 16
  • 58
  • 81

2 Answers2

0

You are overcomplicating things.

Don't generate any numbers until you have an actual request. Then see how old your last number is, generate enough numbers to cover the intervening time period, update your tables, return the result.

There is no actual need to actually generate a random number every 10 seconds here. You only need to produce the illusion that the numbers have been generated every 10 seconds, that more than suffices for your use case.

A good database will handle concurrent access for you, and most will also let you set exclusive locks. Grab a lock when you need to update the numbers. Fail to grab the lock? Something else is already updating the numbers.

Pre-generate numbers; nothing says you actually have to only generate the numbers for the past time slot. Randomize what requests pre-generate to minimize lock contention. Append the numbers to the end of the pool, so that if you accidentally run this twice all you get is double the extra random numbers, so you can wait twice as long before you need to generate more.

Most of all, generating a sequence of random numbers is cheap, so doing this during any request is hardly going to slow down your responses.

Martijn Pieters
  • 1,048,767
  • 296
  • 4,058
  • 3,343
  • The updates must be done live, updated via Ajax. He _does_ need to generate a number every 10 seconds. – Racso Jul 12 '13 at 16:24
  • Then the AJAX call can trigger the update. Until you have an actual visitor with a page that drives the AJAX calls there is no point in generating the numbers. – Martijn Pieters Jul 12 '13 at 16:25
  • You're right with that. The critical point, however, remains: if only one visitor is conected, a process will be run every 10 seconds to generate the numbers. – Racso Jul 12 '13 at 16:27
  • A note on my last comment: yet, this approach is better than the one of having an ever-running process. While you don't save database load, you do save CPU. – Racso Jul 12 '13 at 16:31
  • Thanks Martijn. I want exclusive locks. User 2 should wait while User 1's process is updating the DB. I didn't find a way to block until the lock is released, so I guess I could put the DB code inside a while loop that runs until it's able to acquire the lock. – RexE Jul 12 '13 at 16:54
0

I would pre-generate a lot of numbers (say, enough numbers for 1 week; do the math) and store them. That way, the Ajax calls would only load the next number on the list. When you are running out of numbers, pre-generate again. The process of generating and writing into DB would be only executed one in a while (e.g. once a week).

EDIT: For a full week, you would need 60480 numbers at the most. Using what Martijn Pieters recommends (only reading a new number when a visitor indeed asks for one), and depending on your specific need (as you may need to still burn the numbers even if nobody is seeing them), those numbers may last way more than the week.

Racso
  • 2,310
  • 1
  • 18
  • 23