0

I've written a web application that is intended to be used to view large log files. Each log file can be upwards of 500MB and will contain 100K lines or less per file. I've used supervisord to manage starting 8 instances of tornado and nginx to redirect requests to each of the 8 tornado instances. Upon deployment I've noticed that requests were locking up and there was a ton of memory usage.

My current implementation is simply reading each line from the file one-by-one and sending it over a websocket. I chose to communicate over a websocket since it allowed me to show progress to the user. This works well when there are only one or two users, but once more users begin submitting requests to load files the server bogs down.

I'm hoping someone can advise me on the best way to send a large file to a client for viewing in their web browser without blocking future requests. There can be anywhere from 30 - 50 users simultaneously loading log files. I've read this solution and it seems to be what I need, but I don't know if this is the best solution.

Community
  • 1
  • 1
Anti-Commander
  • 45
  • 1
  • 6

1 Answers1

0

Mentioned Generate large file and send it IMHO is the best (perhaps the only non-blocking without going into internals) way to send large data.

On the other hand, reading file might be implemented as fully non-blocking way, it's a curio, by running separate process cat and use PipeIOStream. Of course reading this way is not the best solution, but it is becoming handy when implementing tail -f.

Different approach

But personally I would use nginx for this task, since you want to serve large file and that's all (no tail -f). Progress could be implemented in js. Useful resources:

Community
  • 1
  • 1
kwarunek
  • 12,141
  • 4
  • 43
  • 48