How to stream a (big) log file?

Question

When automation testing is running I cannot connect to the testing computer and check the log file to check the progress of the testing because I would interrupt the testing. I can map the hdd and check the file this way but I want to add new feature to my sinatra application.

The application runs the tests and displays the test results so I want to add realtime stream of the log file via sintra. The log file could be even 2MB big so I guess it won't be a good idea to send the whole file every time an update to the log file was made although the server client communication will be done in 99% over LAN only. I would also like to have the latest last line from the log file at the top of a web browser.

Could someone suggest how to do that?

I can think of an ajax call made on regular basis that would pass to sinatra a number of the line that was received as a last one. And sinatra would return any update if available.

update

Windows 7 64 bit
ruby 1.9.3p194 (2012-04-20) [i386-mingw32]
sinatra (1.3.3)
sinatra-contrib (1.3.1)
sinatra-reloader (1.0)

You basically need a pusher agent to transmit the new lines added to the log file to your Sinatra application so it can be re-displayed to the various clients, right? What database or storage options do you have? — tadman, Dec 14 '12 at 00:43
@tadman: Yes, you are right I want a pusher. Not sure if I understood your question but I don't use any database. — Radek, Dec 14 '12 at 00:58

the Tin Man · Answer 1 · 2012-12-14T01:31:53.297

You don't say what type of OS your testing system uses, but if it's Linux or Mac OS you're ready to go. If not, and it's Windows, I'd really recommend installing a telnetd or ssh server, and a tail-type app.

SSH and/or Telnet are a lot more lightweight because they're basically just sending text, so they'll impact your testing system less than trying to stream a file via HTTP, especially with the solution you mentioned. Just open a session, tail -f the file, then start the test.

To implement a solution using Sinatra, I'd start with a little piece of code like:

#!/bin/env ruby

filepath = ARGV.shift
start_line, num_lines = ARGV.map(&:to_i)

File.foreach(filepath) do |li|
  case 
  when $. < start_line
    next
  when (start_line .. (start_line + num_lines)) === $.
    puts li 
  when $. > (start_line + num_lines)
    break 
  end
end

Save that to the disk as display_file_block.rb, and call it with the parameters:

path/to/file start_line lines_to_display

Where:

path/to/file is obvious.
start_line is the starting line in the file to display.
lines_to_display is the number of lines to display.

Using those you can open a file to display, send a number of lines, starting at an offset.

In Sinatra, set up a request handler for a GET:

get '/tail' do
  path = params['path']
  start = params['start']
  count = params['count']
  `/path/to/display_file_block.rb #{ path } #{ start } #{ count }`
end

You'll possibly want to set the content-type for the response to 'text/plain'. The Sinatra site can show you how to do that.

Hmmm, I didn't think about that solution especially as the sintra sits on Windows7. I guess it might be the best solution because of the performance. But I still want to know how to implement "the pusher". — Radek, Dec 14 '12 at 01:00
If you're measuring the performance of your test machine, you really don't want to push, or even pull for that matter. Either will steal CPU cycles from the machine and artificially change the metrics. You could possibly gather some metrics for the pull/push when the tests are not running, then adjust your values to compensate, but again, that's not the best way to go. Using the code in my answer, you can control when you want to update, and you could even jump around in the file. When there is no request for lines, there will be no additional load on the host to affect the results. — the Tin Man, Dec 14 '12 at 03:24

score 1 · Answer 2 · edited May 23 '17 at 12:30

Here is an example using "push", i.e. WebSockets:

Minimum Websocket Nodejs Tail Example

In my opinion AJAX polling is a duct-tape type of solution to this kind of problem. WebSockets is the way to go, and done correctly will have very little impact on system performance.

By googling "websockets sinatra" you can search for similar solutions for Sinatra, like the first hit for example:

https://github.com/simulacre/sinatra-websocket

How to stream a (big) log file?

update

2 Answers2