logstash : file input is not working

Question

I am trying to run sample eg. using logstash-1.4.2 in CDH 4.4. Whenever I use file input instead of stdin, the window freezes at the following message:

Using milestone 2 plugin 'file'. This plugin should be stable but if you see strange behavior, please let us know! For more information..... My code looks like this:

input {
  file {
    path => "/tmp/access_log"
    start_position => "beginning"
  }
}

filter {
  if [path] =~ "access" {
    mutate { replace => { "type" => "apache_access" } }
    grok {
      match => { "message" => "%{COMBINEDAPACHELOG}" }
    }
  }
  date {
    match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
  }
}

output {
  file{
path =>"/logs/output_log"
}
}

Command- bin/logstash -f logstash-apache.conf

I have tried deleting all my previous sincedb files in the $HOME. directory and re-run logstash, but that doesn't seem to work either. Am I missing something?

Not certain what you are expecting, but you should not see anything else in the window with this config - your output goes to a file, there is no output directed to stdout. Is anything showing up in your output_log? — John Petrone, Jul 03 '14 at 00:59
It doesn't show anything in output_log .I tried with stdout as well bt it didn't work as well. — ashwini, Jul 03 '14 at 04:49
Can you add the first couple of lines from "/tmp/access_log" to the question? — John Petrone, Jul 03 '14 at 04:57
71.141.244.242 - kurt [18/May/2011:01:48:10 -0700] "GET /admin HTTP/1.1" 301 566 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3" 134.39.72.245 - - [18/May/2011:12:40:18 -0700] "GET /favicon.ico HTTP/1.1" 200 1189 "-" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; InfoPath.2; .NET4.0C; .NET4.0E)" — ashwini, Jul 03 '14 at 05:03
How long are you waiting to let it write to both the stdout and the file? Logstash takes a while to boot properly and it's pretty silent when it is. Also, if the file is small it may not properly flush to your output until logstash terminates. — John Petrone, Jul 03 '14 at 05:31
If you remove the filer do you still see the same behavior? If so, check if you have write access to the output file. — rexposadas, Jul 03 '14 at 14:01
when i used stdin{} then it ran filter and gave the correct output on console - stdout{}.but didn't work with file input. — ashwini, Jul 04 '14 at 06:38

kitenco · Answer 1 · 2014-11-14T12:18:34.920

8

if you have just one line in your input file, you should add an empty line at the end! that should work!

edited: AND if you are on a windows machine, you need to write the absolute path like

"c:/dev/access-log.txt"

and take care of just using one / instead of // after the c:

edited Nov 14 '14 at 12:18

answered Nov 14 '14 at 12:13

kitenco

81
1
3

Really it worked with empty line at the end. I do not believe:) – Nitin Jangid Jan 08 '19 at 09:52

score 2 · Answer 2 · edited May 23 '17 at 12:16

I got stuck because logstash tracks which logs it has already read: https://stackoverflow.com/a/24034718/268907

Remember that this option only modifies “first contact” situations where a file is new and not seen before. If a file has already been seen before, this option has no effect. Otherwise you have to set your sincedb_path to /dev/null .

Set sincedb_path to /dev/null and you will prevent it from tracking the position in the file that it last read.

score 1 · Accepted Answer · answered Jul 06 '14 at 22:21

1

Are you running with root permissions? It looks like /logs/output_log needs root permission to be written to.

I tried your configuration locally with logstash 1.4.1 (and sudo) and it seems to be working fine.

answered Jul 06 '14 at 22:21

Vinay

333
3
8

score 0 · Answer 4 · edited Dec 17 '14 at 15:15

0

Could you try the below one. It worked for me.

path => "/tmp/access_log/*"

instead of

path => "/tmp/access_log"

edited Dec 17 '14 at 15:15

BoJack Horseman

4,406
13
38
70

answered Sep 09 '14 at 09:09

user3045232

67
4

tried that,didn't work.Any problem with plugin?Is File plugin already there or Do I need to install the same? – ashwini Sep 09 '14 at 12:42
You can apply the above one and delete the existing sincedb file. Are you sure that command line was freezed , could you try type something in the command line. To identify the real problem may be remove your filter and try the above solution. For debug add the stdout in the output. So you will know whether your input is processed or not. stdout { codec => rubydebug } – user3045232 Sep 09 '14 at 13:13

logstash : file input is not working

4 Answers4