1

i am running one instance of elastic and one of logstash in parallel on the same computer.

when trying to load a file into elastic, using logstash that is running the config file below, i get the follwing output msgs on elastic and no file is loaded (when input is configured to be stdin everything seems to be working just fine)

any ideas?

" [2014-06-17 22:42:24,748][INFO ][cluster.service ] [Masked Marvel] removed {[logstash- Eitan-PC-5928-2010][Ql5fyvEGQyO96R9NIeP32g][Eitan-PC][inet[Eitan-PC/10.0.0.5:9301]]{client=true, data=false},}, reason: zen-disco-node_failed([logstash-Eitan-PC-5928-2010][Ql5fyvEGQyO96R9NIeP32g][Eitan-PC][inet[Eitan-PC/10.0.0.5:9301]]{client=true, data=false}), reason transport disconnected (with verified connect)

[2014-06-17 22:43:00,686][INFO ][cluster.service ] [Masked Marvel] added {[logstash-Eitan-PC-5292-4014][m0Tg-fcmTHW9aP6zHeUqTA][Eitan-PC][inet[/10.0.0.5:9301]]{client=true, data=false},}, reason: zen-disco-receive(join from node[[logstash-Eitan-PC-5292-4014][m0Tg-fcmTHW9aP6zHeUqTA][Eitan-PC][inet[/10.0.0.5:9301]]{client=true, data=false}]) "

config file: 

    input { 
            file {
                path => "c:\testLog.txt"
            }
        } 


    output {
        elasticsearch { host => localhost  
                index=> amat1
                 }

}
Eitan Vesely
  • 125
  • 3
  • 16

1 Answers1

0

When you use "elasticsearch" as your output http://logstash.net/docs/1.4.1/outputs/elasticsearch as opposed to "elasticsearch_http" http://logstash.net/docs/1.4.1/outputs/elasticsearch_http you are going to want to set "protocol".

The reason is that it can have 3 different values, "node", "http" or "transport" with different behavior for each and the default selection is not well documented.

From the look of your log files it appears it's trying to use "node" protocol as I see connection attempts on port 9301 which indicates (along with other log entries) that logstash is trying to join the cluster as a node. This can fail for any number of reasons including mismatch on the cluster name.

I'd suggest setting protocol to "http" - that change has fixed similar issues before.

See also:

http://logstash.net/docs/1.4.1/outputs/elasticsearch#cluster http://logstash.net/docs/1.4.1/outputs/elasticsearch#protocol

EDIT:

A few other issues I see in your config -

  • Your host and index should be strings, which in a logstash config file should be wrapped with double quotes, "localhost" and "amat1". No quotes may work but they recommend you use quotes.

    http://logstash.net/docs/1.4.1/configuration#string

  • If you don't use "http" as the protocol or don't use "elasticsearch_http" as the output you should set cluster equal to your ES cluster name (as it will be trying to become a node of the cluster).

  • You should set start_position under file in input to "beginning". Otherwise it will default to reading from the end of the file and you won't see any data. This a particular problem with Windows right now as the other way of tracking position within a file, sincedb, is broken on Windows:

    https://logstash.jira.com/browse/LOGSTASH-1587

    http://logstash.net/docs/1.4.1/inputs/file#start_position

  • You should change your path to your log file to this: "C:/testLog.txt". Logstash prefers forward slashes and upper case drive letters under Windows.

    https://logstash.jira.com/browse/LOGSTASH-430

John Petrone
  • 26,943
  • 6
  • 63
  • 68
  • Thanks John, i've tried it with no success :-(. tried elasticsearch_http {} and elasticsearch{protocol => "http"} yet neither worked... any other idea? – Eitan Vesely Jun 19 '14 at 08:09
  • By the way, when using elasticsearch_http{ } i get an error msg saying i must provide a "tamplate", yet in the tamplate docs. it says that this is an optional configuration... – Eitan Vesely Jun 19 '14 at 08:21
  • you are getting the exact same error message when using both of these setups as you did the first time? – John Petrone Jun 19 '14 at 14:19
  • Hi, when this is the config file: http://justpaste.it/fwze this is the output msg i get on logstash: http://justpaste.it/fwzd (nothing in elastic). – Eitan Vesely Jun 19 '14 at 17:09
  • and when this is the config file: http://justpaste.it/fwzh i get no output on elastic, in this is the only output on logstash: http://justpaste.it/fwzi. Thanks again for trying to help :-) – Eitan Vesely Jun 19 '14 at 17:13
  • read the edits I've made to the answer, there are a number of additional changes you should make – John Petrone Jun 19 '14 at 17:35
  • Thanks John. this time i got it! it was the drive letter, it wasnt capital C... :-) (the / vs. \ depends on the OS you are using. i guess windows likes the \ while Linux prefers /) – Eitan Vesely Jun 20 '14 at 20:47
  • DAMN. now im realy confused. it worked once! now im back to square 1. doesnt work again.. using the exact same config file. this is the it: http://justpaste.it/fxim . any other hint? – Eitan Vesely Jun 20 '14 at 21:20
  • can you expand a bit on "doesn't work again" - errors, no data in Elasticsearch? – John Petrone Jun 20 '14 at 21:59
  • Hi, "doesnt work again" means that out of tens (or houndred) of trys, i only got the file loaded into elastic twice. the strange thing is that i've made no changed to the config file what so ever... no output or error msg on both elastic or logstash – Eitan Vesely Jun 21 '14 at 08:26
  • Ok, i think i got it this time. the problem was that i was uploading the same log file over and over again. logstash remembers where it stoped reading the file the last time it was open so it would only update elastic with changes. since i deleted the data from elastic, and the log file i loaded didnt change from the last time it was loaded nothing happend. Thanks for trying to help! – Eitan Vesely Jun 21 '14 at 20:20