9

When using Logstash and Elasticsearch together, fields with .raw are appended for analyzed fields, so that when querying Elasticsearch with tools like Kibana, it's possible to use the field's value as-is without per-word splitting and what not.

I built a new installation of the ELK stack with the latest greatest versions of everything, and noticed my .raw fields are no longer being created as they were on older versions of the stack. There are a lot of folks posting solutions of creating templates on Elasticsearch, but I haven't been able to find much information as to why this fixes things. In an effort to better understand the broader problem, I ask this specific question:

Where do the .raw fields come from?

I had assumed that Logstash was populating Elasticsearch with strings as-analyzed and strings as-raw when it inserted documents, but considering the fact that the fix lies in Elasticsearch templates, I question whether or not my assumption is correct.

Community
  • 1
  • 1
Brad
  • 159,648
  • 54
  • 349
  • 530

1 Answers1

8

You're correct in your assumption that the .raw fields are the result of a dynamic template for string fields contained in the default index template that Logstash creates IF manage_template: true (which it is by default).

The default template that Logstash creates (as of 2.1) can be seen here. As you can see on line 26, all string fields (except the message one) have a not_analyzed .raw sub-field created.

However, the template hasn't changed in the latest Logstash versions as can be seen in the template.json change history, so either something else must be wrong with your install or you've changed your Logstash config to use your own index template (without .raw fields) instead.

If you run curl -XGET localhost:9200/_template/logstash* you should see the template that Logstash has created.

Val
  • 207,596
  • 13
  • 358
  • 360
  • Thank you. Unfortunately, no template is created in Elasticsearch at all, and definitely not one named `logstash`. I tried specifying a local one in my Logstash config, and that hasn't worked either. But, your information gets me closer to the problem and clarifies things. I'll keep digging! – Brad Nov 29 '15 at 04:31
  • Weird. So you confirm that you have ES 2.1 + Logstash 2.1 installed, right? – Val Nov 29 '15 at 04:34
  • Logstash `2.1.0`, and Elasticsearch `Version: 2.1.0, Build: 72cd1f1/2015-11-18T22:40:03Z, JVM: 1.7.0_91` I tried editing that `elasticsearch-template.json` and specifying the path to the new JSON file, setting all string fields to `not_analyzed`, nuked all my indices in Elasticsearch and ran Logstash again. That made no difference, strings were still analyzed and no raw fields created. – Brad Nov 29 '15 at 04:38
  • Can you update your question with your elasticsearch output config? – Val Nov 29 '15 at 04:40
  • 1
    I'll post a new question and link you to it in a minute, so as not to get this one off topic. However, the only output configuration specifies the elasticsearch host. I experimented with adding the new template but it made no difference. `output { elasticsearch { hosts => ['localhost:9200'] } } ` – Brad Nov 29 '15 at 04:43
  • Good move, I was going to suggest ;-) – Val Nov 29 '15 at 04:45
  • For some reason, manually putting it into Logstash worked for me. It didn't work when I specified it in the config, nor by default like it probably should have, but using curl to `PUT` it up to Logstash solved my problem. No idea what the root problem was, but perhaps I'll mess with it another day to reproduce. Thanks again for your help. – Brad Nov 29 '15 at 04:52
  • Ok, I'm glad you figured it out. Feel free to come back with more details later if needed. – Val Nov 29 '15 at 04:54