0

I'm tring to read a json log file and insert into solr collection using apache nifi.logfile is in following format(one json object perline)

{"@timestamp": "2017-02-18T02:16:50.496+04:00","message": "hello"}
{"@timestamp": "2017-02-18T02:16:50.496+04:00","message": "hello"}
{ "@timestamp": "2017-02-18T02:16:50.496+04:00","message": "hello"}

I was able to load the file and split by lines using different processes. How can i proceed further ?

Alfred Francis
  • 451
  • 1
  • 6
  • 20

1 Answers1

3

You can use the PutSolrContentStream processor to write content to Solr from Apache NiFi. If each flowfile contains a single JSON record (and you should ensure you are splitting the JSON correctly even if it covers multiple lines, so examine SplitJSON vs. SplitText), each will be written to Solr as a different document. You can also use MergeContent to write in batches and be more efficient.

Bryan Bende wrote a good article on the Apache site on how to use this processor.

Andy
  • 13,916
  • 1
  • 36
  • 78
  • thanks @Andy, that helped. Now is it possible to add an id element(for solr) to each json record ? – Alfred Francis Mar 15 '17 at 18:33
  • 1
    If you are just looking for a unique id for each document it would probably be easier to do this on the Solr side of things https://stackoverflow.com/questions/18698651/configuring-solr-to-use-uuid-as-a-key – Bryan Bende Mar 15 '17 at 20:32