Not all data from logstash is being indexed by elasticsearch

Question

I'm new to the ELK stack and I have got LogStash to send data from MySQL to ElasticSearch, and on the terminal, it looks like it has sent all 40,000 records but when I go and look in Kibana I see that only 200 records have been inputted.
Here is my LogStash configuration file I used.

# file: simple-out.conf
input {
    jdbc {
        # Postgres jdbc connection string to our database, mydb
        jdbc_connection_string => "jdbc:mysql://localhost:3306/tweets_articles"
        # The user we wish to execute our statement as
        jdbc_user => "root"
        # The path to our downloaded jdbc driver
        jdbc_driver_library => "/etc/elasticsearch/elasticsearch-jdbc-2.3.3.1/lib/mysql-connector-java-5.1.38.jar"
        # The name of the driver class for Postgresql
        jdbc_driver_class => "com.mysql.jdbc.Driver"
        jdbc_user => "**"
        jdbc_password => "***"
        # our query
        statement => "SELECT * from tweets"
    }
}
output {
        elasticsearch { hosts => ["localhost:9200"] }
        stdout { codec => rubydebug }
}

Would this be an issue with the dates? in MySQL when I print the time of a record it's in this format.

+---------------------+
| PUBLISHED_AT        |
+---------------------+
| 2017-03-06 03:43:51 |
| 2017-03-06 03:43:45 |
| 2017-03-06 03:43:42 |
| 2017-03-06 03:43:30 |
| 2017-03-06 03:43:00 |
+---------------------+
5 rows in set (0.00 sec)

But when I see the output from the config in Terminal it looks like this.

             "id" => 41298,
         "author" => "b'Terk'",
  "retweet_count" => "0",
 "favorite_count" => "0",
"followers_count" => "49",
  "friends_count" => "23",
           "body" => "create an ad",
   "published_at" => "2017-03-06T07:30:47.000Z",
       "@version" => "1",
     "@timestamp" => "2017-03-06T06:44:04.756Z"

Can anyone else see why I can't get all 40,000 records in?
Thanks.

Did you configure your kibana to search for all log entries at all time? — pandaadb, Mar 06 '17 at 14:12
I used an ELK stack droplet from Digital Ocean not too sure how it is configured. How would I do that? — Definity, Mar 06 '17 at 15:28
In the right upper hand corner (I believe) you can define the time-ranges that you are searching — pandaadb, Mar 06 '17 at 15:30
Also, can you use a manual ES query to count all your documents? You can look up the API on the Elasticsearch documentation. This way you can do a match-all query that will return the document count for all documents. Given that the only thing in your ES are 40k documents, the match count should reflect that as well: https://www.elastic.co/guide/en/elasticsearch/reference/current/search-count.html — pandaadb, Mar 06 '17 at 15:31

score 0 · Answer 1 · edited May 23 '17 at 12:00

0

I found this to be the answer. Kibana doesn't show any results in "Discover" tab

You can see here where to click.

edited May 23 '17 at 12:00

Community

1
1

answered Mar 06 '17 at 15:51

Definity

691
2
11
31

Not all data from logstash is being indexed by elasticsearch

1 Answers1