Set Null values pulled from Mysql to Elasticsearch to a default value using logstash input jdbc plugin

Question

The fields that I want to get them with default value could be NULL in MYSQL. This is my configuration for the logstash plugin.

input {
    jdbc {
        jdbc_connection_string => "jdbc:mysql://localhost:3306/elements"
        jdbc_user => "user"
        jdbc_password => "admin"
        jdbc_validate_connection => true
        jdbc_driver_library => "C:/work/Wildfly/wildfly-9.0.2.Final/modules/com/mysql/main/mysql-connector-java-5.1.36.jar"
        jdbc_driver_class => "com.mysql.jdbc.Driver"
        statement_filepath => "query.sql"
        use_column_value => true
        tracking_column => id
        #schedule => "*/3 * * * *"
        clean_run => true

    }
}

output {
    elasticsearch {
        index => "emptytest"
        document_type => "history"
        document_id => "%{id}"
        hosts => "localhost"
    }
}

I tried to do a filter to test on the null values but it does not detect them.

if [sourcecell_id] == "NULL" {

         mutate {

         }
    }

The farthest thing that I did is to delete the field via a ruby script but I dont want to delete I want to replace it with default value like 0 for example.

This is the ruby script:

   filter {
     ruby {
                code => "
                        hash = event.to_hash
                        hash.each do |k,v|
                                if v == nil
                                        event.remove(k)
                                end
                        end
                "
        }

}

score 2 · Accepted Answer · answered Sep 05 '16 at 09:47

I found the solution is a ruby script. I hope that it will help another ones.

filter {
     ruby {
                code => "
                        hash = event.to_hash
                        hash.each do |k,v|
                                if v == nil
                                        event[k] = 0
                                end
                        end
                "
        }

}

Set Null values pulled from Mysql to Elasticsearch to a default value using logstash input jdbc plugin

1 Answers1