Stack Overflow
Stack Overflow

Questions tagged [logstash-jdbc]

187 questions
7
votes
4 answers

Logstash 'com.mysql.jdbc.Driver' not loaded

I have a problem with jdbc_driver_library. I'm using ELK_VERSION = 6.4.2 and I use Docker for ELK. When I run: /opt/logstash# bin/logstash -f /etc/logstash/conf.d/mysql.conf I'm getting an error: error: com.mysql.jdbc.Driver not loaded. Are you…
mkop
  • 91
  • 1
  • 6
5
votes
2 answers

Running Logstash on multiple nodes with JDBC input plugin

I have a basic HA setup for Logstash - two identical nodes in two separate AWS availability zones. Each node runs a pipeline that extracts a dataset from DB cluster and then outputs it downstream it to ELasticSearch cluster for indexing. This works…
demisx
  • 7,217
  • 4
  • 45
  • 43
5
votes
2 answers

Saving date in microsecond format in ElasticSearch

I am trying to save set of events from MySQL database into elastic search using jdbc input plugin with Logstash. The event record in database contains date fields which are in microseconds format. Practically, there are records in database between…
BarathVutukuri
  • 1,265
  • 11
  • 23
5
votes
1 answer

Logstash: TZInfo::AmbiguousTime exception when parsing JDBC column Logstash

I am getting this exception when getting my data with the Logstash JDBC input plugin: error: 26413962 Sequel::InvalidValue TZInfo::AmbiguousTime: 2017-11-05T01:30:00+00:00 is an ambiguous local time. This is probably because I am already…
Olivier
  • 454
  • 8
  • 19
5
votes
3 answers

Logstash-JDBC plugin : Importing millions of records from Oracle to ElasticSearch

Scenario : - I am trying to import a large data set from Oracle DB (Few Million records) into ElasticSearch. - I am able to import data using logstash jdbc input plugin. Issue : - Its taking a very long time (5 hrs) to complete. Is there a better…
shan
  • 288
  • 4
  • 11
5
votes
1 answer

Logstash JDBC - how to process json field?

I have postgresql which stores some data as json fields, eg: {"adults":2,"children":{"total":0,"ages":[]}} I'm using logstash-input-jdbc plugin to process the data How do i parse json from jdbc? From logs i see that the fields arrive as…
gerasalus
  • 7,538
  • 6
  • 44
  • 66
4
votes
1 answer

logstash Error: com.mariadb.jdbc.Driver not loaded

I'm trying to sync some tables from MariaDB to ElasticSearch using LogStash. I'm on a Debian Buster (10) Server $ java -version openjdk version "11.0.4" 2019-07-16 OpenJDK Runtime Environment (build 11.0.4+11-post-Debian-1deb10u1) OpenJDK 64-Bit…
Abdelmalek Samih
  • 41
  • 5
4
votes
1 answer

How to run the schedule every five minutes in Logstash 5.0?

I'm having a jdbc connection within my input in logstash where as I'm trying to execute the query according to the schedule property. I went through jdbc and rufus-scheduler, but still pretty unclear of what those five stars(*) represent…
Kulasangar
  • 9,046
  • 5
  • 51
  • 82
3
votes
1 answer

Logstash timezone exception with SQL Server

I am using LogStash 7.3.2 version to extract data from SQL Server. It is throwing below exception: Exception when executing JDBC query {:exception=># transition (daylight savings time 'gap'): 1942-09-01T00:00:00.000 (Asia/Kolkata)>} I have…
Raj
  • 429
  • 2
  • 6
  • 26
3
votes
3 answers

How to stop logstash to write logstash logs to syslog?

I have my logstash configuration in my ubuntu server which reads data from the postgres database and send the data to elastic search. I have configured a schedule at each 15 minutes the logstash will look the postgres table, if there is any change…
Prakash
  • 591
  • 3
  • 9
  • 28
3
votes
0 answers

Sequel + Logstash JDBC input: handle square brackets in column names

Question: could Logstash JDBC + Sequel be choking on column names that contain square brackets? And if so, is there something that could be done to get them to play nicely? I'm using Logstash 7.3.0 and its bundled Sequel in a JDBC input plugin to…
user3456014
3
votes
1 answer

How to connect postgres database with logstash from JDBC to import data?

I'm trying to connect PostgreSQL database with Logstash to import data from postgres to elasticsearch. I'm using JDBC driver to connect Logstash with postgres. But i'm getting following error [2019-06-27T13:04:05,943][ERROR][logstash.javapipeline …
Roshan
  • 3,236
  • 10
  • 41
  • 63
3
votes
0 answers

LogStash JDBC input scheduling issue - is it possible to run query a minute after the previous query finished

I'm using the Jdbc input in LogStash to retrieve data from MS SQL database once in a minute. Usually it works fine. But we know database performance is not very reliable thing and sometime it's takes longer than one minute to a query to return.…
Kamarey
  • 10,832
  • 7
  • 57
  • 70
3
votes
1 answer

Logstash error when converting MySQL value to nested elasticsearch property on suggestion field

A Huge cry for help here, When i try to convert a MySQL value to a nested elasticsearch field using logstash i get the following error. {"exception"=>"expecting List or Map, found class org.logstash.bivalues.StringBiValue",…
Birdy
  • 775
  • 1
  • 6
  • 21
2
votes
0 answers

Difference between fetch_size and page_size in JDBC input

I'm using Logstash to import a lot of data from a MySQL database. I'm facing a java.lang.OutOfMemoryError: Java heap space error. After a few searches I found out a good solution is to enable paging (here) and set jdbc_page_size. In the current conf…
Winux
  • 31
  • 5
1
2 3
…
12 13