0

I am trying to ingest data from postgres to another DB and I am using camel-jdbc component to do it. I have a large table so I want to read few rows at a time instead of the whole table altogether. so my route looks like below (only for testing purpose) from(fromUri).setBody("select * from table limit 10").to("jdbc://myDataSource?resetAutoCommit=false&statement.fetchSize=2").split(body()).streaming().process(test)

As shown above, I am only getting 10 rows at a time for testing purpose and I have set fetchSize to 2 to only receive 2 rows at a time. However, I am still receiving all 10 rows altogether. When I remove the "limit 10" from the query I get Out of Memory error just before the split command which tells me that its trying to load the entire result set in memory.

What am I missing here or what am I doing wrong?

Thanks for help.

Pri
  • 11
  • 4

1 Answers1

0

I think fetchSize is more of a hint to the JDBC driver. You can use the maxRows option to really limit on the server side, eg statement.maxRows=2. You can read more about these options on the JDBC javadoc documentation.

https://docs.oracle.com/javase/7/docs/api/java/sql/Statement.html

Claus Ibsen
  • 56,060
  • 7
  • 50
  • 65
  • statement.maxRows=2 doesnt really work. all it does is it returns only 2 rows and discards the remaining 8 rows which is not what I want. – Pri May 24 '16 at 14:13
  • Set outputType=StreamList in the Camel JDBC uri and read the documentation: http://camel.apache.org/jdbc – Claus Ibsen May 24 '16 at 14:56
  • I already tried that. Thats where I originally started where my route looked like from(fromUri).setBody(select * from table).to("jdbc://myDataSource?outputType=StreamList).split(body()).streaming().process(test). The above route doesnt event get to process as its trying to load all the data from table in memory and throws out of memory error before splitting the body. – Pri May 24 '16 at 15:12
  • @Pri Did you manage to overcome this problem? I am facing a similar situation. – Themis Pyrgiotis Oct 14 '18 at 14:24