0

I have a job that runs and inserts over 20000 records parsing a json, I am connecting my whole application to oracle db using hibernate. It is taking around 1 hour of time because it also involves json calls and parsing of json, whereas just printing the parsed fields in the logs takes a minute or 2. My question here is, Is there a way to optimize the insertion process using hibernate.

I tried suggestions from Hibernate batch size confusion, but still I feel very slow.

  • I tried increasing batch size.
  • I tried disabling second level cache.
  • I also flushed and cleared my session depending on the batch size

I am planning to move to jdbc batch insertions, but wanna give a try to optimize using hibernate.

I hope this may give a generic expose to most of amateur programmers helping them with the best practises

Community
  • 1
  • 1
kashivivek
  • 13
  • 1
  • 7
  • You mention it takes 1 hour for 20,000 inserts because of the parsing you're doing, but nothing in your post indicates the difference in time if you eliminate Hibernate from inserting your data. So if you eliminated the insert, how quickly does the code run? – Naros Jan 21 '17 at 00:46
  • Sorry for not including it. It would take mostly a minute or 2 for just printing out the fields after parsing in the logs. I edited my question and apologies for that – kashivivek Jan 21 '17 at 00:48
  • Does your entities that you're inserting use an `IDENTITY` or `AUTO_INCREMENT` identifier like column by chance? – Naros Jan 21 '17 at 00:57
  • Yes on one of my table there is a primary key that is auto generated. – kashivivek Jan 21 '17 at 00:59
  • Using auto generated identifiers will absolutely destroy performance and will avoid being able to use any form of batch insertion over JDBC. My suggestion would be to change it to one which you provide, select the max value at the top of your insert task and then during your loop just increment it by 1 until you're done. – Naros Jan 21 '17 at 01:07
  • Thanks, let me give a try and check – kashivivek Jan 21 '17 at 01:12
  • still i cant see any improvement, I just made a combination of two fields as embedded id and served my purpose of unique records, but still its taking same long time – kashivivek Jan 25 '17 at 05:12
  • Perhaps post some code and maybe we can identify some issue? – Naros Jan 25 '17 at 14:28

0 Answers0