2

I am calling a web service which can return very high volume of data (>100K records = 200 MB). I have to insert this data in to SQL Server too. I have the following questions.

  1. I know it depends on the server resources, but is there a ball park advice on limit of how much data should I store in any java structure (Collection - with item having 4,5 string members each of length < 255) at run-time? I am already using 50,000 records in each call (I am not sure how much memory does it take)...

  2. I then upload this data using batch sizes of 1000 to database using JDBC. Is this correct approach? Would there be any benefit if I use JPA for this instead of JDBC?

  3. Also any standard design to handle this? I can think of breaking down the web service calls into pages of limited size and then using Java Threads to handle them. Is this the right direction?

Thanks

Tintin
  • 2,853
  • 6
  • 42
  • 74

1 Answers1

0

First of all a web service which can return very high volume of data is no t enough information.A web service which can return very high volume of data ALWAYS, ONCE IN A WHILE , X% of THE TIME etc can help in designing a better system.

Its not advisable to use web services to exchange such a large quantity of data because it puts a strain on physical network infrastructure too but I guess that service is not part of your system.

Your application will be very unreliable with that amount of data per hit and you will need a very fast network too to get that amount of data.

and now coming to your points,

1.You have guessed it right, it all depends on server resources. There are applications which might be comfortable with a million records in a collection and at some places few thousands might be too much. You have to keep heap space and limits imposed by OS in mind. All in all - this is very specific to an application.

Purpose of collection plays a role too - is it for look up or just temporary storage to pass around data? how frequently does it get cleaned up? is it on stack or as object field? is it loaded once and cleaned before next load or keeps growing?

2.JDBC batch is a correct approach and not JPA.

3.if reading data from web service and storing data in DB is the main flow of the job , Spring Batch API might better fit into your design.

Hope it helps !!

Community
  • 1
  • 1
Sabir Khan
  • 9,826
  • 7
  • 45
  • 98