0

I have a code which does a load process for a particular student id perfoming inserts for a particular student id. I have around 500 student id which have to go through the exact same procedure. I wanted to use multithreading here as i did not want to go about loading the data for a student id ne by one. How can i use multithreading in this scenario or is multithreading helpful in this scenario. Here how can i use this huge code in multithreading.

gautam vegeta
  • 653
  • 6
  • 13
  • 28
  • what have you tried? Multithreading will be helpful if you have a capable machine, and granularity of tasks (which is true from the information; am assuming students size can grow). Hint : create a thread pool (use Executors), create tasks and submit the tasks to the thread pool. Also, make the db commits in batches. – Scorpion Apr 11 '12 at 04:23
  • I have not used multi-threading for a program this large.That's the reason i asked. – gautam vegeta Apr 11 '12 at 04:32
  • What do you mean by "load"? Do you have a database? – trutheality Apr 11 '12 at 04:33
  • By load i mean each student have several thousand rows in different tables and i have to load based on several conditions to another table pertaining to student here.Yes i have a database and its sybase. – gautam vegeta Apr 11 '12 at 04:35
  • 1
    Performing operations in parallel from the same client to the same database is not a good idea. – trutheality Apr 11 '12 at 04:39
  • Can you please explain why its a problem here.I thought this scenario is demanding threading.Its just a back end load.There wont be any synchronization problems here as all data for one student cannot be rerepeating for the other. – gautam vegeta Apr 11 '12 at 04:44
  • 2
    Well, you get overhead on the client side creating multiple threads for connections, you get overhead on the DB side creating multiple threads to receive those connections, and the DB still has to take these parallel updates that it receives and process them one by one. Use batch updates instead, that way the DB can optimize for the best way to insert many records at once. – trutheality Apr 11 '12 at 04:54

3 Answers3

4

Do not use multithreading here. It will take a simple task and make it much more complex. For 500 records, just do them one at a time. If you really must, use JDBC batching to reduce the number of database hits.

Jim Garrison
  • 85,615
  • 20
  • 155
  • 190
  • What's the issue here.Since i have a multi processor system shouldn't threading help here.Wont the multiple threads running will be able to load each student id parallel instead of one after the other which will save time here. – gautam vegeta Apr 11 '12 at 04:31
  • Its five hundred students for which i have to perform loading where each sudent will have several thousand records associated with a particular student.Its not 500 records. – gautam vegeta Apr 11 '12 at 04:40
  • 1
    Still doesn't make a difference. The database will likely be multi-threaded, and will allocate CPU power appropriately. This is what databases are good at. The cost of opening 500 connections, even if you can get that many, will swamp any savings. The information in your question is rather sparse, and you haven't given anything that would indicate that multithreading is warranted. You're over-analyzing and engaging in premature optimization. Try it single-threaded and then adapt if it's really too slow. – Jim Garrison Apr 11 '12 at 06:12
  • Well i wont be having 500 threads.I wanted around 3 or 4 threads doing the job,I tried with the single threaded method and i quite slow but gets the job done one by one.What information is missing from my question. – gautam vegeta Apr 11 '12 at 06:30
0

Sounds like something that would work well with a "Producer/Consumer" pattern. The wikipedia article on it has a reasonable Java example to start from:

http://en.wikipedia.org/wiki/Producer-consumer_problem

And make sure that each thread has its own database connection (they're typically not thread-safe).

ɲeuroburɳ
  • 6,990
  • 3
  • 24
  • 22
0

If you truly feel that you need the performance of parallel access to the database then you will need:

  • To use a ThreadPool with a limit on the number of threads it will hold.
  • Each thread must hold a ThreadLocal connection to the database.
  • Each thread must hold a ThreadLocal statement to perform the inserts.

It is not simple but it certainly can be done. Please make sure you really need to do it. It is well worth investigating batching first to see if that will help.

I will try to post some sample code here later if you are interested.

OldCurmudgeon
  • 64,482
  • 16
  • 119
  • 213
  • Thanks for the reply.Can you please post some sample code for the same.And do you mean to say the connection object for the database should be exclusive for each thread and the database statement should be created inside the thread.And why should i create a thread pool.I was just planning to have 3-4 threads for the entire work. – gautam vegeta May 02 '12 at 09:46
  • @guatam - You are asking for a significant amount of code. I am sorry I do not have time at the moment to put together code that would be both succinct and of value to you. [Here](http://stackoverflow.com/q/9440505/823393) is a question that has a `ThreadLocal` db connection sample. Please read the comments there too. – OldCurmudgeon May 02 '12 at 11:22