I am developing a REST web service that should receive files from the clients and precess them. After that I receive the file I want to create a new thread for processing the file, so I am not obliged to wait the end of the processing function.
If I am receiving a lot of files, I will create a lot of thread. Is there any limit or danger to do this?
If I am receiving a lot of files, I will create a lot of thread. Is there any limit or danger to do this?
Yes there is. I'm not sure there is a limit on the number of threads but at some point you will run out of memory. Each of the threads have stack and other local storage that will add up.
I'd limit the number of threads that you have forked by not accepting new connections if the limit has been reached. Then additional connections will wait in the TCP queues until the previous requests have been completed.
A better mechanism may be to use a fixed ExecutorService thread pool instead of forking a new thread for each request:
// create a thread pool with 10 workers
ExecutorService threadPool = Executors.newFixedThreadPool(10);
while(!shutdown) {
// receive a request from your service -- this is tricky
// process it using the thread pool
threadPool.submit(new MyRequest(...));
}
...
// once we have submitted all jobs to the thread pool, it should be shutdown
threadPool.shutdown();
The tricky part of this is how to determine which connections are readable. That takes NIO code, channel selectors, etc. to multiplex the connections you have to see which ones can be read.
When you receive files from the clients it aoutomatically will be processed in new thread. The amount of threads are limited by 2 things: 1. Application server configuration (As example Jboss or Tomcat by default could process 100 threads simultaniously); 2. Sometimes It appears that WS providers contains thread queues. As example CXF by default will not allow to process 10 threads simultaniously (in case you will not change this configuration).
A best approach would be to use Thread Pools which will provide better performance as far as thread creation is concerned (they will be pre created) and better resource management as to not overload the system and run out of memory, for example.
Theoretically you can create as many threads as your machine can process. There is no limit on number of threads but if you open more than 1000 threads you machine is expected to slow down.
I am sorry to say it but your question shows that you are trying to re-invent the wheel.
First, people tend to use thread pools when creating servers. This prevent their application from creating extra threads an improve performance. Second, all modern servers use NIO and do not create thread per request. Third, and the last: why are you developing something that have been already developed? There are a lot of perfect java web servers that support standard API. There are a lot of REST frameworks. One even the most talented programmer has no chance to develop web server that works better than those existing during reasonable time.
The last point is irrelevant if you are implementing homework.
Use a ThreadPool...
private static final int THREAD_COUNT = 10;
private static final ThreadPoolExecutor pool = new ThreadPoolExecutor(
THREAD_COUNT, THREAD_COUNT, 10, TimeUnit.SECONDS,
new LinkedBlockingQueue<Runnable>());
public static void submitAgentToPool(Runnable thread) {
try {
if(thread != null) {
LOG.info("ActiveThreads :" + pool.getActiveCount());
pool.execute(thread);
}
catch (Exception e) {
LOG.error("Exception while Starting thread: " + e.getMessage(), e);
}
}
Keep adding your threads to this pool...it will make sure that only 10 threads are running in an instant
