2

I need to read a file from online url and process it in multithread and write it in another output . https://www.w3.org/TR/PNG/iso_8859-1.txt Solution I tried : If cluster is master I forked new thread based on cpu count .and outside for loop read the file. I have to make sure each thread is given certain lines to print in output file as same line should not be repeated.

I am not sure if this is correct way to do it. Please share your answers or suggestion.

rakesh ROCKING
  • 81
  • 5
  • 1
    I don't think it is a good idea to process one file in multiple threads, it will complicate the things very much. Maybe you want to process the file in a single thread different from the main thread and return result to the main thread once the file is processed? Something else you can do is process the file in the main thread in a non-blocking way. – codtex Feb 21 '19 at 08:00

1 Answers1

1

Could you elaborate why you think you should process something in a different thread?

Please check my detailed reply here regarding a similar question: How to truly make a time consuming request async in node

To summarize:

  • Node.JS is executing your javascript code in one thread only (mainthread)
  • Built-in modules like fs, http, net etc. make use of libuv worker threads to process I/O in a background thread
  • Thirdparty modules might do the same, it depends on their implementation
  • If you want your code to be executed in a background thread, you have 2 options:
    • Write a C-wrapper module which uses libuv worker threads to do the stuff
    • Use WorkerThreads in your javascript code which were introduced in Node.JS 10 (Docs), but please note that the status of this module is experimental and not ready for production

Generally speaking, I highly doubt you want to use threads on your own. Your question indicates you want to download a file from a webserver. The available modules (fs, request, http, ...) already make use of background threads to do the I/O, so it is unclear to me where on top of that you would need background processing.

Please clarify more, or think about sticking to the available modules in case you don't need "more multithreading".

user826955
  • 3,137
  • 2
  • 30
  • 71