0

My idea is to divide a big response text into small parts to load them concurrently.

The following code helps me open stream from an URL but I want to load its whole content from multithreads to optimize performance, then I will merge them into a single result. However, the method return a ReadableByteChannel which cannot specify the start position and I have to transfer it linearly:

URL url = new URL("link");
InputStream fromStream = url.openStream();
ReadableByteChannel fromChannel = Channels.newChannel(fromStream);

Is there any way to specify the position like SeekableByteChannel (seem likes this interface only works with file)? Thanks you :D

Trần Nam Trung
  • 103
  • 1
  • 6
  • 1
    What's wrong with just reading and ignoring the part you don't want? – Jim Garrison Aug 03 '18 at 17:15
  • I want the whole content, but from many threads, then I will merge them into a single result. – Trần Nam Trung Aug 03 '18 at 17:17
  • 3
    The response from a single request is inherently serial. There is no reason to think that using multiple threads just to read such a response should provide any benefit whatever. – John Bollinger Aug 03 '18 at 17:21
  • my idea is to divide a big response text into small parts to load them concurrently. does it make sense? – Trần Nam Trung Aug 03 '18 at 17:24
  • 2
    The only way to accomplish that would be to split it up at the client side and send over multiple sockets, in which case you don't need to skip anything, each thread would only see its own fragment. There is NO WAY to split up an incoming stream at the server and achieve any speedup. As @JohnBollinger says, it is a serial stream. If you want to split up _processing_, receive the entire stream into memory on one thread, then split it up and allocate to threads. You cannot split up the ***reading*** of the stream. – Jim Garrison Aug 03 '18 at 17:27
  • Okay I got it, but can someone help me explain Vipul Gaur's answer on this post : https://www.quora.com/How-does-IDM-internet-download-manager-work-in-layman%E2%80%99s-terms – Trần Nam Trung Aug 03 '18 at 17:36
  • 1
    That is explained by the answer that you accepted. (According to Vipul Gaur, the IDM is issuing a number of separate requests to download different chunks of the file using "Range" headers.) It is unclear if this would actually help in your case. There are to many "unknowns". – Stephen C Aug 04 '18 at 01:21

1 Answers1

2

If you can manipulate the request before it's a stream then yes, you would use the Range http header to specify the chunk of data you wanted...

See Downloading a portion of a File using HTTP Requests

If not then you will manually have to read past the data you don't need.

See Given a Java InputStream, how can I determine the current offset in the stream?

Jay
  • 3,276
  • 1
  • 28
  • 38