2

I am developing a web service that serves some relatively large files, each created dynamically at request time. In my case this is a ZIP archive file that contains a bunch of files, but I assume the same problem will occur with other types of dynamically created files.

The problem is that I would like to avoid creating the ZIP file on disk, but rather just stream it directly to the HTTP response. One way I thought about is to use chunked streaming, which means that a streaming actor sends a chunk at the time, and "waits" for acknowledgment from the responder before sending the next chunks. (see the example of sendStreamingResponse in https://github.com/spray/spray/blob/release/1.1/examples/spray-routing/on-spray-can/src/main/scala/spray/examples/DemoService.scala)

Unfortunately all the examples I could find show how to do it when your stream is pre-defined, but I'm not quite sure what's the best way to approach this when the stream of data is being prepared in some other future.

In my case, there's a Future started by the HTTP request that does all the heavy work of fetching the files, and writing them into a java.util.zip.ZipOutputStream. But the way streaming works in Spray is the opposite, because the streaming actor needs to pull the data when it is ready - I can't just push all the data to the stream.

Is this a familiar use case, and what's the best way to tackle it?

yby
  • 915
  • 1
  • 8
  • 24

1 Answers1

0

I think we can use Spray's builtin marshaller if we produce a Steam[Byte]. I haven't tried it but something like this should work (I've done something similar in the past with dynamic image resizing and serving large files). 

val pipeIn = new PipedInputStream()
val pipeOut = new PipedOutputStream(pipeIn)
val out = new ByteArrayOutputStream()
val st = new ZipOutputStream(out)
Future{
  writeToZip(st)
  out.writeTo(pipeOut)
}
val streamResponse: Stream[Byte] = Stream.continually(pipeIn.read).takeWhile(_ != -1).map(_.toByte)
complete(streamResponse)

Spray's stream marshaller will automatically produce a chunked response.

rahilb
  • 726
  • 3
  • 15