takeUntil? Processing chunked input. How?

Question

Suppose I have a generator, that returns me some results broken up into chunks, that I want to pull into a flat list:

 def pull(chunk: Chunk, result: Stream[Item] = Stream.empty): Stream[Item] = {
    val soFar = chunk.items ++ result
    if(chunk.hasNext) pull(generator.next(chunk), soFar) else soFar
 }

Conceptually, this is what I want, except, it fetches the entire content upfront, and I want it to be lazy. Something like this:

 Stream.iterate(generator.first)(generator.next)
   .takeWhile(_.hasNext)
   .flatMap(_.items)

almost works, but it discards the last chunk.

It seems like I need a .takeUntil here: like takeWhile, but go through the entire chain before terminating. How do I do this idiomatically?

Related? http://stackoverflow.com/questions/33602714/how-to-implement-takeuntil-of-a-list — evan.oman, Dec 28 '16 at 23:15
@evan058 that doesn't help much, because `.span` materializes the whole thing upfront. :/ — Dima, Dec 28 '16 at 23:20

score 0 · Answer 1 · answered Dec 29 '16 at 13:19

This is what I came up with ... Looks kinda yucky but this is the best I could think of:

 Stream.iterate(generator.first) {
   case chunk if chunk.hasNext => generator.next
   case _ => null
 }.takeWhile(_ != null)
  .flatMap(_.items)

Any better ideas?

takeUntil? Processing chunked input. How?

1 Answers1