0

I am trying to develop an http client by using http-simple library. Some implementation of the library seems confusing to me.

This library makes heavy use of Conduit; however there is also this 'setRequestBodyLBS' function and interestingly, the function 'setRequestBodyBS' is missing here. It is documented that Conduit and lazy IO do not work well together. So my question is, why not the other way around? i.e., implement the BS version of the function instead of the LBS version? What is the idea behind the choice made here?

user2812201
  • 437
  • 3
  • 7

1 Answers1

1

Internally, a lazy bytestring is like a linked list of strict bytestrings. Moving from a strict bytestring to a lazy one is cheap (you build a linked list of one element) but going in the reverse direction is costlier (you need to allocate a contiguous chunk of memory for the combined bytes, and then copy each chunk from the list).

Lazy IO uses lazy bytestrings, but they're also useful in other contexts, for example when you have strict chunks arriving from an external source and you want an easy way of accumulating them without having to preallocate a big area of memory or perform frequent reallocations/copies. Instead, you just keep a list of chunks that you later present as a lazy bytestring. (When list concatenations start getting expensive or the granularity is too small, you can use a Builder as a further optimization.)

Another frequent use case is serialization of some composite data structure (say, aeson's Value). If all you are going to do is dump the generated bytes into a file or a network request, it doesn't make much sense to perform a relatively costly consolidation of the serialized bytes of each sub-component. If needed, you can always perform it later with toStrict anyway.

danidiaz
  • 26,936
  • 4
  • 45
  • 95
  • For example, if in a Conduit I receive a chunk of strict ByteString, convert it to lazy and call 'setRequestBodyLBS' to send it out, the overhead is negligible because the conversion is O(n); but wouldn't this be bad for exception handling? I have seen it documented that it is very hard to deal with exception properly when lazy IO is involved. – user2812201 Aug 16 '18 at 17:10
  • @user2812201 Using `setRequestBodyLBS` by itself does not perform lazy I/O, it's a perfectly normal function that accepts lazy bytestrings. Likewise I doubt that the ByteString that you are receiving come from lazy I/O. In fact, streaming libraries like conduit were invented in large part to avoid the weirdness and pathological cases of lazy I/O! Do not use functions like `hGetContents`, `getContents`, and `readFile` from `System.IO`. But working with lazy bytestrings obtained by other means is fine. – danidiaz Aug 16 '18 at 19:39
  • Thank you so much for clarifying! The data I receive is from the upstream conduit and in fact I receive strict BS and have to convert to LBS to be sent out by `setRequestBodyLBS`. – user2812201 Aug 17 '18 at 13:11