How do I serialize or save to a file a Thunk?

Question

In Haskell, you can have infinite lists, because it doesn't completely compute them, it uses thunks. I am wondering if there is a way to serialize or otherwise save to a file a piece of data's thunk. For example let us say you have a list [0..]. Then you do some processing on it (I am mostly interested in tail and (:), but it should support doing filter or map as well.) Here is an example of sort of what I am looking for.

serial::(SerialThunk a)=>a->serThunk
serialized = serial ([0..] :: [Int])
main=writeToFile "foo.txt" serialized

And

deserial::(SerialThunk a)=>serThunk->a
main=do
    deserialized <- readFromFile "foo.txt" :: IO [Int]
    print $ take 10 deserialized

Similar questions (perhaps, more elaborate on where to look for these mechanisms in the implemented Haskell stuff, i.e., distributed programming): [Can Haskell functions be serialized?](http://stackoverflow.com/q/17785916/94687). — imz -- Ivan Zakharyaschev, Feb 10 '15 at 12:26
See also [Haskell for all: The internet of code](http://www.reddit.com/r/haskell/comments/36d12v/haskell_for_all_the_internet_of_code/) for a (theoretical) suggestion how to encode functions for sending them. — imz -- Ivan Zakharyaschev, Jun 05 '15 at 00:26

score 6 · Answer 1 · answered Mar 08 '14 at 17:43

6

No. There is no way to serialize a thunk in Haskell. Once code is compiled it is typically represented as assembly (for example, this is what GHC does) and there is no way to recover a serializable description of the function, let alone the function and environment that you'd like to make a thunk.

Yes. You could build custom solutions, such as describing and serializing a Haskell expression. Deserialization and execution could happen by way of interpretation (ex. Using the hint package).

Maybe. Someone (you?) could make a compiler or modify an existing compiler to maintain more information in a platform-agnostic manner such that things could be serialized without the user manually leveraging hint. I imaging this is an are under exploration by the Cloud Haskell (aka distributed-haskell) developers.

Why? I have also wanted an ability to serialize functions so that I could pass closures around in a flexible manner. Most of the time, though, that flexibility isn't actually needed and instead people want to pass certain types of computations that can be easily expressed as a custom data type and interpretation function.

answered Mar 08 '14 at 17:43

Thomas M. DuBuisson

64,245
7
109
166

For my Why, I just wanted to keep a list of avaible ids in a database, so instead of "auto-incrementing ids", it would just be "infinite list of ids." That would also make it easier to add back in deleted ids. – PyRulez Mar 08 '14 at 18:18
I found [a discussion (laziness and serialization)](https://groups.google.com/forum/#!topic/parallel-haskell/t-IK4DI2WIU) where it is discussed that thunks might not be forced before sending. I haven't yet understood whether this can be somehow used to send thunks. " no deepseq evaluation will be done for you. If you want deepseq evaluation to be done on your behalf, use the unsafe primitives mirrored in distributed-process-platform instead, which utilise NFData via a new typeclass NFSerializable. ... a receiving Process might crash (due to unevaluated thunks)" – imz -- Ivan Zakharyaschev Feb 03 '15 at 15:20
And `:sprint` in GHCi seems to have access to internal hunk representation -- http://stackoverflow.com/q/24755672/94687 . Perhaps GHCi works with some special, non-optimized code. So in principle one could use this representation and the implementation of `:sprint` if one wants to serialize hunks, isn't that true? – imz -- Ivan Zakharyaschev Feb 04 '15 at 10:49

score 2 · Answer 2 · answered Mar 08 '14 at 20:10

2

Cloud Haskell supports serialization of function closures. http://www.haskell.org/haskellwiki/Cloud_Haskell

answered Mar 08 '14 at 20:10

haroldcarr

1,523
15
17

1

Not exactly. The code must already be shared and only function pointer are transmitted. This suffices for some users but is sadly limiting. – Thomas M. DuBuisson Mar 09 '14 at 01:30
@ThomasM.DuBuisson Could you please clarify this thing about "shared" code: if I have the same binary "a.out", and run two different processes "./a.out", can I safely send a serialized closure from one process to the other one? – imz -- Ivan Zakharyaschev Feb 02 '15 at 18:46
@ThomasM.DuBuisson Why? I wanted to send intermediate, lazy results to another process, so that it can decide when and how to use it: when it needs to do something with it, it would use an expression where this result is used inside (and possibly send it to another process for evaluation). Sharing the same code between evaluators is not a problem, but they might be different instances... – imz -- Ivan Zakharyaschev Feb 02 '15 at 18:54
@imz--IvanZakharyaschev You seem to have the right idea. If the systems are running the same binary then passing a closure could be implemented as sending 1) a function pointer 2) arguments. This isn't really a complete "closure" as the implementation only allows top level functions (not lambda's, for example) and the solution doesn't allow passing new code from node to node. It is sort of a frustrating half solution. – Thomas M. DuBuisson Feb 02 '15 at 18:55
@ThomasM.DuBuisson After reading about HdpH, I have the impression that one could approach having a "deep" grip on an expression for sending it as partially-evaluated structure if you write your code not as composition of functions, but rather as a composition of HdpH's closures. (Perhaps, one could employ an automatic translator for pure Haskell code into closures...) I've written down my thoughts at http://stackoverflow.com/a/17791889/94687 . – imz -- Ivan Zakharyaschev Feb 10 '15 at 12:48

imz -- Ivan Zakharyaschev · Accepted Answer · 2015-07-10T11:29:00.763

packman: "Evaluation-orthogonal serialisation of Haskell data, as a library" (thanks to a reddit link) -- is exactly what we have been looking for!

...this serialisation is orthogonal to evaluation: the argument is serialised in its current state of evaluation, it might be entirely unevaluated (a thunk) or only partially evaluated (containing thunks).

...The library enables sending and receiving data between different nodes of a distributed Haskell system. This is where the code originated: the Eden runtime system.

...Apart from this obvious application, the functionality can be used to optimise programs by memoisation (across different program runs), and to checkpoint program execution in selected places. Both uses are exemplified in the slide set linked above.

...Another limitation is that serialised data can only be used by the very same binary. This is however common for many approaches to distributed programming using functional languages.

...

imz -- Ivan Zakharyaschev · Answer 4 · 2015-07-10T11:10:00.447

Apart from the work in Cloud Haskell and HdpH on "closures", and part from the answers stating that thunks are not analyzable at runtime, I've found that:

:sprint in GHCi seems to have access to internal thunk representation -- . Perhaps GHCi works with some special, non-optimized code. So in principle one could use this representation and the implementation of :sprint if one wants to serialize thunks, isn't that true?
http://hackage.haskell.org/package/ghc-heap-view-0.5.3/docs/GHC-HeapView.html -- "With this module, you can investigate the heap representation of Haskell values, i.e. to investigate sharing and lazy evaluation."

I'd be very curious to know what kind of working solutions for seriliazing closures can be made out of this stuff...

How do I serialize or save to a file a Thunk?

4 Answers4