Haskell: save one listitem at a time to file

Question

I'd like to save a huge list A to a textfile. writeFile seems to only save the list at the very end of the calcultaion of A, which crashes because my memory is insufficient to store the whole list.

I have tried this using

writeFile "test.txt" $ show mylistA

Now I have tried saving the elements of the list, as they are calculated using:

[appendFile "test2.txt" (show x)|x<-mylistA]

But it doesn't work because:

No instance for (Show (IO ())) arising from a use of `print' Possible fix: add an instance declaration for (Show (IO ())) In a stmt of an interactive GHCi command: print it

Can you help me fix this, or give me a solution which saves my huge list A to a text file?

Thank you

There shouldn't be a problem in the first place. I tested with `main = writeFile "test.txt" $ show [x | x <- [1..10^9]]` and this runs in constant space (1.4MB used) even without optimizations. Maybe it's something in how you build `mylistA` that prevents it from being consumed lazily (do the first elements depend on later ones?). In that case, your alternative solution wouldn't help at all and you should give us more code to determine the cause of your problem. — firefrorefiddle, Aug 01 '13 at 10:54
Yeah, you are right, @jozefg Solution works fine, but my memory usage seems to be caused by the generation of mylistA. myListA is basically [link](http://stackoverflow.com/questions/1139953/haskell-faster-summation-of-primes) @ephemient s solution, where i add some additional contraints like `myListA = [x|x<-(takeWhile (<=10000000000) primes),... ] ` — daviid, Aug 01 '13 at 11:14
Maybe it can be transformed to run in constant space. Maybe not. You could ask a separate question about it. — firefrorefiddle, Aug 01 '13 at 11:19

daniel gratzer · Accepted Answer · 2013-08-01T10:38:12.850

The problem is that your list has the type [ IO () ] or "A list of IO actions". Since the IO is on the "inside" of out type we can't execute this in the IO monad. What we want instead is IO (). So a list comprehension isn't going to hack it here.

We could use a function to turn [IO ()] -> IO [()] but this case lends itself to a much more concise combinator.

Instead we can use a simple predefined combinator called mapM_. In the Haskell prelude the M means it's monadic and the _ means that it returns m () in our case IO (). Using it is trivial in this case

[appendFile "test2.txt" (show x)|x<-mylistA]

becomes

mapM_ (\x -> appendFile "test2.txt" (show x)) myListA

mapM_ (appendFile "test2.txt" . show) myListA

This will unfold to something like

appendFile "test2.txt" (show firstItem) >>
appendFile "test2.txt" (show secondItem) >>
...

So we don't ever have the whole list in memory.

score 2 · Answer 2 · answered Aug 01 '13 at 10:34

You can use the function sequence from Control.Monad to take a (lazily generated) list of IO actions and execute them one at a time

>>> import Control.Monad

Now you can do

>>> let myList = [1, 2, 3]
>>> sequence [print x | x <- myList]
1
2
3
[(),(),()]

Note that you get a list of all the return values at the end. If you want to discard the return value, just use sequence_ instead of sequence.

>>> sequence_ [print x | x <- myList]
1
2
3

score 0 · Answer 3 · answered Aug 01 '13 at 13:46

I just wanted to expand on jozefg's answer by mentioning forM_, the flipped version of mapM_. Using forM_ you get something that looks like a foreach loop:

-- Read this as "for each `x` in `myListA`, do X"
forM_ myListA $ \x -> do
    appendFile "test2.txt" (show x)

Haskell: save one listitem at a time to file

3 Answers3