27

I'm going out of my mind trying to simply output UTF-8-encoded data to the console.

I've managed to accomplish this using String, but now I'd like to do the same with ByteString. Is there a nice and fast way to do this?

This is what I've got so far, and it's not working:

import Prelude hiding (putStr)
import Data.ByteString.Char8 (putStr, pack)

main :: IO ()
main = putStr $ pack "čušpajž日本語"

It prints out uapaj~�,�, ugh.

I'd like an answer for the newest GHC 6.12.1 best, although I'd like to hear answers for previous versions as well.

Thanks!

Update: Simply reading and outputting the same UTF-8-encoded line of text seems to work correctly. (Using Data.ByteString.Char8, I just do a putStr =<< getLine.) But packed values from inside the .hs file, as in the above example, refuse to output properly... I must be doing something wrong?

Don Stewart
  • 137,316
  • 36
  • 365
  • 468
  • What platform are you on? Unicode on UNIX-like platforms works quite well now; Windows support is lagging a bit. See the documentation for System.IO: "(GHC note: on Windows, we currently do not support double-byte encodings; if the console's code page is unsupported, then localeEncoding will be latin1.)" – ephemient Jan 18 '10 at 16:27
  • 64-bit Linux. Doesn't `System.IO` work only with `String`? –  Jan 18 '10 at 16:30
  • 1
    You should not use `BS.Char8`, because that one assumes 8-Bit encoding and truncates multi-byte Unicode characters. Use normal ByteStrings unless you absolutely *know* that `BS.Char8` is the right data type (that includes knowing why normal ByteStrings are explicitly not the right type for that use case). – David Aug 08 '13 at 14:44

3 Answers3

27

utf8-string supports bytestrings.

import Prelude hiding (putStr)
import Data.ByteString.Char8 (putStr)
import Data.ByteString.UTF8 (fromString)

main :: IO ()
main = putStr $ fromString "čušpajž日本語"
Wei Hu
  • 2,888
  • 2
  • 27
  • 28
22

bytestrings are strings of bytes. When they're output, they will be truncated to 8 bits, as it describes in the documentation for Data.ByteString.Char8. You'll need to explicitly convert them to utf8 - via the utf8-string package on Hackage, which contains support for bytestrings.

However, as of 2011, you should use the text package, for fast, packed unicode output. GHC truncating Unicode character output

Your example becomes a lot simpler:

{-# LANGUAGE OverloadedStrings #-}

import qualified Data.Text    as T
import qualified Data.Text.IO as T

main = T.putStrLn "čušpajž日本語"

Like so:

$ runhaskell A.hs
čušpajž日本語
Community
  • 1
  • 1
Don Stewart
  • 137,316
  • 36
  • 365
  • 468
  • Doesn't utf8-string work only with Strings, and not ByteStrings? –  Jan 18 '10 at 18:57
  • 3
    No, it also works with bytestrings. See http://stackoverflow.com/questions/2086842/using-haskell-to-output-a-utf-8-encoded-bytestring/2089195#2089195 – Don Stewart Jan 18 '10 at 21:14
-2

This is a known ghc bug, marked "wontfix".

Justin Smith
  • 2,503
  • 14
  • 12
  • Noooooooo. :( But, I'm puzzled... it seems to work fine with regular Strings? –  Jan 18 '10 at 17:06
  • Whatever this is, it's fixed now. Executing the example given on your linked page works as expected. The difference is that I'm trying to output UTF-8-encoded ByteStrings, and not UTF-8-encoded Strings, which is supposed to be more efficient. Keep in mind I'm currently using GHC 6.12.1, although I know the problem doesn't exist in GHC 6.10.4 either. –  Jan 18 '10 at 18:53
  • 3
    No, that's not actually the problem. GHC 6.12 does utf8 *String* IO, if the locale is set to that. Which in fact solves the above bug, which isn't the problem the OP is asking about. – Don Stewart Jan 18 '10 at 21:16