115

If I'm going to put a program into production, there are several things I need that program to do in order to consider it "operationalized" – that is, running and maintainable in a measurable and verifiable way by both engineers and operations staff. For my purposes, an operationalized program must:

  • Be able to log at multiple levels (ex: debug, warning, etc.).
  • Be able to collect and share metrics/statistics about the types of work the program is doing and how long that work is taking. Ideally, the collected metrics are available in a format that's compatible with commonly-used monitoring tools like Ganglia, or can be so munged.
  • Be configurable, ideally via a system that allows configured properties in running programs to be updated without restarting said programs.
  • Be deployable to remote servers in a repeatable way.

In the Scala world, there are good libraries for dealing with at least the first three requirements. Examples:

As for deployment, one approach taken in the Scala world is to bundle together the bytecode and libraries that comprise one's program with something like assembly-sbt, then push the resulting bundle (a "fat JAR") to remote servers with a tool like Capistrano that executes commands in parallel over SSH. This isn't a problem that necessitates language-specific tools, but I'm curious if such a tool exists in the Haskell community.

There are probably Haskell libraries that provide the traits I've described above. I'd like to know which of the available libraries are considered "best"; that is, which are most mature, well-maintained, commonly used in the Haskell community, and exemplary of Haskell best practices.

If there are any other libraries, tools, or practices around making Haskell code "production-ready", I'd love to know about those as well.

Jonas
  • 121,568
  • 97
  • 310
  • 388
Alex Payne
  • 1,141
  • 1
  • 8
  • 9
  • 1
    The fourth bullet can cause trouble, as Haskell is compiled to native. You could try compiling statically, which might or might not work, but optimally you'd have similar environment on production server, than on development server. Cabal-dev is sandboxed environment, which might be suitable for transferring to other machines. Even then it would require at least the base libraries to be installed on target machine. – Masse Apr 27 '11 at 18:42
  • I linkified it a bit for Alex. – Greg Campbell Apr 27 '11 at 18:48
  • Outside of logging, for which I've never been keen on hslogger, I don't think there are common standard libraries to really do what you want in any of these departments. I've got a lot of hand-rolled infrastructure but its never been quite ready for prime time. The "right thing" as far as I'm concerned is to thread your config and logger around in a monad, and its not hard to do it yourself. The issue is how to write something that can satisfy a large number of use-cases in a clean, modular way. – sclv Apr 27 '11 at 19:32
  • 1
    Regarding other tools and techniques, this SO question has an overview: http://stackoverflow.com/questions/3077866/large-scale-design-in-haskell/3077912#3077912 – Don Stewart Apr 27 '11 at 19:36
  • 1
    One other thing -- you can access on *nix systems a huge amount of process stats and metadata directly via the /proc filesystem. So if you write a few routines to introspect that, it helps to substitute for the lack of direct hooks into the runtime. – sclv Apr 27 '11 at 19:40
  • 1
    deploying a binary is easy as long as you build on the same environment (you should have a staging server if your computer is a different architecture). Then you can rsync the binary and any external files. There is no ssh library for haskell for automatically executing restart commands, but you can use capistrano. – Greg Weber Apr 27 '11 at 22:05
  • 1
    @tchrist He spends the rest of the first paragraph and the bulleted list immediately following the word *operationalized* explaining its meaning in plain English. – Will McCutchen Apr 28 '11 at 15:58
  • 1
    @Will, he does, but why does it have to be there at all? Its an awfully ugly, terribly bureaucratic word. – tchrist Apr 28 '11 at 17:09

3 Answers3

54

This is a great question! Here's a first cut.

Be able to log at multiple levels (ex: debug, warning, etc.).

hslogger is easily the most popular logging framework.

Be able to collect and share metrics/statistics about the types of work the program is doing and how long that work is taking. Ideally, the collected metrics are available in a format that's compatible with commonly-used monitoring tools like Ganglia, or can be so munged.

I'm not aware of any standardized reporting tools, however, extracting reports from +RTS -s streams (or via profiling output flags) has been something I've done in the past. 

$ ./A +RTS -s
64,952 bytes allocated in the heap
1 MB total memory in use
 %GC time       0.0%  (6.1% elapsed)
 Productivity 100.0% of total user, 0.0% of total elapsed

You can get this in machine-readable format too:

$ ./A +RTS -t --machine-readable

 [("bytes allocated", "64952")
 ,("num_GCs", "1")
 ,("average_bytes_used", "43784")
 ,("max_bytes_used", "43784")
 ,("num_byte_usage_samples", "1")
 ,("peak_megabytes_allocated", "1")
 ,("init_cpu_seconds", "0.00")
 ,("init_wall_seconds", "0.00")
 ,("mutator_cpu_seconds", "0.00")
 ,("mutator_wall_seconds", "0.00")
 ,("GC_cpu_seconds", "0.00")
 ,("GC_wall_seconds", "0.00")
 ]

Ideally you could attach to a running GHC runtime over a socket and look at these GC stats interactively, but currently that's not super easy (needs an FFI bindings to the "rts/Stats.h" interface). You can attach to a process using ThreadScope and monitor GC and threading behavior.

Similar flags are available for incremental, logged time and space profiling, which can be used for monitoring (e.g. these graphs can be built incrementally).

hpc collects a lot of statistics about program execution, via its Tix type, and people have written tools to log by time-slice what code is executing.

Be configurable, ideally via a system that allows configured properties in running programs to be updated without restarting said programs.

Several tools are available for this, you can do xmonad-style state reloading; or move up to code hotswapping via plugins* packages or hint. Some of these are more experimental than others.

Reproducible deployments

Galois recently released cabal-dev, which is a tool for doing reproducible builds (i.e. dependencies are scoped and controlled).

Don Stewart
  • 137,316
  • 36
  • 365
  • 468
  • 6
    The dyre package is supposed to abstract out xmonad-style state reloading, so should be mentioned in particular, I think. However, it ties together recompilation and redeployment so is really about changes on a machine with the whole toolchain. For remote redeploys you want something more like acid-state, though its a bit heavyweight for my taste. I've got this persistent mvar abstraction which has weaker guarantees, but which you can just treat like a plain MVar that magically is populated on each launch of a binary with the last data it held. – sclv Apr 27 '11 at 19:38
  • 2
    Also, GHC's new [`EventLog`](http://hackage.haskell.org/trac/ghc/wiki/EventLog) logging framework (using `+RTS -l` at runtime) streams output to a file, which can be visualized with any tool reading the eventlog format. – Don Stewart Apr 28 '11 at 16:22
  • 2
    A program will emit logs of its events, like this: http://www.galois.com/~dons/tmp/A.event.log -- which may be visualized as -- http://i.imgur.com/QAe6r.png . I could imagine building other monitoring tools on top of this format. – Don Stewart Apr 28 '11 at 20:54
  • 2
    Also note that lots of the profiling tools are great for testing but not so much for production code. Leaving aside the overhead, -prof for example can only be used with a single processor. – sclv Apr 29 '11 at 02:26
9

I would echo everything Don said and add a few general bits of advice.

For example, two additional tools and libraries you might want to consider:

  • QuickCheck for property based testing
  • hlint as an extended version of -Wall

Those are both targeted at code quality.

As a coding practice, avoid Lazy IO. If you need streaming IO, then go with one of the iteratee libraries such as enumerator. If you look on Hackage you'll see libraries like http-enumerator that use an enumerator style for http requests.

As for picking libraries on hackage it can sometimes help to look at how many packages depend on something. Easily see the reverse dependencies of a package you can use this website, which mirrors hackage:

If your application ends up doing tight loops, like a web server handling many requests, laziness can be an issue in the form of space leaks. Often this is a matter of adding strictness annotations in the right places. Profiling, experience, and reading core are the main techniques I know of for combating this sort of thing. The best profiling reference I know of is Chapter 25 of Real-World Haskell.

Jason Dagit
  • 13,684
  • 8
  • 33
  • 56
9
  • Regarding configuration, I've found ConfigFile to be useful for my projects. I use it for all my daemons in production. It doesn't update automatically.
  • I use cabal-dev for creating reproducible builds across environments (local, dev, colleague-local). Really cabal-dev is indispensable, especially for its ability to support local, patched versions of libraries within the project directory.
  • For what it's worth, I would go with xmonad-style state reloading. Purity of Haskell makes this trivial; migration is an issue but it is anyway. I experimented with hsplugins and hint for my IRCd and in the former case there was a GHC runtime problem, and in the latter a segmentation fault. I left the branches on Github for later postmortem: https://github.com/chrisdone/hulk

Example of ConfigFile:

# Default options
[DEFAULT]
hostname: localhost
# Options for the first file
[file1]
location: /usr/local
user: Fred
Christopher Done
  • 5,886
  • 4
  • 35
  • 38