After successfully using Postfix on my Ubuntu machines, I decided to try Exim on a new server, based on some people commenting it's easier to set up and use.
Well I ran into issues with my Rails app right away, and, considering how easy Postfix had been to use, I decided to remove Exim and install Postfix instead.
However, how to cleanly remove it was hard to find...