It is much easier to compile it on the target OS than cross compiling it. What you need is a toolchain for every OS and a "make" tool. CMake has powerful crosscompiling abilities. This is not a necessity, but it will save some money: Get virtualization software (e.g. VMWare Player is free) and run on a different OS.
I would recommend clang (OSX), gcc (Linux), TDM gcc (Windows) as toolchains (MSVC is also nice, but is not free). The difference between 32bit and 64bit should not be the problem. When you are working with different compilers, I advise you to stick to the standard by turning the most pedantic compiler flags on at each.
I would also recommend you to have a continuous integration server somewhere with one client for every OS target and architecture. This will ease the pain of incompatible changes that you make in one OS. And you will also be able to ensure installation by package manager or installer.
Just search the web for further readings on cross compilation, toolchains and continuous integration.