If your DLL has other dependencies and you are using STATIC linking to link your DLL to them, you can expect your DLL to grow larger:
Static linking vs dynamic linking
- Static linking can make binaries easier to distribute to diverse user environments (at the cost of sending a large and more resource hungry program).
EDIT:
I found the answers in this thread quite interesting for your problem: Profiling DLL/LIB Bloat
But an interesting experiment would be to verify if for each template instantiation, the resulting executable size grows linearly. If it does, you know your problem is template instantiation. There's a decent article here that talks about this type of problem and presents a technique to refactor it.
EDIT:
There's a big chance that your problem is a consequence of using boost headers. Check this thread to find out why: Why does using boost increase file size so much?