I just implemented this for a project on which I'm working. libmagic is what you're looking for. On RHEL/CentOS its provided by file-libs and file-devel. Debian/Ubuntu appears to be libmagic-dev.
http://darwinsys.com/file/
Here's some example code:
#include <stdio.h>
#include <magic.h>
int main(int argc, char **argv){
const char *mime;
magic_t magic;
printf("Getting magic from %s\n", argv[1]);
magic = magic_open(MAGIC_MIME_TYPE);
magic_load(magic, NULL);
magic_compile(magic, NULL);
mime = magic_file(magic, argv[1]);
printf("%s\n", mime);
magic_close(magic);
return 0;
}
The code below uses the default magic database /usr/share/misc/magic. Once you get the dev packages installed, the libmagic man page is pretty helpful. I know this is an old question, but I found it on my hunt for the same answer. This was my preferred solution.