Update Dec 27,2016:
I've written an article and created a project on GitHub in order to solve the problem. The first results look encouraging. Basically, you only need to include the CSS file resetBSToPF.css add insert the CSS style class "pf" to some of the PrimeFaces components in order to use both BootsFaces and PrimeFaces in the same project. Most CSS rules defined in the file work recursively, so that shouldn't be much of a problem.
Currently, the CSS file is merely a sketch covering some important use cases, but I suppose there's still a lot to be done. Let's make this a community effort! I'm sure this solves the incompatibility quickly.
My old answer:
People often report that it's difficult to combine PrimeFaces and BootsFaces (or Bootstrap in general). However, I think there are a few PrimeFaces components that are more problematic than others. The datatable seems to be one of them. You could try to replaces the PrimeFaces datatable by the BootsFaces datatable. The PrimeFaces datatable is much more powerful, but in most cases, you don't need the advanced features. You still need to override some of the CSS, but hopefully that's not too difficult.
By the way, this Portuguese article explains why Bootstrap and PrimeFaces are incompatible, although it doesn't explain how to solve the issue.