QuickFIX/n expects .pfx extension files. See http://quickfixn.org/tutorial/configuration.html#ssl
This extension is normally used for PKCS #12 encoded files which contain the certificate and the private key, protected by a password.
You may need to convert your .pem files so that you have one .pfx file which contains your private key and the certificate and another file which contains your CA Certificate. The SSLCertificate and SSLCACertificate configuration parameters of the QuickFIX/n session should then be set to the path of these two files. SSLCertificatePassword should contain the password.
QuickFIX/n is strict and requires the FIX Acceptor (Bloomberg EMSX in this case) to have the x509v3 extended key attribute "TLS Web Server Authentication" (1.3.6.1.5.5.7.3.1) to be explicitly present in the server certificate that is presented during the SSL handshake.
If not present, you will get an error message in the QuickFIX/n session event log:
Remote certificate is not intended for server authentication: It is
missing enhanced key usage 1.3.6.1.5.5.7.3.1
Some FIX Acceptors still don't define this in their server certificate, so even if you go to the effort of converting your PEM files to PFX, it still won't work.
You can use Stunnel which doesn't seem to mind what purpose was intended for the server certificate that is presented. As an added bonus Stunnel understands PEM files so no need to convert. However, you should be aware that if your end goal is end-to-end encryption, the hop between your QuickFIX/n application and Stunnel will be in clear text.