This is an old question but I thought I would post something that worked for me as well. Seeing as the .p12 file was created by another part of our company I was not able to get the .p12 file re-exported in the correct manner.
Instead I imported the .p12 to my local certificate store (windows) and then re-exported as a pfx.

Take a note of where the certificate is stored

Then, Use the MMC tool to view and export your certificate, making sure to export the private key as part of the pfx.

(You should probably delete the certificate from your local machine after the export is complete.)
After that you should be able to import your new pfx file into azure via the portal.