My colleagues have problems implementing the above in Java. They wasted some days already.
What does the command do? Create a digest and then sign the digest using the keyfile?
Instructions to create the digest are here: How can I create an SHA512 digest string in Java using bouncy castle? How can I sign in Java?
Which algorithm does the -sign use? Does it depend on the key I am using? My keyfile is in p12 format. Is this correct?
Depending on the type of your key and the digest algorithm, openssl will determine the signature algorithm. See OpenSSL documentation,
When signing a file, dgst will automatically determine the algorithm (RSA, ECC, etc) to use for signing based on the private key's ASN.1 info.
To do the same process in Java, you have to load the keys of PKCS12 keystore and do the signature with the private key. Bouncycastle it is not needed
//Read private key identified by keyAlias from keystore
KeyStore keystore = KeyStore.getInstance("PKCS12");
keystore.load(Files.readAllBytes(Paths.get("keyfile.p12")), password);
PrivateKey key = (PrivateKey)keystore.getKey(keyAlias, password);
//Use SHA512withRSA with RSA keys, SHA512withDSA for DSA keys,...
//See supported algorithm here: https://docs.oracle.com/javase/8/docs/technotes/guides/security/StandardNames.html#Signature
String signatureAlgorithm = "SHA512withRSA";
//Digital Signature
Signature sig = Signature.getInstance(signatureAlgorithm);
sig.initSign(privateKey);
sig.update(Files.readAllBytes(Paths.get(file)));
byte[] signature = sig.sign();
OpenSSL generate HEX output by default. Use -binary to get the binary data or convert Java output to HEX
