A longer explaination for anyone with the same problem.
What would work
Your first command
openssl genrsa -out file.pem -passout pass:file -aes256 1024
generates a PKCS#1 private key file (file.pem):
-----BEGIN RSA PRIVATE KEY-----
Proc-Type: 4,ENCRYPTED
DEK-Info: AES-256-CBC,1DA219DB746F88C6DDA0D852A0FD3232
AEf09rGkgGEJ79GgO4dEVsArwv4IbbODlxy95uHhfkdGYmuk6OlTpiCUE0GT68wn
KFJfBcHr8Z3VqiHGsXxM5QlKhgnfptxfbrdKErgBD5LQcrvnqmf43KeD4lGQcpiy
...
...
mAKMCwiU/GKZz8ZwQ4qGkBlVVCOFfgwmfbqguJF2l8yzM8lYI9MZ9NEwKkvEbc
-----END RSA PRIVATE KEY-----
This private key file can be parsed and decrypted by x509.DecryptPEMBlock() alright.
What would not work and why
Your second command
openssl pkcs8 -topk8 -inform pem -in file.pem -outform pem -out filePKCS8.pem
converts that file into PKCS#8 format (filePKCS8.pem).
The subcommmand genpkey
would directly produce a similar result:
openssl genpkey -algorithm RSA -aes256 \
-pkeyopt rsa_keygen_bits:1024 -out filePKCS8.pem
The generated filePKCS8.pem (either way) would look similar to this:
-----BEGIN ENCRYPTED PRIVATE KEY-----
MIISrTBXBgkqhkiG9w0BBQ0wSjKpBgkqhkiG9w0BBQwwHAQIKL+ordsVfqsCAggB
MAwGCCqGSIb3DQIJCQAwHQYJYIZIWAUDBAEqBBCipOAAxWkC0/zkNLNYTSMgBIIS
...
...
zfdxjZ0XmPiwED2azsLMnRrWnRj2UqMtnv9zO/ucik9za
-----END ENCRYPTED PRIVATE KEY-----
x509.DecryptPEMBlock() does not support this format. And as specified in #8860, the Go's core library has no real plan to support pkcs#8 in the near future.
As mentioned by Gregory, if you want to work with it, you'll have better luck with 3rd party library like github.com/youmark/pkcs8 (Documentation).