This answer is expanding on Patrick Mevzek's answer. To quote:
See https://bugzilla.mozilla.org/show_bug.cgi?id=1034124 and
specifically this answer:
Looks like that certificate has a basicConstraints extension with the
value cA: TRUE. We stopped allowing CA certificates to act as
end-entity certificates. That certificate should be regenerated
without the basicConstraints extension.
This is also explained at
https://wiki.mozilla.org/SecurityEngineering/x509Certs
Error Code: MOZILLA_PKIX_ERROR_CA_CERT_USED_AS_END_ENTITY
What It Means: A certificate with a basic constraints extension with cA:TRUE is being used as an end-entity certificate
What Can I Do: Re-generate the end-entity certificate without the basic constraints extension
In short, to use self-signed SSL certificates to work in Firefox (e.g., for testing HTTPS), you'll have to go the long route of creating and using your own public key infrastructure (PKI):
Step 1. Establish your private certificate authority (CA)
By creating a private key and self-signed certificate:
openssl req -x509 -nodes \
-newkey RSA:2048 \
-keyout root-ca.key \
-days 365 \
-out root-ca.crt \
-subj '/C=US/ST=Denial/L=Earth/O=Atest/CN=root_CA_for_firefox'
Generated files:
Step 2. Create a private key and a certificate signing request (CSR) for your server
openssl req -nodes \
-newkey rsa:2048 \
-keyout server.key \
-out server.csr \
-subj '/C=US/ST=Denial/L=Earth/O=Dis/CN=anything_but_whitespace'
Generated files:
server.key: Private keyserver.csr: Certificate signing request; does not any domain names to be specified.
Step 3. Generate a certificate for your server
NOTE
Make sure to use the right domain name at the subjectAltName = DNS:<domain> part below!
openssl x509 -req \
-CA root-ca.crt \
-CAkey root-ca.key \
-in server.csr \
-out server.crt \
-days 365 \
-CAcreateserial \
-extfile <(printf "subjectAltName = DNS:localhost\nauthorityKeyIdentifier = keyid,issuer\nbasicConstraints = CA:FALSE\nkeyUsage = digitalSignature, keyEncipherment\nextendedKeyUsage=serverAuth")
The <(..) part is called process substitution and it works in Bash. If you use another shell, simply put the text inside the double quotes in a file, replace "\n" with newlines, and specify the file instead for -extfile.
Generated files:
root-ca.srl: I usually ignore itserver.crt: Certificate to be used on your server
Step 4. Add root-ca.crt to Firefox's trust store
By following these steps.
Step 5. Configure your server
You'll need server.key and server.crt. The specific steps depend on the technology.
Why are these steps needed for Firefox?
While the following one-liner below, where one would use the created cert both as a root CA cert (to install in a system's trust store) and a cert for a web server (e.g., in NGINX with ssl_certificate), works both in Chrome and Safari,
openssl req -x509 -new -nodes \
-newkey RSA:2048 \
-days 365 \
-subj '/C=US/ST=Denial/L=Springfield/O=Dis/CN=anything_but_whitespace' \
-addext 'subjectAltName = DNS:doma.in,DNS:anoth.er' \
-addext 'authorityKeyIdentifier = keyid,issuer' \
-addext 'basicConstraints = CA:FALSE' \
-addext 'keyUsage = digitalSignature, keyEncipherment' \
-addext 'extendedKeyUsage=serverAuth' \
-out self-signed.crt \
-keyout private.key
it won't in Firefox, because:
You can't install self-signed.crt in Firefox's trust store, because it explicitly states that the certificate is not a certificate authority (CA:FALSE).
What if you change CA:FALSE to CA:TRUE in the command above?
Chrome and Safari will still accept it, you can install self-signed.crt into Firefox's trust store, but it won't work from the server's side because of the quoted reasons at the top of this answer.
What if you simply delete the -addext 'basicConstraints = CA:FALSE option?
Firefox will complain when trying to import self-signed.crt in its trust store:
(Probably because of the x509 v3 extensions added with -addext.)
What if you omit all the x509 v3 extensions (i.e., -addext options)?
openssl req -x509 -new -nodes \
-newkey RSA:2048 \
-days 365 \
-subj '/C=US/ST=Denial/L=Springfield/O=Dis/CN=anything_but_whitespace' \
-out self-signed.crt \
-keyout private.key
Now you can import self-signed.crt into Firefox's trust store, but you will get SSL_ERROR_BAD_CERT_DOMAIN (and you'll get a similar error in Chrome and Safari as they need the subjectAltName extension present on the certificate from the server's side).
What about changing the command to contain CN=<your domain>?
You'll probably still get the same error as "Firefox from 101.0 onward no longer uses certificate CN (Common Name) for matching domain name to certificate and have migrated to only using SAN (Subject Alternate Name)".
Which means you have to add at least the subjectAltName x509 v3 extension, but then you are back at items 1, 2, or 3.
