The total length of CN here is 86 characters. Why dont I get an error?
It sounds like one of the OpenSSL functions is not validating parameters as you expect (maybe). The question is, what to validate against.
Generally speaking, the browsers follow the CA/Browser Forums issuing policies. Non-browser clients, like cURL, Wget and OpenSSL follow the IETF and the RFCs. They are different issuing policies.
Issuing policies are just the set of rules to follow. They are usually the same for the CA/B and the IETF but they divereg in certain areas. One of the areas is Key Usage for intermediate CA certificates. The IETF treats them as a union so they are additive; while the CA/B treats them as an intersection so a missing key usage in the chain is subtractive. Another detail is Common Name length.
The CA/B has an unbounded length for the Common Name. The IETF bounds the Common name at 64. Changing the IETF's issuing policies came up a while back on the PKIX mailing lists; see Amendment to CABF Baseline Requirements. I don't recall what came of it.
In our application, the CN fields are not the domain names, they are just the information from the device
Good (that a hostname is not in the Common Name). Common Names are friendly names displayed to the user. Host names always go in the Subject Alternate Name, not the Common Name. Both the CA/B and the IETF agree on this. And both the CA/B and the IETF agree the practice of placing a hostname in the Common Name is deprecated but not forbidden. The CA/B is changing it soon, and soon it will be forbidden.
It might even be a good idea to add it as a custom field to the certificate.
-subj "/C=YY/O=XXXX/CN=YY:12/CN=XXX:ABCDDMW0B170XX494/CN=ABCD:01020304050604567892030405060708/CN=BBBBCCCC"
I believe this is malformed. Only one Common Name should be present at a level in the tree.
Again, it might even be a good idea to add it as a custom field to the certificate.
I believe the IETF rules for encoding Distinguished Names are available in RFC 4514 - Lightweight Directory Access Protocol. I don't recall what the CA/B uses.
In python with pyopenssl, i cannot pass this subject because of an error "string too long"
Can someone explain it and tells me how to pass this long subject to pyopenssl
It sounds like PyOpenSSL is using IETF issuing policies. But it probably should have rejected the CN string as malformed rather than too long. I guess the string length check is more efficient so it is performed first.
In this case, you should add it as a custom field to the certificate. You might even add it as a Subject Alternate Name of type otherName or maybe directoryName. However, I don't know what the name length constraint is.
Microsoft did similar to encode User Principal Names (UPN's) in a certificate. The company simply added a custom field with the logon credential. Here's a related question: How to encode a username in PKIX certificate?.
You should probably ask a few questions on the Information Security Stack Exchange. They should be able to help you with the question, "How do I encode the name ABCD:01020304050604567892030405060708:BBBBCCCC in a PKIX certificate?"
Here's a related question: Distinguished Name length constraint in X.509 certificate, but it does not answer some of the questions you presented here.
