Xml document signature not valid after deserialization from byte array

Question

I have a problem where I sign xml document and validate signature after that and the validation passes, but when I serialize document to byte array and then deserialize it back to document, signature validation fails.

Here are methods used for validation and serialization/deserialization:

public class DocumentSigner {
    @Override
    public byte[] transformToByteArray(Document doc) throws Exception {
        TransformerFactory transformerFactory = TransformerFactory
                .newInstance();
        Transformer transformer = transformerFactory.newTransformer();      
        ByteArrayOutputStream os = new ByteArrayOutputStream();
        transformer.transform(new DOMSource(doc), new StreamResult(os));
        return os.toByteArray();
    }

    private Document byteArrayToXmlDoc(byte[] documentoXml) throws Exception {
        DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance();
        factory.setNamespaceAware(true);
        factory.setIgnoringElementContentWhitespace(true);
        DocumentBuilder builder = factory.newDocumentBuilder();
        return builder.parse(new ByteArrayInputStream(documentoXml), "UTF-8");
    }

    @Override
    public Boolean validate(byte[] byteArrayDoc, Integer certificatePropertiesId) throws Exception {
        Document doc = byteArrayToXmlDoc(byteArrayDoc);
        return validate(doc, certificatePropertiesId);
    }

    public Boolean validate(Document doc, Integer certificatePropertiesId) throws Exception {
        NodeList nl = doc.getElementsByTagNameNS(XMLSignature.XMLNS,
                "Signature");
        if (nl.getLength() == 0) {
            throw new Exception("No signature element.");
        }

        KeyStore ks = KeyStore.getInstance("JKS");
        CertificatePropertiesDTO certProp = databaseLogic.getCertificateProperties(certificatePropertiesId);
        if (certProp == null || certProp.getCertificatePassword().isEmpty() || certProp.getCertificate() == null){
            throw new RuntimeException("No certificate.");
        }

        ks.load(new ByteArrayInputStream(certProp.getCertificate()), certProp.getCertificatePassword().toCharArray());
        KeyStore.PrivateKeyEntry keyEntry = (KeyStore.PrivateKeyEntry) ks.getEntry(ks.aliases().nextElement(), new KeyStore.PasswordProtection(certProp.getCertificatePassword().toCharArray()));
        X509Certificate[] certs =  (X509Certificate[]) keyEntry.getCertificateChain();
        if (certs == null || certs.length == 0) {
            throw new RuntimeException("No certificate found.");
        }

        XMLSignatureFactory fac = XMLSignatureFactory.getInstance("DOM");
        DOMValidateContext valContext = new DOMValidateContext(keyEntry.getCertificate().getPublicKey(), nl.item(0));
        NodeList els = doc.getElementsByTagNameNS("*", "SignatureProperties");
        Element el = (Element) els.item(0);
        valContext.setIdAttributeNS(el, null, "Id");
        valContext.setDefaultNamespacePrefix("dsig");

        valContext.setProperty("javax.xml.crypto.dsig.cacheReference", Boolean.TRUE);
        try {
            XMLSignature signature2 = fac
                    .unmarshalXMLSignature(new DOMStructure(nl.item(0)));
            boolean coreValidity = signature2.validate(valContext);

            // Check core validation status.
            if (coreValidity == false) {
                log.info("Signature failed core validation");
                boolean sv = signature2.getSignatureValue()
                        .validate(valContext);
                log.info("signature validation status: " + sv);
                Iterator<?> i = signature2.getSignedInfo().getReferences()
                        .iterator();
                for (int j = 0; i.hasNext(); j++) {
                    Reference ref = (Reference) i.next();
                    boolean refValid = ref.validate(valContext);
                    log.info("ref[" + j + "] validity status: " + refValid);
                }
                return false;
            } else {
                log.info("Signature passed core validation");
                return true;
            }
        } catch (Exception ex) {
            log.info("EXCEPTION during validation: " + ex.getMessage());
            return false;
        }
    }

    public void signDocument(Document doc)
    {
        ....
    }



public void writeToDisk(String path, String rac)
    {
        BufferedWriter writer = null;
        try
        {
            writer = new BufferedWriter(new FileWriter(path));
            writer.write(rac);

        }
        catch ( IOException e)
        {
        }
        finally
        {
            try
            {
                if ( writer != null)
                writer.close( );
            }
            catch ( IOException e)
            {
                try {
                    throw e;
                } catch (IOException e1) {
                    // TODO Auto-generated catch block
                    e1.printStackTrace();
                }
            }
        }
    }

    @Override
    public String transformToString(Document doc,
            Boolean omitXmlDeclaration) throws Exception {
        TransformerFactory transformerFactory = TransformerFactory
                .newInstance();
        //transformerFactory.setAttribute("indent-number", 4);
        Transformer transformer = transformerFactory.newTransformer();      
        if (omitXmlDeclaration)
            transformer.setOutputProperty(OutputKeys.OMIT_XML_DECLARATION,
                    "yes");
//      transformer.setOutputProperty(OutputKeys.INDENT, "yes");
//      transformer.setOutputProperty(OutputKeys.ENCODING, "UTF-8");
//              
        StringWriter sw = new StringWriter();
        transformer.transform(new DOMSource(doc), new StreamResult(sw));
        //String output = sw.getBuffer().toString().replaceAll("\n|\r", ""); 
        return sw.toString();
    }
}

Here is where it passes/fails:

public void SignAndValidate()
{
    ...
    Document doc = createDocument();
    documentSigner.signDocument(doc);

    validate(doc, 1); 

    // OUTPUT:
    // Signature passed core validation

    byte[] docArr = documentSigner.transformToByteArray(doc);

    validate(docArr, 1);

    // OUTPUT:
    // Signature failed core validation
    // signature validation status: false
    // ref[0] validity status: false
    // ref[1] validity status: true
}

If necessary I will post methods for creating/signing document but it's big.

Here is the signing method:

private void signDocument(Document document) throws Exception {
        //Remove ApacheXMLDSig because after every SOAP message signing it's set as default provdier, but doesn't know about here() function from XPATH2
        Security.removeProvider("ApacheXMLDSig");

        XMLSignatureFactory sigFactory =  XMLSignatureFactory.getInstance("DOM", "XMLDSig");
        String id = String.format("id%s", UUID.randomUUID().toString());

        KeyStore ks = KeyStore.getInstance("JKS");
        CertificatePropertiesDTO certProp = databaseLogic.getCertificateProperties(1);
        if (certProp == null || certProp.getCertificatePassword().isEmpty() || certProp.getCertificate() == null){
            throw new RuntimeException("No certificate.");
        }

        ks.load(new ByteArrayInputStream(certProp.getCertificate()), certProp.getCertificatePassword().toCharArray());
        KeyStore.PrivateKeyEntry keyEntry = (KeyStore.PrivateKeyEntry) ks.getEntry(ks.aliases().nextElement(), new KeyStore.PasswordProtection(certProp.getCertificatePassword().toCharArray()));
        X509Certificate[] certs =  (X509Certificate[]) keyEntry.getCertificateChain();
        if (certs == null || certs.length == 0) {
            throw new RuntimeException("No certificate found.");
        }


        Element propSig = XMLElement(document, "PROP_Sig", "");
        Attr propNS = XMLAtribut(document, "xmlns", "http://ns.adobe.com/pdf/2006");
        propSig.setAttributeNodeNS(propNS);
        propSig.setAttribute("type", "cabinet");
        DateFormat df = new SimpleDateFormat("yyyyMMddHHmmssZZ");
        Element m = XMLElement(document, "M", String.format("D:%s", df.format(new Date())));
        m.setAttribute("type", "text");
        Element name = XMLElement(document, "Name", cert.getSubjectX500Principal().getName());      
        name.setAttribute("type", "text");
        propSig.appendChild(m);
        propSig.appendChild(name);
        SignatureProperty sp = sigFactory.newSignatureProperty(Collections.singletonList(new DOMStructure(propSig)), "data_signature", null);
        SignatureProperties sps = sigFactory.newSignatureProperties(Collections.singletonList(sp), id);

        CanonicalizationMethod cm = sigFactory.newCanonicalizationMethod(CanonicalizationMethod.EXCLUSIVE_WITH_COMMENTS, (XMLStructure) null);
        SignatureMethod sm = sigFactory.newSignatureMethod(SignatureMethod.RSA_SHA1, null);
        DigestMethod dm1 = sigFactory.newDigestMethod(DigestMethod.SHA1, null);
        Transform tf1 = sigFactory.newTransform(CanonicalizationMethod.EXCLUSIVE_WITH_COMMENTS, (TransformParameterSpec) null);     
        Reference ref1 = sigFactory.newReference("#" + id, dm1, Collections.singletonList(tf1), "http://www.w3.org/2000/09/xmldsig#SignatureProperties", null); 

        DigestMethod dm2 = sigFactory.newDigestMethod(DigestMethod.SHA1, null);
        String here = "here()/ancestor::dsig:Signature[1]/../../../../../..//. | "
                + "here()/ancestor::dsig:Signature[1]/../../../../../..//@* | "
                + "here()/ancestor::dsig:Signature[1]/../../../../../..//namespace::*";
        HashMap<String, String> hm = new HashMap<String, String>();
        hm.put("dsig", "http://www.w3.org/2000/09/xmldsig#");
        XPathType xp = new XPathType(here, XPathType.Filter.INTERSECT, hm);
        TransformParameterSpec paramsXPath2= new XPathFilter2ParameterSpec(Collections.singletonList(xp));
        Transform tf2 = sigFactory.newTransform(Transform.XPATH2, paramsXPath2);
        Transform tf3 = sigFactory.newTransform(Transform.ENVELOPED, (TransformParameterSpec) null);
        Transform tf4 = sigFactory.newTransform(CanonicalizationMethod.EXCLUSIVE_WITH_COMMENTS, (TransformParameterSpec) null); //"http://www.w3.org/2001/10/xml-exc-c14n#WithComments"
        List<Transform> lt2 = new ArrayList<Transform>();
        lt2.add(tf2);
        lt2.add(tf3);
        lt2.add(tf4);
        Reference ref2 = sigFactory.newReference("", dm2, lt2, null, null);

        List<Reference> lr = new ArrayList<Reference>();
        lr.add(ref1);
        lr.add(ref2);
        SignedInfo si = sigFactory.newSignedInfo(cm, sm, lr);

        KeyInfoFactory kif = KeyInfoFactory.getInstance("DOM", "XMLDSig");

        ArrayList<Object> x509Content = new ArrayList<>();
        for (int i = 0; i < certs.length; i++) {
            x509Content.add(certs[i]);
        }

        X509Data xd = kif.newX509Data(x509Content);
        KeyInfo ki = kif.newKeyInfo(Collections.singletonList(xd));

        DOMSignContext dsc = new DOMSignContext(keyEntry.getPrivateKey(), document.getElementsByTagName("sac:SignatureInformation").item(0));       
        XMLSignature signature = 
                sigFactory.newXMLSignature(si, ki, Collections.singletonList( sigFactory.newXMLObject(Collections.singletonList(sps), null, null, null)), "data_signature", null);

        dsc.setProperty("javax.xml.crypto.dsig.cacheReference", Boolean.TRUE);
        dsc.setDefaultNamespacePrefix("dsig");
        try {
            signature.sign(dsc);
        }
        catch (Exception ex) {
            log.warn(ex.getMessage());
            throw new RuntimeException("Signing failed");
        }
    }

Here is part of sample XML document which is signed:

        <ext:UBLExtension>
            <ext:ExtensionContent>
                <sig:UBLDocumentSignatures>
                    <sac:SignatureInformation>
                        <dsig:Signature xmlns:dsig="http://www.w3.org/2000/09/xmldsig#" Id="data_signature">
                            <dsig:SignedInfo>
                                <dsig:CanonicalizationMethod Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#WithComments"/>
                                <dsig:SignatureMethod Algorithm="http://www.w3.org/2000/09/xmldsig#rsa-sha1"/>
                                <dsig:Reference Type="http://www.w3.org/2000/09/xmldsig#SignatureProperties" URI="#idfe5688f4-583f-4a98-b26c-9d651b2f8918">
                                    <dsig:Transforms>
                                        <dsig:Transform Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#WithComments"/>
                                    </dsig:Transforms>
                                    <dsig:DigestMethod Algorithm="http://www.w3.org/2000/09/xmldsig#sha1"/>
                                    <dsig:DigestValue>iq802IBHl5kVdIMWA5Wlb5hYEoY=</dsig:DigestValue>
                                </dsig:Reference>
                                <dsig:Reference URI="">
                                    <dsig:Transforms>
                                        <dsig:Transform Algorithm="http://www.w3.org/2002/06/xmldsig-filter2">
                                            <dsig:XPath Filter="intersect" xmlns:dsig="http://www.w3.org/2002/06/xmldsig-filter2">here()/ancestor::dsig:Signature[1]/../../../../../..//. | here()/ancestor::dsig:Signature[1]/../../../../../..//@* | here()/ancestor::dsig:Signature[1]/../../../../../..//namespace::*</dsig:XPath>
                                        </dsig:Transform>
                                        <dsig:Transform Algorithm="http://www.w3.org/2000/09/xmldsig#enveloped-signature"/>
                                        <dsig:Transform Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#WithComments"/>
                                    </dsig:Transforms>
                                    <dsig:DigestMethod Algorithm="http://www.w3.org/2000/09/xmldsig#sha1"/>
                                    <dsig:DigestValue>2jmj7l5rSw0yVb/vlWAYkK/YBwk=</dsig:DigestValue>
                                </dsig:Reference>
                            </dsig:SignedInfo>
                            <dsig:SignatureValue>d+DRc25SXnhxwXJs10A9Hnf1g0gG2bZqqnpTbZvrzp8X3EvtOVr3dBP6Ldc1RMTJYSF+guanlWKn
liaKlu7VbdB+SiQRuAMAZt+9Cnbn0CMlIzt22nRJNzjbeLBpCm7K63jCHGOXsWCW43DI/DYeZwq+
Q2j7WESgOtWLqUO0Jn8=</dsig:SignatureValue>
                            <dsig:KeyInfo>
                                <dsig:X509Data>
                                    <dsig:X509Certificate>...</dsig:X509Certificate>
                                    <dsig:X509Certificate>...</dsig:X509Certificate>
                                </dsig:X509Data>
                            </dsig:KeyInfo>
                            <dsig:Object>
                                <dsig:SignatureProperties Id="idfe5688f4-583f-4a98-b26c-9d651b2f8918">
                                    <dsig:SignatureProperty Target="data_signature">
                                        <PROP_Sig xmlns="http://ns.adobe.com/pdf/2006" type="cabinet">
                                            <M type="text">D:20151130163741+0100</M>
                                            <Name type="text">CN=<CN>,L=<City>,O=<Organization>,C=<Country></Name>
                                        </PROP_Sig>
                                    </dsig:SignatureProperty>
                                </dsig:SignatureProperties>
                            </dsig:Object>
                        </dsig:Signature>
                    </sac:SignatureInformation>
                </sig:UBLDocumentSignatures>
            </ext:ExtensionContent>
        </ext:UBLExtension>
    </ext:UBLExtensions>

I don't understand why validation says reference[0] fails (the one which referes to element with id), but reference to whole document passes?

Could you please try to remove `factory.setIgnoringElementContentWhitespace(true);` in `byteArrayToXmlDoc`? It could cause the issue. The problem is your XML from/to byte array conversions are not inverse operations. — vojta, Nov 30 '15 at 09:41
@vojta It didn't help, same exact output, It was probably left in code accidently anyway since I played with different options prior to posting hoping it would help. — formatc, Nov 30 '15 at 09:56
Please, post `String` representations of your XML document before converting to `byte` array and after converting back to `Document` in your `validate(byte[] byteArrayDoc,...)` method. There must be a visible difference. (XML to String: http://stackoverflow.com/questions/5456680/xml-document-to-string) — vojta, Nov 30 '15 at 10:01
@vojta I already tried that and used WinMerge to compare them, the only thing I noticed was the extra line at the end, but let me try that agian just to be sure. — formatc, Nov 30 '15 at 10:06
@vojta Nope, I tried it agian, WinMerge informs me that the files are indentical, I added methods used to write to files above. — formatc, Nov 30 '15 at 11:11
If you really want this question answered, please, spend some time and create a complete program (also known as [mcve]), otherwise we can't debug it. If it is big, you can create a project on github, for example. — Roman, Dec 07 '15 at 14:46
If feels like the you're not signing and verifying the same thing (e.g. a docArr should be on both sides of the sign/verify - input, and then output). Try a sample program. A signature applies to a binary sequence. So either when you're signing, of verifying, what you've passed in doesn't match. Try calculating an in-memory SHAx for the data immediately prior to the sign, then immediately prior to the verify. If the hashes don't match, then see why. Maybe you're including the signature nodes that weren't in the original document, so the hash differs, so the sig verify fails. — user326608, Dec 08 '15 at 11:21

score 2 · Answer 1 · answered Dec 02 '15 at 16:38

2

@formatc I don´t have privileges to comment, but can you try and view the hexadecimal values in both files (sign and deserialized). I had the same problem, for some reason in my case when constructing back the xml some non-visual characters are inserted in front of the document. You won´t see them unless you use HexView or some tool like that.

I was able to remove those characters programmatically and all went well.

answered Dec 02 '15 at 16:38

lumee

613
1
5
15

I checked both with hex editor, they are same, at least on disk. Thanks for suggestion anyway. – formatc Dec 07 '15 at 13:42
@formatc I had the same issue, it was bugging me for months. The way I ended up fixing it was to explicitly disable indentation when serializing with the transformer: `transformer.setOutputProperty(OutputKeys.INDENT, "no");` Somehow indentation was corrupting the digital signature. – I am a Terrible Programmer Nov 05 '20 at 07:47

score 0 · Answer 2 · answered Dec 09 '15 at 08:06

0

Be sure their array character longs for the case are they equal? before the validation you must see some differencies.
Also, some Signture technics can use any inner sign prefixes like a GUID, control them.

Use Utf8 on both

answered Dec 09 '15 at 08:06

Hamit YILDIRIM

4,224
1
32
35

score 0 · Answer 3 · answered Apr 05 '16 at 14:31

i had exactly the same issue as you have. signature was valid but references not. the problem is (at least in my case), that serialization and deserialization can affect the content of the xml. in my case it helped to call document.normalizeDocument() before signing the document and now the signature validates even after serialization/deserialization.

score 0 · Answer 4 · answered Oct 14 '16 at 11:20

In my case it was the difference in the header value which is why it was failing.

The original xml document has and when it was written header was changed to

Which is why signature verification was failing.

Hence, remove xml declaration while parsing the doc and while writing the doc.

Xml document signature not valid after deserialization from byte array

4 Answers4