Yes, you can check well-formedness, or the validity of just part of a document, with an XSD validator.
As others have pointed out, if you really want to check merely for well-formedness, you don't need an XSD validation step at all.
But it says here they are wrong to say you cannot use an XSD validation step to check for well-formedness: all you need is an essentially vacuous schema and a validator that you can invoke in 'lax validation' mode (which essentially validates elements and attributes against matching declarations -- in a vacuous schema, none will be found). Since any normal XSD validator will parse the XML if you hand it XML (instead of, say, a DOM object), well-formedness will be checked as a side effect. (It's possible to argue, of course, that in that case the well-formedness checking is not really part of the XSD validation process, just a necessary accompaniment to it. I am among the people who enjoy such casuistry; I don't have the impression that you care about such distinctions.)
But in fact you say you know how to describe (and thus, I suppose, to validate) the header area, and it's just the payload area that should be unconstrained. For that you want a schema of approximately the following form. It's very similar in basic idea to the sketch provided by asmith1024, except that that schema uses an explicit wildcard for its objects
element, while this one just relies on the default type of xs:anyType; one consequence is that this one's tns:payload
element will accept character data as content, while the objects
element won't.
<?xml version="1.0" encoding="UTF-8"?>
<xs:schema
xmlns:xs="http://www.w3.org/2001/XMLSchema"
elementFormDefault="qualified"
targetNamespace="http://example.com/nss/target"
xmlns:tns="http://example.com/nss/target">
<!--* a message contains a header and
* a payload. *-->
<xs:complexType name="message">
<xs:sequence>
<xs:element ref="tns:header"/>
<xs:element ref="tns:payload"/>
</xs:sequence>
</xs:complexType>
<xs:element name="message" type="tns:message"/>
<!--* a header has a defined structure
* (to be specified ...) *-->
<xs:element name="header">
<!--* ... your definition of header
* validity here ... *-->
</xs:element>
<!--* other types and elements used in
* header ... *-->
<!--* A payload has NO defined structure. *-->
<!--* no definition of any type for payload,
* so it defaults to xs:anyType, and
* can contain ... any well-formed XML
* content. *-->
<xs:element name="payload"/>