The declaration of the parameter with <xsl:param name="indent" select="'yes'"/>
is correct but not all attributes of all elements allow an attribute value template. If we look at http://www.w3.org/TR/xslt20/#serialization then we see that those attributes don't allow an attribute value template as otherwise the syntax would say e.g. indent={yes|no}
.
If you want to define the indentation in your Java code then check the API of your XSLT processor, it probably has a method to set output serialization settings.
Based on your comment, you are using IBM's Websphere XSLT 2.0 API, I don't have experience using that API so the following is an attempt to try to read the API online documentation to suggest a possible approach to serialize with your custom settings:
XOutputParameters params = yourXSLTExecutableInstance.getOutputParameters();
params.setIndent(true);
List<XItemView> result = yourXSLTExecutableInstance.executeToList(yourJAXPInputSource);
result.get(0).exportItem(yourJAXPStreamResult, params);
That's roughly what I would try, I don't have any access to the API to test.