The formula looks straightforward to me; why do you think it's "too complicated for XSLT"?
It looks as if your initial problem is that you don't know how to translate hex numbers into decimals, in XSLT 1.0. Here is a simple template to do that; it accepts a two-digit hex number and returns a decimal equivalent.
<!--* Given a two-digit hex string, return the equivalent number in decimal. *-->
<xsl:template name="hex_to_dec">
<xsl:param name="raw-hex" select="'00'"/>
<!--* a list of hex digits, in numerical order *-->
<xsl:variable name="hex-digits" select="'0123456789ABCDEF'"/>
<!--* strip space and uppercase a-f *-->
<xsl:variable name="xy"
select="normalize-space(translate($raw-hex,'abcdef','ABCDEF'))"/>
<xsl:choose>
<xsl:when test="translate($xy,$hex-digits,'') != ''">
<xsl:message><xsl:value-of select="$raw-hex"
/> is not a legal hexadecimal string</xsl:message>
</xsl:when>
<xsl:when test="string-length($xy) < 2">
<xsl:message>Hex string too short</xsl:message>
</xsl:when>
<xsl:when test="string-length($xy) > 2">
<xsl:message>Hex string too long</xsl:message>
</xsl:when>
<xsl:otherwise>
<xsl:variable name="X" select="substring($xy,1,1)"/>
<xsl:variable name="Y" select="substring($xy,2,1)"/>
<xsl:variable name="Xval"
select="string-length(substring-before($hex-digits,$X))"/>
<xsl:variable name="Yval"
select="string-length(substring-before($hex-digits,$Y))"/>
<xsl:value-of select="16 * $Xval + $Yval"/>
</xsl:otherwise>
</xsl:choose>
</xsl:template>
If I were you, I'd first write a template named rgb_to_hsl
to accept three decimal numbers between 0 255 as parameters R, G, and B, and calculate an HSL string. Then I'd write a second template to accept an RGB hex string, parse out the hex values for red, green, and blue, convert them to decimals using the template given above, and then call rgb_to_hsl
.