As you said you have incompatible Java version in your Tibco JRE and custom function .class file.
Unsupported major.minor version 51.0 error comes when you run a class file created using Java 1.7 (major version 51.0) into a lower Tibco environment JRE version for example JRE 6 or 5.
The major numbers are:
Java SE 13 = 57,
Java SE 12 = 56,
Java SE 11 = 55,
Java SE 10 = 54,
Java SE 9 = 53,
Java SE 8 = 52,
Java SE 7 = 51,
Java SE 6.0 = 50,
Java SE 5.0 = 49,
JDK 1.4 = 48,
JDK 1.3 = 47,
JDK 1.2 = 46,
JDK 1.1 = 45
(Source: https://en.wikipedia.org/wiki/Java_class_file#General_layout)
You can check Tibco JRE version in Tibco Designer Help->Runtime Environment -> Java tab ->(java.runtime.version)
You can check version of you custom function by
javap -verbose MyFunction.class| findstr "major"
how to check the jdk version used to compile a .class file
So, you need either change your tibco JRE version to be compatible with your class version or re-build you custom function class with the version that compatible with your Tibco JRE.
You can use -target option for specifying java version in java compiler when build your function.
javac -target 1.6 MyFunction.java
If you don't have source code of the function you can use decompiler like JD http://jd.benow.ca/
For changing JRE in Tibco you need to reconfigure all .tra files to new java version path. You can use /tra//bin/traUpgradeManager utility for that:
traUpgradeManager -path C:/tibco/bw/5.7 -jre C:/tibco/jre/1.7
traUpgradeManager -path C:/tibco -jre C:/tibco/jre/1.7
Please see https://support.tibco.com/s/article/Tibco-KnowledgeArticle-Article-33612 for the details.
Please note that changing JRE in all tibco environments may be not trivial process. In your case it's match easier to re-compile the custom function.