First of all, you should be looking at why the memory process is that intensive. It could be that you are able to factor out the hashing from the operation with the private key (padding, modular exponentiation, encoding). That the signing process takes so much memory indicates that something is wrong. It could be as simple as calling Signature#update
instead of Signature#doFinal
directly.
As for the 32 bit .dll, no you cannot just load that into a 64 bit application such as the JVM. For instance, take a look here to see that this would not work. There are of course ways around it; create a separate 64 bit process, connect to that using your own SignatureSpi
class in your own Java provider (which requires to be signed using a key provided by Oracle). Needless to say, that's a lot of work for fixing something that should just work.
It's of course a shame to go through all this trouble while you have a very high end system without a valid upgrade strategy, using a deprecated version of Java. You should focus on fixing that first. You should also wonder why you cannot get a 64 bit PKCS#11 .dll
for your situation.