Caveats:
- This is less of an answer to the direct question about a given encryption algorithm, but more about your overall goal, based upon your two posted questions and answers and comments.
- While I follow software security issues generally, I'm not an expert, so take anything I say herein accordingly.
You're trying to protect your software. That is, what you're looking for is a way to implement a licensing scheme.
What you need [and hopefully have], but did not post is a functional/design specification for this. The two way encryption is but a part of this.
You aren't the first person to need this. There are already schemes to do this. I'd recommend using an existing solution. There is probably a public/free open source version.
You talked about email and registration form and cut-and-paste but didn't detail how that works. Nor did you mention how this interacts with your application. Do you register with a web browser or have the app present a registration dialog?
I'm guessing that you have a user enter an email address into a form. Then, your server will send back an email with some special code [e.g. that is valid for 15 minutes] that the user must enter to complete registration to prove the email is valid, along with the system identity information.
You mentioned that the cut-and-paste is the limiting factor. But, IMO, it isn't really as you could put the key in a MIME encoded attachment and have the user save it to a file. The registration form could then upload this file. Even a lengthy value that is (e.g. 10 lines of 72 chars / line) will still fit inside a clipboard.
You gather [hopefully] unique identifying information about the client system. The CPU serial number is now 0 on most post pentium-III processors [unless enabled in the BIOS]. But, other CPUID parts may provide some uniqueness. The Ethernet MAC address is unique, but many NICs allow it to be changed in software. A disk serial number isn't a good choice [IMO] because disks have the highest failure rate and need to be replaced over time.
But, let's assume you can gather enough information from the above sources [and maybe some others] to get a "system ID". You can concatenate this with other stuff like username, etc. You can run this through a one way hash like sha1 [or whatever] if you choose. So, now, you've got an "system identity". Save that somewhere (e.g. file or registry)
But, storing the private key in the app [or anywhere on the client system] is a non-starter, IMO. You only need this during the generation phase.
So, have the registration process send the system identity to a private server that you control (e.g. Amazon EC2). The server encrypts/signs [using public/private key crypto like RSA] the system identity and sends back the public key and the "signed result". These get stored.
The verification for any app is to regather the system identity information, apply the algorithm with it and the public key. It should match the signed result.
This eliminates the need to store the private key with the app, keep it encrypted with a password that uses RC4 [or equiv]. This eliminates complexity that just served to weaken your protection in the first place.
I'm a bit confused about the "system administrator" or "serial license generation person". If that's not you [or your server], then it implies you're going to give out OEM contracts and the local admin can generate licenses for a given subset of systems under this control. So, you'll need an OEM key [that gets sent to your server] to allow multiple licenses to be generated.
Now, the bad news.
Any software licensing scheme is [easily] breakable. In the app, no matter how complex [or simple] the verification algorithm is, it boils down to a [or multiple] go/nogo tests. That is, something like:
#include <stdlib.h>
int user_authorized(void);
void run_program();
int
main(void)
{
if (! user_authorized())
abort();
run_program();
return 0;
}
This translates to:
.globl main
main:
subq $8, %rsp
call user_authorized
testl %eax, %eax
je .L5 # change this to a nop
xorl %eax, %eax
call run_program
xorl %eax, %eax
addq $8, %rsp
ret
.L5:
call abort
Notice the je .L5 to abort the program. If the program is patched to change this instruction to nop, the license test "passes", regardless of what user_authorized does.
This can be repeated for as many security checks as the program has.
Security is a relative term. A $50 app doesn't need as much protection as one that sells for $500,000 (e.g. some CAD/CAE programs from Xilinx).
Take a leaf from Microsoft's playbook. They present a "licensing" dialog that you must accept (i.e. "shrink wrap"). One must accept the agreement. It prohibits reverse engineering, etc. This gives you legal cover. Consult an attorney on this.
This is, in addition to, whatever software scheme you ultimately decide on.
