If your webhost allows it, or you will need to deal with sensitive data, then use HTTPS, period. (It's often required by the law afaik).
Otherwise if you want to do something over HTTP. I would do something like this.
- The server embeds its public key into the login page.
- The client populates the login form and clicks submit.
- An AJAX request gets the current timestamp from the server.
- Client side script concatenates the credentials, the timestamp and a salt (hashed from analog data eg. mouse movements, key press events), encrypts it using the public key.
- Submits the resulting hash.
- Server decrypts the hash
- Checks if the timestamp is recent enough (allows a short 5-10 second window only). Rejects the login if the timestamp is too old.
- Stores the hash for 20 seconds. Rejects the same hash for login during this interval.
- Authenticates the user.
So this way the password is protected and the same authentication hash cannot be replayed.
About the security of the session token. That's a bit harder. But it's possible to make reusing a stolen session token a bit harder.
- The server sets an extra session cookie which contains a random string.
- The browser sends back this cookie on the next request.
- The server checks the value in the cookie, if it's different then it destroys the session, otherwise all is okay.
- The server sets the cookie again with different text.
So if the session token got stolen, and a request is sent up by someone else, then on the original user's next request the session will be destroyed. So if the user actively browsing the site, clicking on links often, then the thief won't go far with the stolen token. This scheme can be fortified by requiring another authentication for the sensitive operations (like account deletion).
EDIT: Please note this doesn't prevent MITM attacks if the attacker sets up their own page with a different public key and proxies requests to the server.
To protect against this the public key must be pinned in the browser's local storage or within the app to detect these kind of tricks.
About the implementation: RSA is probably to most known algorithm, but it's quite slow for long keys. I don't know how fast a PHP or Javascript implementation of would be. But probably there are a faster algorithms.