Rainbow / Dictionary Attacks, and salted passwords - a heretics approach
Passwords should not be stored in your database. Do not roll your own authentication mechanism - you will almost certainly get it wrong. Use Kereberos for valuable stuff, and well I dont have a good suggestion otherwise.
However, this question has been bashing around in my skull for a while (see the edits)
and I would like to put a heretical viewpoint over.
Rainbow tables are so-called because of the way the lookup mechanism (chaining) is used - but they are just dictionary attacks. Millions of dictionary words and common passphrases are hashed up front and then used to compare stolen hashed passwords against.
This worked well for the NT4 password process which used md5 hashes and no salt. But
when a salt is added to a password before hashing, then the rainbow table is useless - instead of looking for the md5 hash of "mypass" it has to precompute "mypass-random-string_of-letters"
Its impossible to guess what salt someone will use, so salting makes rainbow tables as a generic, use anywhere against any server solution dead in the water.
but ...
Thats only one use case - certainly a big threat, certainly one to defend against. But
salts have a problem. You must keep the salt around for when you want to authenticate the next time user logs in. They send their plaintext (over ssl!), you append the salt and hash, comapre to the hash stored in database. But if you dont keep the salt with the password, you cannot do that, and errr... no login
But we are not only defending against people passing around a table desgined to crack NT4 passwrods. We are supposed to protect our users individually.
A salt adds a two factor defence to the passwords - even with the hash an attacker will need the salt to have any chance of cracking it. But the standard advice just gives away that two factor defence. Probably its reasonable, but I am not convinced.
There is a bit of maths behind this. The usual advice (given by RSA as well - ftp.rsa.com/pub/pkcs/pkcs-5v2/pkcs5v2-0.pdf) is build a 64 bit salt, store it with the
hashed password. This way you can re-confirm the password by rehashing with the salt and reversing the hash is next to impossible.
NB - I could be wrong here ...
The "next to impossible" bit comes from simple maths.
Lets assume an 8 digit password, with 62 possible characters (letters, upper lower and numbers)
Thats 62^8 combinations, or a little over 200 million trillion.
To brute force this, by computing the hash directly, (ie making my salt specific rainbow tables) I should see a collision after 62^8/2 and at lets say 100 hashes per second, it will take around 12 million days. Yes days.
OK, so even storing the hash with the password makes the task of finding the hash quite infeasible.
However there are some assumptions in the above. Firstly that the password is a random choice of the range 62^8. In practise most passwords a much much weaker - rainbow tables aren't really based on all 62^8 possibilities - they are built out of dictionaries, and real password tables found over the years.
So the search space, instead of being 62^8 is really smaller. How small ? The depends on the password "strength".
There are about 250,000 - 750,000 words in english language (http://oxforddictionaries.com/page/93). Lets take 500,000 as a simple case. Then lets take variations that can be applied - add a digit, add a year, convert vowels to digits. That gives us say 3 possible new words per word, or 2 million possible passwords.
Generating 2 million hashes at 100 / sec give 20,000 seconds or 5 hours - well within the range of any laptop.
So if there is a specific user being targeted, (root, admin, spolsky etc) then storing a salt with the password immediately makes the cracking feasible.
Storing the salt away from the password increases the difficulty of the crack - not in any mathematical manner, just in difficulty of getting the salt and the password. One could envisage a seperate server that just takes plaintext, username and hash, and looks up the
salt used on that users join date, and returns 1/0
So, in short, if you store the salt with the password and someone accesses the passwords, then depending on password strength, each password is crackable in a reasonable length of time. A different salt per user protects all the other users, but if you are after a specific account, that is irrelevant. If the hacker is only after one password then storing a salt with password makes the crack feasible. Keeping a rotating salt elsewhere from the password table means the hacker now needs two data steals to crack the password, because without the salt any attack is doomed to thousands of years of work.
This is all a trade off - forcing people to use 15 digit passwords means they all get post-it noted to the screen, or become "easy to remember" i.e. words.
At this point you may as well move to Kerberos or similar - if the data you are protecting is valuable the users will understand and rspect it. If not why are we bothering.
But I still recommend you do not implement your own auth mechansim. Use Kerberos, use a semi public PKI, dont roll your own. I have no idea how to tell if my server just swapped the RAM holding my salt into disk and thats only one mistake I can immediately think of in rolling my own authentication.
HTH