Is the reason we can run 32bit application on 64bit OS is because of
the x86-64?
Yes for the most part. But let me briefly explain that 32bit/64bit does not necessarily actually refer to register size because the question would be which register (type) are you talking about? More accurately, the 32bit/64bit refer to the width of the address bus (ie how much memory can be addressed) or in other words the size of a pointer in C. Then, in turn the 32/64bit would (indirectly) refer to the word size aka the size of the basic 'integer' type of the machine. Anyway, the 32bit/64bit has to do more with addressable memory but is also related to word size (sizeof(int)) and in turn register size.
So, back to the question. Alot of 32Bit binaries can still run on 64bit systems. Yes, because the "new" x86-64 only extends the original x86 ISA (with more instructions of course). Therefore, your old 32bit binary (which consists of x86 instructions) may still be runnable in the new system.
If so, why are 32bit applications sometimes not compatible in 64bit
windows? Why do we need WOW64? (Sometimes we are asked to choose which
version to install.)
Yes, there are a number of cases where old x86 32bit binary/application won't be runnable in a new 64bit system. For example, your binary may contain one or more instructions that (for some reason) are no longer supported (see this for examples). Or, If your binary/application specifically requires the use of a 32bit library that does not exist in your new system. In that case, your system will complain that you can't run this 32bit application.
Does x64 instruction set have any other instruction set except x86-64?
Some people told me x64 is just to extend memory range and the
instruction set they have is identical. Is this correct?
First, the Intel64, AMD64, x86-64, x64 all pretty much (more or less) refer to the same thing!
Yes, the x86-64 instruction set is HUGE and receives extensions on a regular basis (think MMX, SSE, AVX....etc). Each extension is a like a mini instruction set. Look up x86 instruction set on Wikipedia you should get a ton of info.