x87 is the legacy 80bit floating point subset of the x86 instruction set. It's obsoleted by SSE/SSE2 for float/double, but is still useful for 80bit extended precision. The 32bit x86 ABI also returns FP results in an x87 register. See the tag wiki for a tutorial link.
See the x86 wiki for links, including the most useful one: http://www.ray.masmcode.com/tutorial/index.html.