c++ handles numbers in two different ways: integer overflow or undefined behavior.
To put it simply, in c++, long doubles contain 8 bytes of data. This means you have 64 bits to store the data of your number. If you put in a large number like 300400, consider how large 300400! is (approximately 1.36 * 10 ^ 1515042). It's not a number that can be represented by 64 bits (a long double's maximum value is roughly 1.8 * 10 ^ 308).
In C++, integers will simply overflow. This means that if all the bits in the variable are 1, adding 1 more to that variable will cause everything to reset to 0, causing you to go from the maximum possible value to the minimum possible value. Doubles, by virtue of needing to maintain data regarding where the decimal point is, don't have this behavior, and instead represent themselves as inf, or infinity, in the program. When you try to divide factorial / (24 * factorial), in essence you are telling c++ to divide infinity by infinity, which isn't a number, hence why you're getting nan (i.e. "not a number").
For more information and a more in depth explanation of overflows vs inf, check out this post. For a better explanation of how integers and numbers are represented in binary/to the program, check out this video.