As the comments on this question note, recursion was prohibited simply by saying that recursion was not allowed. Fortran 77 for example has the statement:
A subprogram must not reference itself, either directly or indirectly.
The consequence of this is that it is a restriction on the programmer to ensure that recursion doesn't happen, and a compiler can assume that this condition is met.1
Fortran 90 permitted recursion, and from then up to Fortran 2008 a subprogram which is to be potentially used recursively must have the recursive
prefix.
Even in Fortran 2018 (where allowing recursion is the default), correct use of recursion is a restriction on the program:
The NON_RECURSIVE prefix-spec shall not appear if any procedure defined by the subprogram directly or indirectly invokes itself
This remains even now something that the compiler is not required to validate.
1 The discussion in the answer about detection of violations doesn't truly apply to Fortran 77. Back then, a compiler could be much more trusting of the programmer to give a correct program.