The reason for the NaN's you get is division by zero.
Quoting this post:
The up vector is basically a vector defining your world's "upwards"
direction. In almost all normal cases, this will be the vector (0, 1,
0) i.e. towards positive Y. eye is the position of the camera's
viewpoint, and center is where you are looking at (a position).
How does the setup work? The below diagram illustrates:

u
is the up vector (third argument)
p
is the target position (second argument)
c
is the camera position (first argument)
d
is the direction vector (= p - c
)
r
is the right vector
lookAt
returns the view matrix, which has the following format:

(Where the "hats" mean normalized - having length 1)
Since you only supply the up vector and (effectively, because d = p - c
) the direction vector, we must compute the right vector using the cross-product:

(Note the above was before normalization)
Now for why your example does not work as expected. The arguments as such:
u = (0, 1, 0)
p = (0, 0, 0)
c = (0, 1, 0)
- And thus
d = (0, -1, 0)
The above gives r = d * u = (0, 0, 0)
! This is because the direction and up vectors are anti-parallel, whereas there should be an angle between them for the routine to work (ideally 90 degrees). When r
is normalized there is a division by zero, and hence the entire first row of your matrix should be occupied by NaN's.
(As a possible fix, try up = (0, 0, 1)
instead.)