Edit: My question may be too complex for what I am really asking, so skip to the TLDR; if you need it.
I have been programming 3D graphics for a while now and up until now I never seemed to have this issue, but maybe this is the first time I really understand things like I should (or not). So here's the question...
My 3D engine uses the typical OpenGL legacy convention of a RH coordinate system, which means X+ is right, Y+ is up and Z+ is towards the viewer, Z- goes into the screen. I did this so that I could test my 3D math against the one in OpenGL.
To coop with the coordinate convention of Blender 3D/Collada, I rotate every matrix during importing with -90 degrees over the X axis. (Collada uses X+ right, Y+ forward, Z+ up if I am not mistaken)
When I just use the projection matrix, an identity view matrix and a model matrix that transforms a triangle to position at (0, 0, -5), I will see it because Z- is into the screen.
In my (6DOF space) game, I have spaceships and asteroids. I use double-precision coordinates (because they are huge) and by putting the camera inside a spaceship, the coordinates are made relative every frame so they are precise enough to fit a single-precision coordinate for rendering on the GPU.
So, now I have a spaceship, the camera is inside, and it its rotation quaternion is identity. This gives an identity matrix and if I recall correctly, row columns 1-3 are representing the X, Y and Z axis of where the object is pointing at. To move the ship, I use this Z axis to go forward. With the identity matrix, the Z-axis will be (0, 0, 1).
Edit: actually, I don't take the columns from the matrix, I extract the axes directly from the quaternion.
Now, when I put the the camera in the spaceship, this means that its nose is pointing at (0, 0, 1) but OpenGL will render with -1 going into the screen because of its conventions.
I always heard that when you put the camera inside an object in your scene, you need to take the model matrix and invert it. It's logical: if the ship is at (0, 0, 1000) and an asteroid is at (0, 0, 1100), then is makes sense that you need to put the camera at (0, 0, -1000) so that the ship will be at (0, 0, 0) and the asteroid will be at (0, 0, 100).
So when I do this, the ship will be rendered with its nose looking at Z-, but now, when I start moving, my ship moves to its rotation (still identity) Z being (0, 0, 1) and the ship will back up instead of going forward. Which makes sense if (0, 0, 1) is towards the viewer...
So now I am confused... how should I handle this correctly??? Which convention did I use incorrectly? Which convention did I forget? It doesn't seem logical, for example, to invert the rotation of the ship when calculation the movement vectors...
Can someone clarify this for me? This has been bothering me for a week now and I don't seem to get it straight without doubting that I am making new errors.
Edit: isn't it at all very strange to invert the rotational part of the model's matrix for a view matrix? I understand that the translation part should be inverted, but the view should still look at the same direction as the object when it would be rendered, no?
TLDR; If you take legacy OpenGL, set a standard projection matrix and an identity modelview matrix and render a triangle at (0, 0, -5), you will see it because OpenGL looks at Z-.
But if you take the Z-axis from the view matrix (3rd row column), which is (0, 0, 1) on an identity matrix, this means that going 'forward' means that you will be getting further away from that triangle, which looks illogical.
What am I missing?
Edit: As the answer is hidden in many comments below, I summarize it here: conventions! I chose to use the OpenGL legacy convention but I also chose to use my own physics convention and they collide, so I need to compensate for that.
Edit: After much consideration, I have decided to abandon the OpenGL legacy convention and use whatever looks most logical to me, which is the left-handed system.