Opengl - GLM::Ortho + GLM_COORDINATE_SYSTEM = weird?

Question

While i am new to opengl and rarly ask for help i think a need some clarification regarding Z-axis, depth-test and GLM:ortho. Been strugling with this for a week or two and everything have from the start been "reversed".

So im using:

• glDepthFunc(GL_LESS);
• glDepthRange(0.0f, 1.0f);
• glm::ortho(0.0f, 800, 0.0f, 800), 0.0f, 1.0f);
• Z-axis is Y / 800(height), objects with Y = 20 is behind Y = 10.

Okej, so everything should add up, small Z is in front and big in the back, but no. Everything is the opposite.

According to this thread and default value of glDepthFunc/glClearDepth, everything should be in left-hand-coordinates when MVP is provided to a shader.

So while i read the linked thread i discovered that glm::ortho SHOULD convert right-hand-coordinates (because apperently thats what everybody is using in examples) to left-hand-coordinates. Sound greats, so why does it not work?

When you look into the library of GLM you will find glm::ortho, glm::orthoLH and glm::orthoRH which sounds very intressting. If i use glm::orthoLH everything adds up and works perfect, since it converts all my coordinates to left-hand. So why does glm::ortho not do this for me?

Apperently there is a settings for this when you compile the GLM-library. The GLM_COORDINATE_SYSTEM controls if the system is going to have glm::orthoLH or glm::orthoRH as default when glm::ortho is called. And to my surprise, glm::orthoLH is NOT default.

If you look in the source code of GLM you will find these lines,

#define GLM_LEFT_HANDED                         0x00000001      // For DirectX, Metal, Vulkan
#define GLM_RIGHT_HANDED                        0x00000002      // For OpenGL, default in GLM

#ifdef GLM_FORCE_LEFT_HANDED
#       define GLM_COORDINATE_SYSTEM GLM_LEFT_HANDED
#else
#       define GLM_COORDINATE_SYSTEM GLM_RIGHT_HANDED
#endif

OpenGL is right-handed and do not need to convert ortho to left-handed?

So my questions are,

• Why does GLM think OpenGL is right-handed?
• Is there any case where OpenGL acually want MVP as right-handed while processing shaders?

Please correct me if this doesnt make any sense. im confused too :)

Answer 1

I think your main confusion is related to this question:

Why does GLM think OpenGL is right-handed?

OpenGL is not right-handed, nor left-handed. OpenGL is just a rendering API, not a coordinate system.

In a typical rendering pipeline, you do have a lot of different coordinate systems, like:

• oject space / model space
• world space
• eye space / camera space / view space
• clip space
• normalized device coordinates
• window sapce

Each and every of these coordinate systems can have it's own conventions, including its own handedness.

Legacy OpenGL with the fixed function pipeline and the integrated matrix stack had some coordinate conventions in mind (but it did not completely enforce that you use these conventions either).

So classic OpenGL conventions are:

Normalized Device Space / Window Space left-handed, with x pointing to the right, y up, and z into the screen, and eye space right handed with x to the right, y up and camera looking into -z direction. Object and world space were typically also just right-handed (so that the view transformation is only rotation + translation, and no mirroring).

The flipping of the handedness was done in the projection matrix. No some other convention of legacy GL was that the near and far clip plane position was always defined as distances into the viewing direction, so glOrtho(..., 2,5) did actually set up a transformation which maps z_eye=-2 to z_ndc=-1 and z_eye=-5 to z_ndc=1.

If you look at the mapping I jst wrote, it should explain this quesion:

Okej, so everything should add up, small Z is in front and big in the back, but no. Everything is the opposite.

As you see, "small" z (-5) is in the back, and "big" z (-2) is in the front, like it should be in a right-handed coordinate system.

When you NOT set GLM_FORCE_LEFT_HANDED, glm uses the same conventions as legacy GL did. It will establish a right-handed view space by flipping z in the proejction matrix (and assuming all the later spaces NDC and window space are configured as left-handed).

If you set up GLM_FORCE_LEFT_HANDED, it will not introduce a mirroring along z, but it will still treat near and far as distance into viewing direction, which now is +z. This will have the result of establishing a left-handed view space (as long as the NDC and Window space is set up as left-handed, too).

Is there any case where OpenGL acually want MVP as right-handed while processing shaders?

Those are all just conventions. YOu have to just decide them in one way or the other, and no one is better than the other per-say. THis stuff only becomes interesting (and annoying) if you have different parts or pieces which uses different conventions, where you have to be very carefult to correcly convert the data at the right places.