Applying a projection transformation is the most standard way of solving this. You can find details on how to do that in almost any tutorial that shows how to use OpenGL with shaders. In summary, you typically have a projection matrix as a uniform variable in your vertex shader, and apply it to the input positions:
uniform mat4 ProjMat;
in vec4 InPosition;
...
gl_Position = ProjMat * InPosition;
In your C++ code, you calculate the matrix to apply the necessary scaling, and set it using glUniformMatrix4fv().
Another option is that you keep the viewport square. While the viewport is often set to match the size of the window, this does not have to be the case. For example, the viewport can extend beyond the size of the window.
To use this approach, you would have something like this in the code where you set the viewport in response to window (re-)size events. With w and h the width and height of the window:
if (w > h) {
glViewport(0, (h - w) / 2, w, w);
} else {
glViewport((w - h) / 2, 0, h, h);
}
This takes the larger of the two window dimensions as the viewport size, and extends the viewport beyond the window in the other dimension to produce a square viewport.
