Quality differences between HTML, Canvas and WebGL

Question

I want to draw an image in WebGL, but downscaled. When I don't scale it, then the image has a good quality, but if I scale it down then it has poor quality.

I read about 'Handling High DPI (Retina) displays in WebGL' from here: http://www.khronos.org/webgl/wiki/HandlingHighDPI and I tryed to do the same. My code in WebGL is:

        Initializations:

        var devicePixelRatio = window.devicePixelRatio || 1;

        gl.canvas.style.width = "800px";
        gl.canvas.style.height = "600px";

        canvas.width = Math.round(800 * devicePixelRatio);
        canvas.height = Math.round(600 * devicePixelRatio);


        For drawing:

        gl.viewport(0, 0, gl.drawingBufferWidth, gl.drawingBufferHeight);
        matrix = m3.scale(matrix, 28/800, 35/600); // matrix for scaling texture

        Textures:

        gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_S, gl.CLAMP_TO_EDGE);
        gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_T, gl.CLAMP_TO_EDGE);
        gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, gl.NEAREST);
        gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MAG_FILTER, gl.NEAREST);

But this didn't have the same quality like the HTML image. My code in HTML is:

        setTimeout(function() {

            var imagine = new Image();
            imagine.src = 'Tank.png';
            imagine.width = '28';
            imagine.height = '35';
            document.body.appendChild(imagine);

        }, 1000);

My code in canvas is:

        var imagine = new Image();
        imagine.src = 'Tank.png';
        imagine.width = '28';
        imagine.height = '35';
        imagine.onload = function() {
            context.drawImage(imagine, 0, 0, 28, 35);
        }
        context = canvas.getContext('2d');

The difference between quality of images is below ( quality: WebGL < Canvas < HTML)

I know that is not a very big difference, but I really want to show pictures at good quality. In HTML the picture is smoother. I observed that if I zoom in the Chrome browser, than the picture from HTML is increasing in quality, but the WebGL picture it remains at the same resolution and it will decrease quality. And if I refresh the page, to update the devicePixelRatio for WebGL, then the picture has better quality, but my browser is working slower at 500% zoom, I think because it makes the canvas bigger and it has to draw more. But in HTML, if I move the image at 500% zoom, there are not problems, the image is moving well and it has good quality.

In this situation - WebGL image rendering bad quality - the image is not scaled, but I need scaling down pictures.

And compared to this situation - canvas drawImage quality - I put the values as integers, in all three programs.

Final Question:

How can I draw images in webGL as in HTML at the same quality (not seeing that lines from those tracks) and to have good quality even the browser zoom in ? What other possibilities do I have to draw them ? What technology should I use ? I wanted to use WebGL because it has some features that the HTML doesn't have and I wanted to draw some things from scratch, like lines or points.

EDIT 1: This is the picture at the normal resolution.

In this picture it seems that the lines of the tracks are not straight, but they are crossing. If you can look close to the images from Canvas and WebGL, the first lines of the tracks, first on left and first on right, they are straight. More, the picture with Canvas has more straight tracks, even the last ones. The last image of HTML has all tracks of the same shape.

I am sorry because I put very small pictures. It's because I need this scale and I need that pictures not to be bigger. If I would draw bigger pictures then the devicePixelRatio would make pictures to look much better, but I don't use the pictures at normal dimensions. In that way, the problem could have been solved by devicePixelRatio. For smaller dimensions this method with devicePixelRatio doesn't help me. That's why I am looking for another solution.

If those images are ordered based on the order you established above them: the first image looks crisp and perfect, the other two look blurred to the point of useless. So webgl wins. If you have a different ordering for your images than for your code: please fix that. — Mike 'Pomax' Kamermans, Feb 08 '20 at 15:13
Yes, the first picture might be seem perfect, but the tracks are not drawn well. Maybe there are more details that try to fit in there, but are not looking good. Of course, the second and the third are blurred, but because they don't force the pixels to be drawn they look better. And the third looks better than the others because it is less blurred than second. I don't like that crisp, this is why I said that that image is drawn a little bit worse. If you look carefully, the first and the second images have the first lines straight, but the third didn't. The third has all lines a little curved — Niqusor31, Feb 08 '20 at 15:30
Please provide the original image in full (original resolution) as there is no way to know what is the idea quality. — Blindman67, Feb 08 '20 at 16:09

score 4 · Answer 1 · edited Jun 20 '20 at 09:12

4

Random scale, position and rotate

If the image is transformed and animated you are best to use linear interpolation with the original image at least 2 times larger than the rendered result.

Make sure the canvas (2D or WebGL) is aligned to device pixels and there is a one to one match with the physical device pixels.

Do not use devicePixelRatio if it is any other value than 1 or 2 (2 for HDPI, or retina) as other values mean the page has been zoomed and you can not get a one to one pixel to device pixel match. Attempting to scale the canvas (2D or WebGL) will only decrease quality.

The original image

This applies best to axis aligned rendered image drawn to pixel boundaries. However it still has some benefit for scaled, rotated, and or unaligned rendering.

The most important part of good quality canvas (2d or webGL) image rendering is the relationship between the original image size and the rendered result.

If you scale the image down to 28 by 35 pixels then the original image MUST be 2* (68, 92) 4* (136, 184), etc... a power of 2 times the size of the final rendered result. NOTE you must consider the GPU RAM cost of larger images.

Original image detail aligned to rendered pixels

Important details in the image that must remain as crisp as possible must be aligned to the rendered pixel edges or you end up with blurred details.

The image below is 16* the final size. It has a checkerboard (each square 16 By 16 pixels) over it to shows the size of rendered pixels.

Note how the edges of the tracks, body, barrel, turret are all aligned with final scaled down pixels.

Rendering

The next image show the rendered results, pixel aligned (WebGL top tow rows have same result in 2D)

From left to right is the size of the original source image, 2 times to 16 times the final size.

Each row uses a different pixel lookup

Nearest. The fastest rendering the pixel nearest the top left of each pixel the final render
Linear. Default for 2d (smoothing = true), webGL (gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, gl.LINEAR)). The pixel color is a linear interpolation of the 4 pixels near (under, right, below, below right) the final pixel.
Log mean. Custom WebGL only. The fragment shader uses the total photon count of all pixels under the rendered pixel to calculate the final RGB pixel color. For original 2 times the size GPU cost is about the same as Linear, For 16 times is only for the high end devices, or low image rendering counts (64 times slower than linear per pixel)
Log Mean +. Uses same fragment shader as Log Mean to render but pre processes (Just in time) the original using a modified sharpen convolution filter to sharpen at 2/3 * invScale (Log calcs) to increase the visual contrast at low contrast boundaries. The Cost is a one time init cost.

Personally for real-time projects I use second row first column 2 * linear, and when rendering on HDPI or high render counts I use the simplest 1 * linear (not shown).

Most people can not tell the difference between the 2nd, 3rd and last rows. Can you without zooming in on the image?

Rotation

The next image shows the same method from 2 times to 16 times left to right, then rows linear, nearest, and log mean, +.

Notes

Log RGB image processing only has a significant improvement with highly saturated images with high colour contrast. The tank image has very low colour contrast and thus does not get much benefit from Log Mean. The next image shows from left to right nearest, linear, and log mean of high colour contrast version.

All images in this answer were created on GPU accelerated Chrome 80 Win 10 x64. Original tank reference image by the OP.
I just noticed, as I finish, that the Nearest rotated images all have anti-aliasing on the outer edges. This is an error on my part as I forgot to give the image some transparent padding. The anti-aliasing is due to the polygon edge and not part of the pixel gl.TEXTURE_MIN_FILTER, gl.LINEAR lookup process. To much work to fix, sorry.

edited Jun 20 '20 at 09:12

Community

1
1

answered Feb 08 '20 at 20:16

Blindman67

51,134
11
73
136

Great answer! Indeed, I don't see the differences between those tanks on the last rows. You said lots of interesting things, but I am a beginner and I don't know as much as you do. I searched on google, but I didn't find something clear about Log Mean, for instance. Can you provide some code or articles for these solutions, please ? – Niqusor31 Feb 08 '20 at 21:36
@Niqusor31 For details on sRGB https://en.wikipedia.org/wiki/SRGB What I call Log RGB, and Log Mean are variants on sRGB model that is optimized for performance, It is my own invention. The point of showing the more complex processing is that it is not worth the CPU/GPU cycles. Ensuring a one to one pixel to device pixel ratio and a good quality appropriate sized source image with contrasting/important detail aligned with rendered pixel boundaries is the most important thing in terms of rendering quality. – Blindman67 Feb 08 '20 at 22:38
1

"Make sure the canvas (2D or WebGL) is aligned to device pixels and there is a one to one match with the physical device pixels.". FYI: this is currently impossible. [They are adding APIs to kind of make to possible](https://github.com/w3c/csswg-drafts/issues/3554). Also "Do not use devicePixelRatio if it is any other value than 1 or 2" seems fishy given [lots of devices have devicePixel ratios of other values](https://www.mydevice.io/#compare-devices). – gman Feb 09 '20 at 04:13
@gman "impossible" No.. I sit in front of a canvas right now and it has perfect alignment and can reliably flag that as true (clearly not impossible). When I zoom instantly I knows that alignment is no longer possible. There are some edge cases when all indications show alignment but it is not true, such as when window crosses two or more display devices with different resolutions. – Blindman67 Feb 09 '20 at 05:41
@gman From MDN `devicePixelRatio` is *"...ratio of the resolution in physical pixels to the resolution in CSS pixels."*. then `screenX` is *"...the horizontal distance, in CSS pixels."* also `screen.width` and `screen.availWidth` is *"...(in pixels)..."* (not CSS pixels!!). However `screenX + screen.availWidth === screen.width` is `true` when `devicePixelRatio !== 1` Thus when ratio is not 1 or 2 dont use `devicePixelRatio` you can not be confident of alignment. Values other than 1,or max 2 and you start to mess with device designed presentation parameters, not everyone has perfect vision. – Blindman67 Feb 09 '20 at 05:45
That it works for you doesn't mean it will work on all other devices. Did you even follow the link? The whole point is that there is no way to do it today that works across devices and browsers. That it's working for your situation is luck. – gman Feb 09 '20 at 06:14
@gman There is only ambiguity when ratio is not an integer. One assumes that the canvas is sized to integer values you control that. You can know (apart from a few edge case) that you have pixel alignment when ratio is an int and you have sized the canvas to whole pixel values. – Blindman67 Feb 09 '20 at 07:10
No, you can't. There are 1000s of cases where something won't be an even number of pixels in size. Any element sized with %, em, rem, vh, vw, inside a flexbox, a grid, a table, all over. In my computer your user name is 68.66 CSS pixels wide. This comment box is 130.78 CSS pixels tall. The page itself is 8054.78 css pixels tall. – gman Feb 09 '20 at 08:48
@gman None of which are a canvas. Now on the *SAME* page, anywhere in body, in a visible element add ``. Find the red box, take a screen shot (via OS), open in image editor and measure the size of the red box. I know that if `devicePixelRatio` (dPR) was 1 that box WILL be 50 by 20px, if dPR was 2 that box WILL be 100 by 40 and the top left corner will be pixel aligned Any device, any display, any current browser. All you need is one example to prove your case. – Blindman67 Feb 09 '20 at 12:48
@gman ps Dont cheat by finding an element that will modify or clip its siblings size, that is easily discoverable, but the comment is not long enough to put all that in. – Blindman67 Feb 09 '20 at 12:57
1

The majority of WebGL apps don't set the size of the canvas in pixels, they set the size of the canvas to 100% of their container whether that container is the body or some sub-portion of the window or less. That's the norm. Your example is an exception. Further you keep mentioning DPR of 2. I already posted a list of other devices. Some have a DPI of 1.5, 1.8, 2.4, 2.5, 2.75, 3, 4 etc. There isn't only DPRs of 1 and 2 – gman Feb 09 '20 at 14:23
If and when the new API ships you'll be able to make the canvas match. Until then, the fact that it matches is for you is just luck. You happen to be on a device with an integer DPR and picking an exceptional layout instead for making it work for all cases. For example if you try to implement [this model viewer](https://modelviewer.dev/) who have no control over the size or alignment it will be at. That is the case that needs to be solved, not some minor exception – gman Feb 09 '20 at 14:37
And I use a Pixel 2 XL with a DPR of 3.5 so a canvas of size 11px could end up being 38 pixels or 39 depending on if the browser uses ceil, round, or floor. That is not specified by any spec and is up to the browser which is the entire reason an API is being added, so you can ask the browser what it did. – gman Feb 10 '20 at 03:25
@gman Totally agree with API, a step in the right direction if not 5 years late. Next is sub pixels. Clear type side by side with canvas text, canvas looks BAD (app killer bad), It can be duplicated (the API will help) if only there was a way to query the sub pixel layout. Quattron RGBY, (Y yellow) PenType RGBW (W white) and two versions of GRGB square and diamond layout, Stunning rendering solutions that makes clear type look so very last century, oh but a dream Call me clinical but W3C Screen Orientation Draft does not contain the word Pixel, *"square things right? not important"* LOL – Blindman67 Feb 10 '20 at 08:10
Another example is on my 175% scaled 4k Windows machine, devicePixelRatio is 1.7647058823529411. This is because it allows for treating the size as if it had the CSS pixels size of 1920*2 / devicePixelRatio = 2176 width, 1224 width. It's fairly viable to presnap canvases to device pixels given a little work: https://jdashg.github.io/misc/webgl/device-pixel-presnap.html – Jeff Gilbert Feb 11 '20 at 20:54

score 0 · Accepted Answer · answered Feb 08 '20 at 15:22

If you want your pictures to be drawn more smooth, then you could use mipmaps! Try to make the pictures to have both dimensions a power at 2 and after you can generate mipmaps, like this:

gl.generateMipmap(gl.TEXTURE_2D);

With mipmaps you can choose what WebGL does by setting the texture filtering for each texture. And if you want to your image to be smooth, then you have to change the texture filtering to be LINEAR_MIPMAP_LINEAR. So after you generate the mipmaps you have to write:

gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, gl.LINEAR_MIPMAP_LINEAR);

The pictures with the differences are below ( LINEAR vs LINEAR_MIPMAP_LINEAR ):

In first picture the tracks are asymmetric, but the second picture is drawn smooth. I think that the second picture looks more like the HTML picture that you provided.

There is a great tutorial about WebGL Textures: https://webglfundamentals.org/webgl/lessons/webgl-3d-textures.html. There is written that this type of texture filtering is the slowest and it takes 33% more memory, so be aware of that!