21

In a 3D engine I'm working on I've succesfully managed to draw a cube in 3D. The only method to fill the sides is using either a solid color or gradient as far as I'm concerned. To make things more exciting, I'd really love to implement texture mapping using a simple bitmap.

The point is that I can hardly find any articles or code samples on the subject of image manipulation in JavaScript. Moreover, image support in HTML5 canvas seems to be restricted to cropping.

How could I go about stretching a bitmap so that a rectangular bitmap can fill up a unregular cube face? In 2D, a projected square cube face is, due to perspective, not of a square shape, so I'll have to stretch it to make it fit in any quadrilateral.

Hopefully this image clarifies my point. The left face is now filled up with a white/black gradient. How could I fill it with a bitmap, after it has been texture-mapped?

Cube

Does anyone have any tips on perspective texture mapping (or image manipulation at all) using JavaScript and HTML5 Canvas?

Edit: I got it working, thanks to 6502!

It is, however, rather CPU intensive so I'd love to hear any optimization ideas.

Result using 6502's technique - Texture image used

pimvdb
  • 151,816
  • 78
  • 307
  • 352

1 Answers1

48

I think you will never get an accurate result... I spent some time investigating how to do 3d graphics using canvas 2d context and I found it viable to do texture mapping gouraud shading by computing appropriate 2d gradients and matrices:

  • Solid polygons are of course easy
  • Gouraud filling is possible only on one component (i.e. you cannot have a triangle where every vertex is an arbitrary RGB filled with bilinear interpolation, but you can do that filling using for example three arbitrary shades of a single color)
  • Linear texture mapping can be done using clipping and image drawing

I would implement perspective-correct texture mapping using mesh subdivision (like on PS1).

However I found many problems... for example image drawing with a matrix transform (needed for texture mapping) is quite inaccurate on chrome and IMO it's impossible to get a pixel-accurate result; in general there is no way to turn off antialiasing when drawing on a canvas and this means you will get visible see-through lines when subdividing in triangles. I also found multipass rendering working really bad on chrome (probably because of how hw-accellerated rendering is implemented).

In general this kind of rendering is surely a stress for web browsers and apparently these use cases (strange matrices for example) are not tested very well. I was even able to get Firefox crashing so bad that it took down the whole X susbsystem on my Ubuntu.

You can see the results of my efforts here or as a video here... IMO is surely impressing that this can be done in a browser without using 3D extensions, but I don't think current problems will be fixed in the future.

Anyway the basic idea used to draw an image so that the 4 corners ends up in specific pixels position is to draw two triangles, each of which will use bilinear interpolation.

In the following code I assume you have a picture object texture and 4 corners each of which is an object with fields x,y,u,v where x,y are pixel coordinates on the target canvas and u,v are pixel coordinates on texture:

function textureMap(ctx, texture, pts) {
    var tris = [[0, 1, 2], [2, 3, 0]]; // Split in two triangles
    for (var t=0; t<2; t++) {
        var pp = tris[t];
        var x0 = pts[pp[0]].x, x1 = pts[pp[1]].x, x2 = pts[pp[2]].x;
        var y0 = pts[pp[0]].y, y1 = pts[pp[1]].y, y2 = pts[pp[2]].y;
        var u0 = pts[pp[0]].u, u1 = pts[pp[1]].u, u2 = pts[pp[2]].u;
        var v0 = pts[pp[0]].v, v1 = pts[pp[1]].v, v2 = pts[pp[2]].v;

        // Set clipping area so that only pixels inside the triangle will
        // be affected by the image drawing operation
        ctx.save(); ctx.beginPath(); ctx.moveTo(x0, y0); ctx.lineTo(x1, y1);
        ctx.lineTo(x2, y2); ctx.closePath(); ctx.clip();

        // Compute matrix transform
        var delta = u0*v1 + v0*u2 + u1*v2 - v1*u2 - v0*u1 - u0*v2;
        var delta_a = x0*v1 + v0*x2 + x1*v2 - v1*x2 - v0*x1 - x0*v2;
        var delta_b = u0*x1 + x0*u2 + u1*x2 - x1*u2 - x0*u1 - u0*x2;
        var delta_c = u0*v1*x2 + v0*x1*u2 + x0*u1*v2 - x0*v1*u2
                      - v0*u1*x2 - u0*x1*v2;
        var delta_d = y0*v1 + v0*y2 + y1*v2 - v1*y2 - v0*y1 - y0*v2;
        var delta_e = u0*y1 + y0*u2 + u1*y2 - y1*u2 - y0*u1 - u0*y2;
        var delta_f = u0*v1*y2 + v0*y1*u2 + y0*u1*v2 - y0*v1*u2
                      - v0*u1*y2 - u0*y1*v2;

        // Draw the transformed image
        ctx.transform(delta_a/delta, delta_d/delta,
                      delta_b/delta, delta_e/delta,
                      delta_c/delta, delta_f/delta);
        ctx.drawImage(texture, 0, 0);
        ctx.restore();
    }
}

Those ugly strange formulas for all those "delta" variables are used to solve two linear systems of three equations in three unknowns using Cramer's method and Sarrus scheme for 3x3 determinants.

More specifically we are looking for the values of a, b, ... f so that the following equations are satisfied

a*u0 + b*v0 + c = x0
a*u1 + b*v1 + c = x1
a*u2 + b*v2 + c = x2

d*u0 + e*v0 + f = y0
d*u1 + e*v1 + f = y1
d*u2 + e*v2 + f = y2

delta is the determinant of the matrix

u0  v0  1
u1  v1  1
u2  v2  1

and for example delta_a is the determinant of the same matrix when you replace the first column with x0, x1, x2. With these you can compute a = delta_a / delta.

Nightfirecat
  • 11,432
  • 6
  • 35
  • 51
6502
  • 112,025
  • 15
  • 165
  • 265
  • 1
    Wow, your torus application is impressive... I don't mind if it's not exact on a pixel basis. Thanks for the great answer. – pimvdb Jan 23 '11 at 14:32
  • 2
    Ok... I'll add the texture mapping part to the answer (the source of torus.html is hard to read because I "had" to shrink it down to 4K). – 6502 Jan 23 '11 at 14:36
  • Just looked at the source and it was indeed inscrutable, very nicely done! – pimvdb Jan 23 '11 at 14:40
  • Hehehe... that's the result of a google javascript compiler and of a custom packer I coded to be able to get down to 4k. There are no logical reasons for that... simply that this torus program is a javascript remake of an old 4K demo I wrote in assembler many years ago. – 6502 Jan 23 '11 at 15:36
  • Wonderful, it's a little bit above my maths level, but I'm playing with it and it works beautifully! It only gets a little distorted if the image is rather large, using two triangles only. But it seems I can also rather easily change that. Thank you so extremely much! Wish I could vote more than 1... – pimvdb Jan 23 '11 at 16:15
  • I've used this in my Html5 game framework and it works spectacularly. Wonderfully complete answer! Took me just a few minutes to understand the textureMap method enough to re-implement it in GWT. :D – Emperorlou Aug 24 '11 at 02:34
  • I'm currently drawing a ` – pimvdb Sep 02 '11 at 20:51
  • It took me a while to understand your code, but I implemented to my HTML5/Canvas 3D engine with almost no modifications. Thanks a lot, I doubt I could've done it without this. BTW how many lines of assembly did it take you to write this? – Solenoid Sep 17 '11 at 13:40
  • @Solenoid: the source code is about 4000 lines, but it's bad code and with a lot of commented out sections (also IIRC the Phong part is not a fake phong based on texture). Probably half that line count could be enough. The assembler source code for the program and the DOS .com file can be downloaded from my dead website in this page http://www.gripho.it/demo.en.html (note that comments are few and in Italian). It runs fine with dosbox under linux, no idea if it still works on windows today or not. – 6502 Sep 17 '11 at 14:35
  • Did I missed something? For mee it's not working. See: http://jsfiddle.net/j2LfS/ – Tarion May 12 '14 at 19:39
  • Wow, great work! I see you derived the transform from a system of equations. Where did that system of equations come from? – William Jun 12 '23 at 22:29
  • 1
    @William: bilinear interpolation means `value=ax+by+c`, if doing rendering in software at pixel level you'd need pixel at position `(x₀, y₀)` to be `(u₀, v₀)` in texture coordinates, i.e. you'de be looking for a, b,... f so that `ax₀+by₀+c=u₀` and `dx₀+ey₀+f=v₀` and similar equations for other two points. Using canvas drawImage however requires providing the inverse, so the equations are for the inverse problems (i.e. given u,v find x,y). – 6502 Jun 13 '23 at 05:57