The way to determine whether one object is before the other requires a bit more linear algebra.
First of all, I would suggest to translate the coordinates from the "world" coordinates to the "view" 2D coordinates, i.e. to the rows and columns of the display.
Note also that the original Z coordinate does not influence the sort order (imagine that an object would be lifted up along the Z axis: we can find a sort order where this move would not have any impact). So the above-mentioned translation could assume all points are at Z=0.
Let's take this set-up, but depicted from "above", so when looking along the Z axis down to the game floor:

In the picture there are 7 objects, numbered from 0 to 6. The line of view in the game would be from the bottom-left of this picture. The coordinate system in which I would suggest to translate some points is depicted with the red row/col axis.
The white diagonals in each object link the two points that would be translated and used in the algorithm. The assumption is that when one object is in front of another, their diagonal lines will not intersect. If they would, it would mean that objects are overlapping each other in the game world, which would mean they are like gasses, not solids :) I will assume this is not the case.
One object A could be in front of another object B when in the new coordinate system, the left-most column coordinate of B falls between the two column coordinates of A (or vice versa). There might not really be such an overlap when their Z coordinates differ enough, but we can ignore that, because when there is no overlap we can do no harm in specifying a certain order anyway.
Now, when the coordinates indicate an overlap, the coordinates of diagonals (of A and B) must be compared with some linear algebra formula, which will determine which one is in front of the other.
Here is your adapted function that does that:
topologicalSort() {
// Exit if sorting is a non-operation
if (this.Stage.children.length < 2) return;
// Add two translated coordinates, where each of the resulting
// coordinates has a row (top to bottom) and column
// (left to right) part. They represent a position in the final
// rendered view (the screen).
// The two pairs of coordinates are translations of the
// points (posX + sizeX, Y, 0) and (posX, posY + sizeY, 0).
// Z is ignored (0), since it does not influence the order.
for (let obj of this.Stage.children) {
obj.leftCol = obj.posY - obj.posX - obj.sizeX;
obj.rightCol = obj.posY - obj.posX + obj.sizeY;
obj.leftRow = obj.posY + obj.posX + obj.sizeX;
obj.rightRow = obj.posY + obj.posX + obj.sizeY;
obj.isoSpritesBehind = [];
}
for(let i = 0; i < this.Stage.children.length; i++) {
let a = this.Stage.children[i];
// Only loop over the next objects
for(let j = i + 1; j < this.Stage.children.length; j++) {
let b = this.Stage.children[j];
// Get the two objects in order of left column:
let c = b.leftCol < a.leftCol ? b : a;
let d = b.leftCol < a.leftCol ? a : b;
// See if they overlap in the view (ignoring Z):
if (d.leftCol < c.rightCol) {
// Determine which is behind: some linear algebra
if (d.leftRow <
(d.leftCol - c.leftCol)/(c.rightCol - c.leftCol)
* (c.rightRow - c.leftRow) + c.leftRow) {
// c is in front of d
c.isoSpritesBehind.push(d);
} else { // d is in front of c
d.isoSpritesBehind.push(c);
}
} // in the else-case it does not matter which one comes first
}
}
// This replaces your visitNode function and call:
this.Stage.children.forEach(function getDepth(obj) {
// If depth was already assigned, this node was already visited
if (!obj.isoDepth) {
// Get depths recursively, and retain the maximum of those.
// Add one more to get the depth for the current object
obj.isoDepth = obj.isoSpritesBehind.length
? 1+Math.max(...obj.isoSpritesBehind.map(getDepth))
: 1; // Depth when there is nothing behind it
}
return obj.isoDepth; // Return it for easier recursion
});
// Sort like you did, but in shorter syntax
this.Stage.children.sort((a, b) => a.isoDepth - b.isoDepth);
}
I add a snippet where I completed the class with a minimum of code, enough to make it run and output the final order in terms of object index numbers (as they were originally inserted):
class Game {
constructor() {
this.Stage = { children: [] };
}
addObject(posX, posY, posZ, sizeX, sizeY, sizeZ) {
this.Stage.children.push({posX, posY, posZ, sizeX, sizeY, sizeZ,
id: this.Stage.children.length}); // add a unique id
}
topologicalSort() {
// Exit if sorting is a non-operation
if (this.Stage.children.length < 2) return;
// Add two translated coordinates, where each of the resulting
// coordinates has a row (top to bottom) and column
// (left to right) part. They represent a position in the final
// rendered view (the screen).
// The two pairs of coordinates are translations of the
// points (posX + sizeX, Y, 0) and (posX, posY + sizeY, 0).
// Z is ignored (0), since it does not influence the order.
for (let obj of this.Stage.children) {
obj.leftCol = obj.posY - obj.posX - obj.sizeX;
obj.rightCol = obj.posY - obj.posX + obj.sizeY;
obj.leftRow = obj.posY + obj.posX + obj.sizeX;
obj.rightRow = obj.posY + obj.posX + obj.sizeY;
obj.isoSpritesBehind = [];
}
for(let i = 0; i < this.Stage.children.length; i++) {
let a = this.Stage.children[i];
// Only loop over the next objects
for(let j = i + 1; j < this.Stage.children.length; j++) {
let b = this.Stage.children[j];
// Get the two objects in order of left column:
let c = b.leftCol < a.leftCol ? b : a;
let d = b.leftCol < a.leftCol ? a : b;
// See if they overlap in the view (ignoring Z):
if (d.leftCol < c.rightCol) {
// Determine which is behind: some linear algebra
if (d.leftRow <
(d.leftCol - c.leftCol)/(c.rightCol - c.leftCol)
* (c.rightRow - c.leftRow) + c.leftRow) {
// c is in front of d
c.isoSpritesBehind.push(d);
} else { // d is in front of c
d.isoSpritesBehind.push(c);
}
} // in the else-case it does not matter which one comes first
}
}
// This replaces your visitNode function and call:
this.Stage.children.forEach(function getDepth(obj) {
// If depth was already assigned, this node was already visited
if (!obj.isoDepth) {
// Get depths recursively, and retain the maximum of those.
// Add one more to get the depth for the current object
obj.isoDepth = obj.isoSpritesBehind.length
? 1+Math.max(...obj.isoSpritesBehind.map(getDepth))
: 1; // Depth when there is nothing behind it
}
return obj.isoDepth; // Return it for easier recursion
});
// Sort like you did, but in shorter syntax
this.Stage.children.sort((a, b) => a.isoDepth - b.isoDepth);
}
toString() { // Just print the ids of the children
return JSON.stringify(this.Stage.children.map( x => x.id ));
}
}
const game = new Game();
game.addObject( 2, 2, 0, 1, 1, 1 );
game.addObject( 1, 3, 0, 3, 1, 1 );
game.addObject( 6, 1, 0, 1, 3, 1 );
game.addObject( 9, 3, 0, 1, 1, 1 );
game.addObject( 5, 3, 0, 1, 3, 1 );
game.addObject( 7, 2, 0, 1, 1, 1 );
game.addObject( 8, 2, 0, 3, 1, 1 );
game.topologicalSort();
console.log(game + '');
The objects in the snippet are the same as in the picture with the same numbers. The output order is [0,1,4,2,5,6,3]
which is the valid sequence for drawing the objects.