Await skipping and how to fix?

Question

I am using https://github.com/jimmywarting/StreamSaver.js to stream some geometry data into file, however I cannot get it to work with my limited Promise knowledge

        const fileStream = streamSaver.createWriteStream('export.obj')
        const writer = fileStream.getWriter()
        const encoder = new TextEncoder

        object.streamExportGeometry(async data => {    //Callback from streamExportGeometry with data to write. Called many times
            console.log("writerQueued");
            await new Promise((resolve, reject) => {
                writer.write(encoder.encode(data)).then(() => { setTimeout(resolve) })
            });
            console.log("writerDone");
        }, onProgress);
        writer.close()

I tried many variants of await, but it never waits for writer.write to finish. Console output looks like this

24x writerQueued 24x writerQueued exported finished 24x writerDone 24x writerDone writerDone

There are examples provided with this tool, but I cannot figure out how would I place promises for my code

Edit: added streamExportGeometry

 streamExportOBJ(writer, onProgressCallback) {
    var i, j, k, l, x, y, z;

    var vertices = this._vertices ? this._vertices.array : null;

    //Buffer object, to optimize amount of lines per write
    var writeBuffer = {
        _outputBuffer: "",
        _currBuffer: 0,
        _bufferLineLimit: 10000,
        _progress: 0,
        _expectedProgress: 1 + (vertices ? vertices.length / 3 : 0) + (uvs ? uvs.length / 2 : 0) + (normals ? normals.length / 3 : 0) + (indices ? indices.length / 3 : vertices.length / 3),
        writeLine: function (data) {
            this._outputBuffer += data;
            this._currBuffer++;
            if (this._currBuffer >= this._bufferLineLimit)
                this.flush();
        },
        flush: function () {
            if (this._outputBuffer) {
                writer(this._outputBuffer);
                this._outputBuffer = "";
                this._progress += this._currBuffer;
                this._currBuffer = 0;
                onProgressCallback(this._progress / this._expectedProgress * 100);
            }
        }
    }

    writeBuffer.writeLine('o export\n');

    //vertices
    if (vertices !== undefined && vertices && vertices.length >= 3) {
        for (i = 0; i < vertices.length; i += 3) {
            x = vertices[i];
            y = vertices[i + 1];
            z = vertices[i + 2];

            writeBuffer.writeLine('v ' + x + ' ' + y + ' ' + z + '\n');
        }
    }

    //Some more data..

   writeBuffer.flush();
 }

`setTimeout(resolve())` calls `resolve()` and passes the result to `setTimeout`, which tries to call that result as a function in the next event loop. You are likely looking for `setTimeout(resolve)`, or just `resolve()` as using `setTimeout` isn't likely to do much for you. — Heretic Monkey, Feb 12 '19 at 22:18
I tried many possible ways, without resolve as you suggested, always the same thing @HereticMonkey . It's clearly not duplicate — Jonzi, Feb 12 '19 at 22:24
You probably just want to do `await writer.write(encoder.encode(data));` I don't see any reason to create a Promise to wrap another promise. — Heretic Monkey, Feb 12 '19 at 22:33
Before you edited the code, it showed the same symptoms as the proposed duplicate. "clearly" is very much a stretch, considering you didn't mention in your question that you had tried those other methods previously. — Heretic Monkey, Feb 12 '19 at 22:35
@HereticMonkey I tried that originally, didn't work. Figured that writer.write might not return Promise value as suggested in this https://stackoverflow.com/questions/48617486/why-async-await-doesnt-work-in-my-case So I put Promise wrap around it. Also tried combinations with timeout as suggested in example. I tried changing example and It didn't work without timeout, so thats why I (tried) to use it. — Jonzi, Feb 12 '19 at 22:39
Consider [edit]ing your question to include what you've tried and what doesn't work so that others don't waste their time asking whether you've tried something. — Heretic Monkey, Feb 12 '19 at 22:40
@HereticMonkey I have been trying different things for hours and I can't list everything I tried — Jonzi, Feb 12 '19 at 22:42
@HereticMonkey I was hoping for someone to point out why it isn't working. Code is short and so are examples I linked. Could you look there, if I am doing the right thing? — Jonzi, Feb 12 '19 at 22:46
You're calling `writer.close()` BEFORE you are done writing. Your code doesn't wait for `object.streamExportGeometry()` to finish before executing `writer.close()`. So your stream ends up closed before you write to it. — jfriend00, Feb 12 '19 at 23:38
To really solve this problem, we'd need to see a larger context for the code (the containing function) and know more about the `streamExportGeometry()` method and how it handles error conditions and what exactly it does with its callbacks (like does it call them only once or multiple times) and to see how you'd know when it was done. Ideally, we'd be able to see the source code for that method. — jfriend00, Feb 12 '19 at 23:50
Also, it appears from your log messages that the code you show is inside some sort of loop. We need to see the code driving the loop too. Basically, you haven't included enough code or enough information about functions you're calling for us to know how to advise further. The problem can't be solved only with the information provided so far. — jfriend00, Feb 12 '19 at 23:50
FYI, I'm willing to help you rewrite this if you can provide the info I've asked for. — jfriend00, Feb 12 '19 at 23:54
@jfriend00 Hey, sure I'd love if you helped me understand this. I will post edit in a moment. — Jonzi, Feb 13 '19 at 00:01
Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/188310/discussion-between-jonzi-and-jfriend00). — Jonzi, Feb 13 '19 at 00:07
@jfriend00 added the export method. There is no loop for main method, just called multiple times by export method. The program itself works, but the problem is that writers are not being awaited and with big files memory heap eventually runs out and browser window crashes — Jonzi, Feb 13 '19 at 00:10
OK, this is really confusing. It appears that `streamExportOBJ()` is trying to behave synchronously (just a regular `for` loop), but it's calling a callback that is asynchronous so it's no wonder that things pile up and nothing waits for anything else. Some meaningful rewrite is going to be needed and promises should be used. Will have to think about it. — jfriend00, Feb 13 '19 at 00:29
@jfriend00 Yeah, I have been trying to force callback to become synchronous with my awaits and promises, but not successfully. I don't understand fully, why is it acting asynchronously ? Because of the writer.write method? There are examples how to do this on tool's github, but I couldn't replicate them.. — Jonzi, Feb 13 '19 at 00:36
@jfriend00 I had to add async and await in callback, since all that writer.write does is `channel.port1.postMessage(chunk)` . So If I call that too quickly they get queued up and I run out of memory — Jonzi, Feb 13 '19 at 00:40

score 1 · Accepted Answer · answered Feb 13 '19 at 00:46

This is what I would suggest. You have streamExportOBJ() which is trying to behave synchronously, but it's calling a writer method that is actually asynchronous so there was no way for streamExportOBJ() to actually know when any of the asynchronous stuff it was calling was done. So, I made the callback you pass to streamExportOBJ() have an asynchronous interface and then streamExportOBJ() can await it.

I'm not entirely sure what you want to do with error handling here. If any errors occur in writer.write(), then the whole process is aborted and the error percolates back up to your top level where a catch(e) {} block will get it. You could develop different strategies there.

async streamExportOBJ(writer, onProgressCallback) {
    var i, j, k, l, x, y, z;

    var vertices = this._vertices ? this._vertices.array : null;

    //Buffer object, to optimize amount of lines per write
    var writeBuffer = {
        _outputBuffer: "",
        _currBuffer: 0,
        _bufferLineLimit: 10000,
        _progress: 0,
        _expectedProgress: 1 + (vertices ? vertices.length / 3 : 0) + (uvs ? uvs.length / 2 : 0) + (normals ? normals.length / 3 : 0) + (indices ? indices.length / 3 : vertices.length / 3),

        writeLine: async function (data) {
            this._outputBuffer += data;
            this._currBuffer++;
            if (this._currBuffer >= this._bufferLineLimit)
                return this.flush();
        },

        flush: async function () {
            if (this._outputBuffer) {
                await writer(this._outputBuffer);
                this._outputBuffer = "";
                this._progress += this._currBuffer;
                this._currBuffer = 0;
                onProgressCallback(this._progress / this._expectedProgress * 100);
            }
        }
    }

    await writeBuffer.writeLine('o export\n');

    //vertices
    if (vertices !== undefined && vertices && vertices.length >= 3) {
        for (i = 0; i < vertices.length; i += 3) {
            x = vertices[i];
            y = vertices[i + 1];
            z = vertices[i + 2];

            await writeBuffer.writeLine('v ' + x + ' ' + y + ' ' + z + '\n');
        }
    }

    //Some more data..

   return writeBuffer.flush();
}

async function someFunction() {
    const fileStream = streamSaver.createWriteStream('export.obj');
    const writer = fileStream.getWriter();
    const encoder = new TextEncoder;

    try {
        await object.streamExportGeometry(async data => {
            console.log("writerQueued");
            await writer.write(encoder.encode(data));
            console.log("writerDone");
        }, onProgress);
    } catch(e) {
        // not sure what you want to do here when there 
        // was an error somewhere in object.streamExportGeometry()
    } finally {
        // always close
        writer.close()
    }
}

FYI, if this were my code, I'd move all the writeBuffer functionality into its own class and get that out of streamExportObj. It appears to be generic buffering functionality and can be implemented/tested separately and then the logic of streamExportOBJ() will appear simpler to understand and follow.

Sorry, still exactly the same. All the writes are called before any of the are done. — Jonzi, Feb 13 '19 at 01:00
@Jonzi - Then perhaps `writer.write(encoder.encode(data));` is not waiting to resolve its returned promise before it's actually done writing. Nothing I can do about that here since I don't have any code for that to look at. ALL asynchronous operations have to correctly return a promise that is only resolved when they are done for you to properly sequence stuff. You can't `await` a promise that lies to you. — jfriend00, Feb 13 '19 at 01:22
@Jonzi - If this https://github.com/jimmywarting/StreamSaver.js/blob/master/StreamSaver.js#L97 is the code for the `.write()` operation, then it does not appear to be returning a promise at all and your original ` writer.write(encoder.encode(data)).then(() => { setTimeout(resolve) })` was flawed in trying to use `.then()`. In order to sequence things, you will have to fix `.write()` to return a promise that is resolved ONLY when the write is actually done. — jfriend00, Feb 13 '19 at 01:27
@Jonzi - FYI, the comments on that `write()` code appear to list as a todo item that it doesn't handle backpressure (writing too much at once) and that needs to be added. — jfriend00, Feb 13 '19 at 01:28
after few more hours I got it to work, similar to your answer. Thanks for your effort :) — Jonzi, Feb 13 '19 at 12:05

Await skipping and how to fix?

1 Answers1