How can I hold on http calls for a while?

Question

I have the following problem and I would appreciate if someone could send me an idea, I have tried some, but it did not work. Consider the code:

 while (this.fastaSample.length > 0) {
      this.datainputService
        .saveToMongoDB(this.fastaSample.slice(0, batch))
        .subscribe();
    }

It supposes to solve the issue that I cannot send my data in a single http call, since it is too big, I was able to send 10% without issue, more than that, it does not work! So I thought, I should send smaller batches, and I have consulted sone Q&As here, and they helped me, but did not solve the problem.

I have tried to use await as I did in node, but it does not work; it sends all the http at once, it would be nice to stop/hold the code until the last http call is complete, that would be nice! Any suggestion?

For one: I don't see any code that shows how you're sending data, nor how you're accepting data. A normal multi-part POST will stay alive for as long as necessary to finish an upload, so how does your endpoint differ? — Mike 'Pomax' Kamermans, Mar 10 '20 at 18:15

Poul Kruijt · Accepted Answer · 2020-03-10T18:48:07.300

1

I suppose you could make it all nice and rxjs by using from and concatAll:

_{untested code}

// first create batches by chunking the array
const batches = Array.from(
  { length: Math.ceil(fastaSample.length / batch) },
  (v, i) => fastaSample.slice(i * batch, i * batch + batch)
)

// Second go over these chunks using `from` and `concatAll`:
from(batches).pipe(
  map((batch) => this.data.inputService.saveToMongoDB(batch)),
  concatAll()
).subscribe();

This will make the calls consecutively. If it's possible to do the requests at the same time, you can do mergeAll().

But like @Mike commented, it seems like the issue should be handled in the MongoDB backend and accept a multipart request. This way you don't need to chunk stuff

edited Mar 10 '20 at 18:48

answered Mar 10 '20 at 18:12

Poul Kruijt

69,713
12
145
149

Hey there, thanks, I am testing here. If I do not chunk, the Angular page crashes, and the node app says it ran out of memory, and crashes also. Chunking the data is not my favority choice, it is too emperical, but it is the best I could come up with. When you say to use mergeall, instead of map? – Jorge Guerra Pires Mar 10 '20 at 18:58
1

@JorgePires instead of `concatAll`. With concatAll it sends the next request once the first has finished, and with mergeAll it just calls them all at once. I think. About the running out of memory, that's also because the data being send is not a stream, as it would be with a multipart request. So my suggestion is still to look into this. How does your `saveToMongoDB` method looks like? – Poul Kruijt Mar 10 '20 at 19:07
mergeall crashed. – Jorge Guerra Pires Mar 10 '20 at 19:11
and concatAll() seemed to work, but it gives an strange behavior, the first http call works fine, but the following ones does not work, it “keeps waiting”, like on hold. – Jorge Guerra Pires Mar 10 '20 at 19:14
I am taking a look at the documentation: https://www.learnrxjs.io/learn-rxjs/operators/transformation/concatmap – Jorge Guerra Pires Mar 10 '20 at 19:17
1

@JorgePires so you do see the 2nd network request, but it's hanging? Also the operator is concatAll, not concatMap. You've linked the wrong one :) – Poul Kruijt Mar 10 '20 at 19:34
I have sent this link because I read that this option can be better, then I want to test. I was researching, maybe a need what Mike said, multipart http, I did not know that. See here a nice Q&A: https://stackoverflow.com/questions/16958448/what-is-http-multipart-request I wanted a simple approach, and yours seemed okay – Jorge Guerra Pires Mar 10 '20 at 19:36
so you do see the 2nd network request, but it's hanging? yet, that is right! I have not idea why – Jorge Guerra Pires Mar 10 '20 at 19:37
Should I send back a response from the server? – Jorge Guerra Pires Mar 10 '20 at 19:44
it is sending back the error Failed to load resource: net::ERR_EMPTY_RESPONSE – Jorge Guerra Pires Mar 10 '20 at 19:49
it happens after the first http is successful, I can see in the serve – Jorge Guerra Pires Mar 10 '20 at 19:49
Hi @PierreDuc I have tried several times your code, and the problems persists, thanks for your kindness in providing the code, I shall try another approach, not as good as I have thought, but it should work! Thanks again! – Jorge Guerra Pires Mar 11 '20 at 10:00
1

@JorgePires multipart is definitely the way to go. It's actually the simplest solution, and won't do any UI blocking. Good luck! – Poul Kruijt Mar 11 '20 at 17:43
Hi there, I have decided to avoid these, nonetheless, I have done a research, and multipart does not seem simple, any reference suggestion? – Jorge Guerra Pires Mar 11 '20 at 19:04
I am using the http call just to let the nodejs knows that it can save the dataset. I keep getting back the error Failed to load resource: net::ERR_EMPTY_RESPONSE , nonetheless, it saves, I have done several researches, and no success to solve it, maybe this is the problem! – Jorge Guerra Pires Mar 11 '20 at 19:05
it seems a core problem, but I am using cors(), and it just appears on the second call, it makes no sense! – Jorge Guerra Pires Mar 11 '20 at 19:06

How can I hold on http calls for a while?

1 Answers1

Linked