I'm making backend in c# with ASP.Net Core. When the client sends HTTP POST I want to start async task that can run for days so I just want to Fire and Forget, I'm doing it that whit Task.Factory.StartNew(), but for some reason every time it runs it just stops randomly, sometimes after several hours sometimes after 30 seconds. You still can create another HTTP POST and start the async but it just won't complete.
[HttpPost]
public async Task<ActionResult<ScrapeCreateDto>> CreateScraper(ScrapeCreateDto scrape)
{
Scrape scrapeModel = new Scrape();
scrapeModel = (Scrape)_mapper.Map(scrape, scrapeModel, typeof(ScrapeCreateDto), typeof(Scrape));
if (!await _validation.ValidateURL(scrapeModel.Url)) return NotFound();
Console.WriteLine($"Going to play {scrapeModel.Url} {scrapeModel.Views} times");
var SoundCloudUrl = new ScrapeUrl();
Task ScrapeSoundCloud = SoundCloudUrl.ScrapeHtml(scrapeModel);
Task.Factory.StartNew(() => ScrapeSoundCloud); // Fire and forget -- it's now stopping after something about 170 mins
return Ok();
}
public class ScrapeUrl{
public async Task ScrapeHtml(Scrape scrape)
{
var scrapeSoundCloud = await ScrapeSoundCloud(scrape);
}
private async Task<string> ScrapeSoundCloud(Scrape scrape)
{
string fullUrl = scrape.Url;
int prevViews = 0;
var options = new LaunchOptions()
{
Headless = true,
ExecutablePath = "C:\\Program Files\\Google\\Chrome\\Application\\chrome.exe",
Args = new string[] { "--incognito"}
};
for (int i = 0; i < scrape.Views; i++)
{
using (var browser = await Puppeteer.LaunchAsync(options, null, PuppeteerSharp.Product.Chrome))
{
using (var page = await browser.NewPageAsync())
{
// this is taking about half a minute but it runs thousand of times
}
}
await Task.Delay(3000);
}
Console.WriteLine("Done");
return "Done";
}
}
I know that right now it's uselless to call ScrapeSoundCloud from another task but later I want to start more things from ScrapeHtml. Also I know that with async there can be problem with errors bacause it sometimes isn't throwing exceptions, but I was printing to console, getting errors and everything, so it didn't throw any error and just stopped.
EDIT Answers are in comments. And this has been related to similar threads.