0

I'm writing a console application that collects data from the websites (doing scraping). I make a webjob application in Visual Studio and contains my console application there and then I publish it in Azure App Service.

I tried to change settings in Web.config of my web job by adding

  <add key="SCM_COMMAND_IDLE_TIMEOUT" value="3600" />
  <add key="WEBJOBS_IDLE_TIMEOUT" value="3600" />

Also, I try to find where I can change the configuration of my web job app in the Azure portal and not really find it. reference to post Azure WebJob timeout configuration settings

This is the error that I received when I run it:

[08/19/2019 14:16:14 > 5a8697: SYS INFO] Status changed to Running
[08/19/2019 14:20:16 > 5a8697: ERR ] Command 'cmd /c ""MainApp.ex ...' was 
aborted due to no output nor CPU activity for 121 seconds. You can 
increase the SCM_COMMAND_IDLE_TIMEOUT app setting (or WEBJOBS_IDLE_TIMEOUT 
if this is a WebJob) if needed.
cmd /c ""MainApp.exe""
[08/19/2019 14:20:16 > 5a8697: SYS INFO] Status changed to Failed

example of scraping code:

public SiteProcessingResult ScrapeSite(string url, int siteId)
    {
        SiteProcessingResult result = new SiteProcessingResult();
        int count = 0;
        WebClient webRequest = new WebClient();
        webRequest.Encoding = Encoding.UTF8;
        string mainSiteHTML = webRequest.DownloadString(url);
        mainSiteHTML = ScrapeHelper.RemoveComments(mainSiteHTML);

        mainSiteHTML = mainSiteHTML.Substring(mainSiteHTML.IndexOf("<div class=\"wrap content\">"));
        mainSiteHTML = mainSiteHTML.Remove(mainSiteHTML.IndexOf("<footer"));

        string depReg = string.Format("{0}.*?{1}", "<div class=\"row\"", "@example.org</span>");
        MatchCollection matchList = Regex.Matches(mainSiteHTML, depReg, RegexOptions.IgnoreCase | RegexOptions.ExplicitCapture | RegexOptions.Singleline);

        foreach (Match match in matchList)
        {
            string articleUrl = GetArticleUrl(match.ToString(), url);
            string title = GetArticleTitle(match.ToString());
            string organization = "";
            string date = "";
            string locationString = GetLocationFromArticle(match.ToString());
            string content = GetContentFromArticle(match.ToString());
            string state = GetState(locationString);

            SitePost record = SitePost.CreateSiteRecord(date, content, title, articleUrl, jobTitle, organization, locationString, state, null, CultureInfo.CreateSpecificCulture("en-US"));
            result.Records.Add(record);
        }

        return result;
    }

example of sending mail:

 public void SendEmail(SiteTypeEnum siteType, string subject, string message, bool isHtml = false, string sendTo = null)
    {
        string userName, password, to;
        GetEmailLoginDetails(siteType, out userName, out password, out to);
        if (sendTo == null)
        {
            sendTo = to;
        }
        LoggerHelper.WriteInfo("Send email, phase {0}, username {1}, password {2}, to {3}, subject {4}", siteType, userName, password, sendTo, subject);
        Logic.Helper.EmailGoogle.SendMessage(sendTo, string.Empty, string.Empty, subject, message, isHtml, null, userName, password, userName, userName);
        //SendEmailDev(phase2, subject, message, isHtml);
    }
DisappointedByUnaccountableMod
  • 6,656
  • 4
  • 18
  • 22
ISTech
  • 51
  • 4
  • Can you compare your app to the walkthrough in the [MS Docs](https://learn.microsoft.com/en-us/azure/app-service/webjobs-sdk-get-started)? The way I would start debugging is to try and get the example up and running, and then gradually modify the code in the direction of your application. If you can tell us what specific modification causes this error (as compared to the tutorial) that will be of great help. – chill94 Aug 19 '19 at 18:05
  • And here's a link to a [similar SO question](https://stackoverflow.com/questions/25489467/azure-webjob-command-timeout?rq=1) – chill94 Aug 19 '19 at 18:10

2 Answers2

0

The default configuration in azure portal doesn't include these key-value,you won't find them there so you need to add it.

In the portal, go to your app service, select Configuration under Settings then choose New application setting. And set your key-value there.

enter image description here

George Chen
  • 13,703
  • 2
  • 11
  • 26
  • Thanks. Yesterday I changed this configuration, turn on 'Alway on' option and turn on Websocket but without success. The application was running 1 hour and after this failed with the same error ..was aborted due to no output nor CPU activity for 3600 seconds.. – ISTech Aug 20 '19 at 07:09
  • If the job is long-running, you could set a big enough number and make sure your webjob will finish in this time. Another way is to implement simple writing to log, which job is running. – George Chen Aug 20 '19 at 07:59
  • And what's your webjob type? Maybe you could paste some code. – George Chen Aug 20 '19 at 08:00
  • WebJob is a console application with DAL architecture (Data Access Layer) example: – ISTech Aug 20 '19 at 09:46
  • I mean continuous or trigger one. – George Chen Aug 20 '19 at 09:47
  • No matter. I can use triggered or use Azure function to schedule this. P.S. I updated my post. – ISTech Aug 20 '19 at 09:55
0

I found the solution just added the file in my console application called:

settings.job

settings file

set schedule value in the file before publishing:

{
  "schedule": "0 */5 * * * *"
}

notice: The web job must be OnDemand or Triggered type.

ISTech
  • 51
  • 4