Get the content of website using HttpClient but without async method

Question

I am trying to fetch the website content using `httpclinet as you can see here

public async Task<List<NewsContent>> parsing(string newsArchive)
{
    List<NewsContent> lstResult=new List<NewsContent>();
    HttpClient http = new HttpClient();
    var response = await http.GetByteArrayAsync(newsArchive);
    String source = Encoding.GetEncoding("utf-8").GetString(response, 0, response.Length - 1);
    source = WebUtility.HtmlDecode(source);
    HtmlDocument resultat = new HtmlDocument();
    resultat.LoadHtml(source);

    List<HtmlNode> toftitle = resultat.DocumentNode.Descendants().Where
    (x => (x.Name == "div" && x.Attributes["class"] != null && x.Attributes["class"].Value.Contains("news_list"))).ToList();
    var li = toftitle[0].Descendants().Where
    (x => (x.Name == "div" && x.Attributes["class"] != null && x.Attributes["class"].Value=="news_item")).ToList();

    foreach (var item in li)
    {
        NewsContent newsContent = new NewsContent();
        newsContent.Url = item.Descendants("a").ToList()[0].GetAttributeValue("href", null);
        newsContent.Img = item.Descendants("img").ToList()[0].GetAttributeValue("src", null);
        newsContent.Title = item.Descendants("h2").ToList()[0].InnerText;

        //finding main news content
        var response1 = await http.GetByteArrayAsync("http://www.nsfund.ir/news" + newsContent.Url);
        String source1 = Encoding.GetEncoding("utf-8").GetString(response1, 0, response1.Length - 1);
        source1 = WebUtility.HtmlDecode(source1);
        HtmlDocument resultat1 = new HtmlDocument();
        resultat1.LoadHtml(source1);
        newsContent.Content = resultat1.DocumentNode.SelectSingleNode("//div[@class='news_content_container']").InnerText;


    }
    return  lstResult;
}

As you can see i used async method to get the data .here :

var response = await http.GetByteArrayAsync(newsArchive);

But the problem is when i call my async function :

News newagent = new News();
Task<List<NewsContent>> lst = newagent.parsing("http://www.nsfund.ir");
Task.WaitAll(lst);
List<NewsContent> enresult = lst.Result;

I don't get any result.so i decided to convert this async function to a normal function ,what kind of code should be replaced with this :

var response = await http.GetByteArrayAsync(newsArchive);

Try `var response = http.GetByteArrayAsync(newsArchive).Result;`. It probably won't help though. — Enigmativity, Jan 18 '16 at 05:05

score 3 · Accepted Answer · answered Jan 18 '16 at 05:15

3

I guess i found the problem with your code.
You're not adding the NewsContent object to your List.

in the foreach loop please add it to the List

lstResult.Add(newsContent)

Hope it solves the issue with your async strategy

answered Jan 18 '16 at 05:15

Vishnu Prasad V

1,228
1
8
18

Thank you ,you are right ,i am running my code to check the result – Ehsan Akbar Jan 18 '16 at 05:19
But one question should i use Task.WaitAll(lst); or Task.WhenAll(lst)? – Ehsan Akbar Jan 18 '16 at 05:19
use Task.WaitAll(lst); . Task.WhenAll(lst) returns a Task that will complete when all the tasks passed have completed. see http://stackoverflow.com/questions/6123406/waitall-vs-whenall – Vishnu Prasad V Jan 18 '16 at 05:23
I run the code i put a breakpoint in foreach loop .but the loop never executed why? – Ehsan Akbar Jan 18 '16 at 05:25

score 1 · Answer 2 · answered Jan 23 '16 at 03:21

But the problem is when i call my async function :

Task.WaitAll(lst);
List<NewsContent> enresult = lst.Result;

Yes, that's a problem, all right. Two problems, actually: Task.WaitAll and Result. They should both be replaced with a single await:

List<NewsContent> enresult = await lst;

The core problem is a deadlock scenario that I explain in full on my blog. In summary, await will capture the current context and use that to resume the async method. But ASP.NET only allows one thread within its request context at a time. So when parsing is first called, it executes until it hits its await and then returns. The calling method then blocks; this is where the problem is, because by blocking, the calling method is keeping a thread in that ASP.NET request context.

Later, when the await inside parsing is done, it attempts to resume the parsing method in that ASP.NET request context but it cannot, because there's a thread stuck in that context and ASP.NET only allows one thread at a time. The calling method is waiting for parsing to complete, and parsing is waiting for the context to be free. Classic deadlock.

Stephen's blog posts regarding `async`/`await` are *invaluable*. If anyone is even the slightest bit confused by how they work, I highly recommend checking them out. — Joshua Shearer, Jan 23 '16 at 03:33

Get the content of website using HttpClient but without async method

2 Answers2