0

I am using the following code to retrieve the html code from a url. It is working fine for a url as:

"The remote server returned an error: (429) unknown.'"

What could be the error or how to get more info about the error?

private void getHtmlFromUrl() {

    string urlAddress = "https://trends.google.es/trends/explore?q=test&geo=US";

    HttpWebRequest request = (HttpWebRequest)WebRequest.Create(urlAddress);
    HttpWebResponse response = (HttpWebResponse)request.GetResponse();

    if (response.StatusCode == HttpStatusCode.OK)
    {
        Stream receiveStream = response.GetResponseStream();
        StreamReader readStream = null;

        if (String.IsNullOrWhiteSpace(response.CharacterSet))
            readStream = new StreamReader(receiveStream);
        else
            readStream = new StreamReader(receiveStream, Encoding.GetEncoding(response.CharacterSet));

        string htmlData = readStream.ReadToEnd();

        response.Close();
        readStream.Close();
    }

}
Jaume
  • 3,672
  • 19
  • 60
  • 119
  • Remove "q=test&" from the URL. – jdweng Apr 11 '21 at 12:54
  • why? 'test' is the keyword that I set to explain the problem. If you open the url in the browser you will see it's valid – Jaume Apr 11 '21 at 12:56
  • Nope, it says that *you have sent too many requests (that's all we know)* and drops it (with 429 as the reason). BTW (unrelated), you're not disposing of any of the disposables. – Jimi Apr 11 '21 at 12:57
  • Anything after the question mark in a URL are parameters (a list of keys and there values with a amphersand between the parameters). The server has to be programmed to accept each of the keys. Does your webpage know what the key "test" is used for? – jdweng Apr 11 '21 at 13:02
  • @jdweng, yes, try to open the url into the browser. Parameters are ok. – Jaume Apr 11 '21 at 13:10
  • @Jimi, 429 use to be the error of many requests, but returned with 'many requests' message, not the 'unknown'. So not this case, I am only sending a single request, verified. – Jaume Apr 11 '21 at 13:12
  • I'm just telling you what I'm seeing in a WebBrowser using that URL. Unknown (429) can be sent when you don't have a more explicit reason besides the message already posted to the petitioner. i.e., not a Protocol or otherwise Well-known error / status. – Jimi Apr 11 '21 at 13:15
  • Some parameters are ok. Only the parameters the website is designed to handle are acceptable. The parameter TEST is not acceptable. – jdweng Apr 11 '21 at 15:37

2 Answers2

1

It's no longer possible to get the source code of google services without an API because specifically the trends service makes many calls when you visit only trends.google.es/trends/explore?q=test hence the error 429.

Do not waste your time digging in proxies, browser emulation or bots none will work. The best way is to use Google API for c# .

Example Projects:

https://github.com/thegreymatter/GoogleTrends (Deprecated)

https://developers.google.com/api-client-library/dotnet/guide/aaa_oauth (Deprecated)

New Solution!

(We install python then convert our py script to an exe file to use from .net code. The good news is no api is needed with this method)

1- Install Python: https://www.python.org/downloads/windows/

2- Install Pytrends using:

pip install pytrends

3- Create a test.py and add support to arguments:

from pytrends.request import TrendReq
import sys

# Only need to run this once, the rest of requests will use the same session.
pytrend = TrendReq()
# Set your region
pytrend.geo = 'ES'
pytrend.build_payload(kw_list=[sys.argv[1]]) # Also supports an array of keywords e.g. kw_list=['test', 'music']
related_queries_dict = pytrend.related_queries()
print(related_queries_dict)
#Check the documentation for other available queries https://github.com/GeneralMills/pytrends#api-methods

Now if you cmd.exe (do not use PowerShell command won't work there) in the same location of test.py use the following command:

test.py test

Screenshot Results output of test.py test

4- Now let's convert test.py to an .exe (windows console file)

(a) Install Pyinstaller:

pip install pyinstaller

(b) Compile test.py to test.exe

pyinstaller test.py -F

5- Your "test.exe" is inside \dist\ directory. Let's see if it works:

test.exe test output

Yes it does. (Keep in mind that the filesize of test.exe is 80 mb since it employs many libraries.)

Now all you have left to do is Process.Start the "test.exe" filename with argument "test" to read the output from your csharp app.

You can use IronPython instead to make python commands through your csharp code

MrJack Mcfreder
  • 692
  • 1
  • 10
  • 21
0

I highly recommend the GoogleTrendsApi package, which allows you to connect to this Google API, and receive data from it in a really simple way, certainly compared to the way of integrating Python into your code.

Full disclosure: I am its creator.

Moshe levy
  • 1
  • 1
  • Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community Mar 12 '23 at 15:13