WebClient or WebRequest to get the re-directed URL of the landing page

Question

From the string I parsed from Bing's Pic of the Day, I got the info of the pic to be downloaded, let's say today it is /az/hprichbg/rb/PearlHarborWindows_EN-US8565186567, then we will have full URL of the image be like http://www.bing.com/az/hprichbg/rb/PearlHarborWindows_EN-US8565186567_1366x768.jpg

Usually Bing has an image of higher resolutions, so I will download the image 1920x1200 too. It's easy with the URL changed to be like http://www.bing.com/az/hprichbg/rb/PearlHarborWindows_EN-US8565186567_1920x1200.jpg, then give the task to a WebClient such as client1.DownloadFile(url, fileName)

The issue here is, some days, the resolution 1920x1200 is not available, and the download URL of this res.(1920x1200) will be re-directed to the URL of the image /sa/simg/hpb/NorthMale_EN-US8782628354_1920x1200.jpg - as default (you can check it).

So my try was a function to get the return/re-directed URL from the input URL:

 Public Function GetWebPageURL(ByVal url As String) As String
    Dim Request As WebRequest = WebRequest.Create(url)
    Request.Credentials = CredentialCache.DefaultCredentials
    Return Request.RequestUri.ToString 
 End Function

and compare to the input URL to see it they are different, but the result was not as expected.

Could anyone let me know the method to check this re-directed URL, like the return URL after we press Enter and wait for the site to load.

Please give me idea to overcome this obstacle. Thank you!

Notes: Some issues related to access rights on different PCs cause me not to use HttpWebRequest, so I prefer the solution not using HttpWebRequest (WebClient or others are better).

With help from @IvanValadares @AlenGenzić, and suggestion of Proxy for HttpWebRequest from @Jimi, I have come to the fair solution, as the below code:

    url1 = "http://www.bing.com/az/hprichbg/rb/PearlHarborWindows_EN-US8565186567_1920x1200.jpg"

    Dim myHttpWebRequest As HttpWebRequest = CType(WebRequest.Create(url1), HttpWebRequest)
    myHttpWebRequest.MaximumAutomaticRedirections = 1
    myHttpWebRequest.AllowAutoRedirect = True
    Dim defaultProxy As IWebProxy = WebRequest.DefaultWebProxy
    If (defaultProxy IsNot Nothing) Then
        defaultProxy.Credentials = CredentialCache.DefaultCredentials
        myHttpWebRequest.Proxy = defaultProxy
    End If

    Dim myHttpWebResponse As HttpWebResponse = CType(myHttpWebRequest.GetResponse, HttpWebResponse)
    url2 = myHttpWebResponse.ResponseUri.ToString

    Label1.Text = url1
    Label2.Text = url2

Possible duplicate of [Download file through code that has a redirect?](https://stackoverflow.com/questions/11749676/download-file-through-code-that-has-a-redirect) — Alen Genzić, Dec 07 '18 at 10:06
Thanks @AlenGenzić, you're so fast on searching the similar issue! But I prefer the `WebClient` to `HttpWebRequest`, because I encountered some issue related to access right on some PCs. — nambk, Dec 07 '18 at 10:21
In the background, WebClient uses HttpWebRequest anyway, so it is interesting that it works in the first place. — Alen Genzić, Dec 07 '18 at 13:57
What has to do `WebRequest` with *access rights*? The file you download is yours to manage. `WebRequest` has nothing related to files. If you prefer to use WebClient (which uses WebRequest, as already noted), because you know how to use it, create a control derived from it. [One of the many related questions on SO](https://stackoverflow.com/questions/6726889/how-do-you-prevent-the-webclient-class-from-automatically-following-the-location). — Jimi, Dec 07 '18 at 14:41
I got the System.Net.WebException on the line `Dim myHttpWebResponse As HttpWebResponse = CType(myHttpWebRequest.GetResponse, HttpWebResponse)` @Jimi @AlenGenzić — nambk, Dec 11 '18 at 02:55
If the below error occurs, what solution should we switch to? `System.Net.WebException: The remote server returned an error: (407) Proxy Authentication Required.` — nambk, Dec 11 '18 at 04:04
Well, apparently there's a Proxy in between your machine and the remote machine. You need credentials to authenticate your request, otherwise the Proxy will block it. Search SO for "HttpWebRequest" (or WebClient of HttpClient) and "Proxy". — Jimi, Dec 11 '18 at 04:10
With @Jimi advice, I navigated to [How can I get WebClient (webservice client) to automatically use the default proxy server?](https://stackoverflow.com/questions/16966486/how-can-i-get-webclient-webservice-client-to-automatically-use-the-default-pro) With the credentials and proxy settings, the WebException is no longer thrown. Thank you all for your kind helps! — nambk, Dec 11 '18 at 05:34
@nambk I'm glad to hear you solved it. Good job :) You could post it as an answer to your own question. — Jimi, Dec 11 '18 at 05:36
Hi @Jimi, could you post another reply so I can mark this question as answered? Thank you. — nambk, Dec 11 '18 at 05:40
I didn't solve the problem. You actually did. Post your answer to the question and then mark it as answered yourself. — Jimi, Dec 11 '18 at 05:42

Ivan Valadares · Answer 1 · 2018-12-07T11:16:49.937

1

Use AllowAutoRedirect and check the StatusCode.

var webRequest = (HttpWebRequest)System.Net.WebRequest.Create("http://www.bing.com/az/hprichbg/rb/PearlHarborWindows_EN-US8565186567_1920x1200.jpg");
    webRequest.AllowAutoRedirect = false;
    using (var response = (HttpWebResponse)webRequest.GetResponse())
    {
       if (response.StatusCode == HttpStatusCode.Found)
       {
          // Have been redirect
       }
       else if (response.StatusCode == HttpStatusCode.OK)
       {
          // Have not been redirect
       }
    }

Using HttpClient

 var handler = new HttpClientHandler()
 {
    AllowAutoRedirect = false
 };
 HttpClient client = new HttpClient(handler);
 HttpResponseMessage response = await client.GetAsync("http://www.bing.com/az/hprichbg/rb/PearlHarborWindows_EN-US8565186567_1920x1200.jpg");
 if (response.StatusCode == HttpStatusCode.Found)
 {
     // Have been redirect
 }
 else if (response.StatusCode == HttpStatusCode.OK)
 {
     // Have not been redirect
 }

edited Dec 07 '18 at 11:16

answered Dec 07 '18 at 10:51

Ivan Valadares

869
1
8
18

Thanks @Ivan! Some issues related to access right on different PCs cause me not to use `HttpWebRequest`, could you advice me other ideas on this? Thanks in advance! – nambk Dec 07 '18 at 10:57
You can also use AllowAutoRedirect with HttpClient – Ivan Valadares Dec 07 '18 at 11:17
I got the System.Net.WebException on the line `using (var response = (HttpWebResponse)webRequest.GetResponse())` – nambk Dec 11 '18 at 02:56
If the below error occurs, what solution should we switch to? `System.Net.WebException: The remote server returned an error: (407) Proxy Authentication Required.` – nambk Dec 11 '18 at 04:04
Do you need to use a proxy? Because your code is : "myHttpWebRequest.Proxy = defaultProxy" , are the credentials correct to that proxy? – Ivan Valadares Dec 11 '18 at 09:12
I guess so, because the code run well with no web exception. It seems it uses the default proxy with the same credentials when using the default web browser. – nambk Dec 11 '18 at 10:16

score 1 · Accepted Answer · answered Dec 11 '18 at 06:54

With help from @IvanValadares @AlenGenzić, and suggestion of Proxy for HttpWebRequest from @Jimi, I have come to the fair solution as below:

    url1 = "http://www.bing.com/az/hprichbg/rb/PearlHarborWindows_EN-US8565186567_1920x1200.jpg"

    Dim myHttpWebRequest As HttpWebRequest = CType(WebRequest.Create(url1), HttpWebRequest)
    myHttpWebRequest.MaximumAutomaticRedirections = 1
    myHttpWebRequest.AllowAutoRedirect = True
    Dim defaultProxy As IWebProxy = WebRequest.DefaultWebProxy
    If (defaultProxy IsNot Nothing) Then
        defaultProxy.Credentials = CredentialCache.DefaultCredentials
        myHttpWebRequest.Proxy = defaultProxy
    End If

    Dim myHttpWebResponse As HttpWebResponse = CType(myHttpWebRequest.GetResponse, HttpWebResponse)
    url2 = myHttpWebResponse.ResponseUri.ToString

    Label1.Text = url1
    Label2.Text = url2

The System.Net.WebException: The remote server returned an error: (407) Proxy Authentication Required. is no longer thrown.

As a note, your `WebRequest` authetication might be rejected by the Proxy if your resource requires an `Https` connection. You could resolve changing `https` to `http`. Most of the services automatically change the protocol to `https` when needed (you might get a redirection here, so also keep `AllowAutoRedirect = true`). Some services don't. — Jimi, Dec 11 '18 at 07:24

WebClient or WebRequest to get the re-directed URL of the landing page

2 Answers2