how to get https files based on URL in asp.net c#

Question

In my requirement, I need to get the file from a URL. I tried, but it always throws a forbidden error. Please try to solve this problem. Please look in to my code.

var webRequest = WebRequest.Create("https://www.fda.gov/ucm/groups/fdagov-public/@fdagov-drugs-gen/documents/document/ucm509432.pdf");  
using (var response = webRequest.GetResponse())
   using (var content = response.GetResponseStream())
      using (var reader = new StreamReader(content))
      {
         var strContent = reader.ReadToEnd();
      }

Does the site require login or auth of some kind? You may need to construct a cookie or user with the right creds for ti to let you in. — Mant101, Oct 23 '17 at 09:31
are you handling the certificate? https://stackoverflow.com/questions/560804/how-do-i-use-webrequest-to-access-an-ssl-encrypted-site-using-https — mikelegg, Oct 23 '17 at 09:32
we dont know any logins. they want to take the file based on URL. — chaitanya, Oct 23 '17 at 09:33
Loads fine in Chrome here so no login required. Also given it's binary data (a PDF in this instance) why are you using `ReadToEnd` and treating it as a string? — Lloyd, Oct 23 '17 at 09:33
yes but need to take that file and store in my DB but still i am not getting any file it throws always forbidden error — chaitanya, Oct 23 '17 at 09:37
This works OK for me. Double-check your URL (for spaces etc.). — SpruceMoose, Oct 23 '17 at 09:51
Works well. Make sure the URL is correct `WebRequest.Create("https://www.fda.gov/ucm/groups/fdagov-public/@fdagov-drugs-gen/documents/document/ucm509432.pdf")` — L J, Oct 23 '17 at 09:53
Im getting the same results as @chaitanya in the question it opens in a browser not in c# despite checking the url is one string etc — BugFinder, Oct 23 '17 at 10:06
The problem is probably the HttpWebRequest class not supporting TLS1.2. Check my answer. — Oguz Ozgul, Oct 23 '17 at 10:13

Oguz Ozgul · Accepted Answer · 2017-10-23T10:23:56.473

6

When I checked the response contents from the server, I realized that it is stating that the client needed to Support TLSv1.2.

To enable TLSv1.2 support, add the following line before creating the HttpWebRequest class: (Thanks BugFinder for the direct enumeration value tip)

ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;

I am posting the complete source code.

You can also see that I tested sending some headers to the server to see if the absence of one of them was the problem:

class Program
{
    static void Main(string[] args)
    {
        ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;

        HttpWebRequest webRequest = HttpWebRequest.Create("https://www.fda.gov/ucm/groups/fdagov-public/@fdagov-drugs-gen/documents/document/ucm509432.pdf") as HttpWebRequest;
        webRequest.UserAgent = "Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; Touch; rv:11.0) like Gecko";
        webRequest.Accept = "text/html, application/xhtml+xml, */*";
        webRequest.Headers.Add("Accept-Encoding", "gzip, deflate");
        webRequest.Headers.Add("Accept-Language", "tr-TR");
        webRequest.Headers.Add("DNT", "1");

        using (var
             response = webRequest.GetResponse())
        using (var content = response.GetResponseStream())
        using (var reader = new StreamReader(content))
        {
            var strContent = reader.ReadToEnd();
        }
    }
}

edited Oct 23 '17 at 10:23

answered Oct 23 '17 at 10:15

Oguz Ozgul

6,809
1
14
26

1

Good catch although ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12; should also be sufficient as its already defined – BugFinder Oct 23 '17 at 10:19
still it gives wrong content in StrContent var. not getting URL based content @BugFinder – chaitanya Oct 23 '17 at 10:23
Ok, did not focus on that part. ReadToEnd() returns string, while the content must be handled as binary. Converting the binary PDF data to string is no use I think. – Oguz Ozgul Oct 23 '17 at 10:28
will you please modify that @Oguz Ozgul – chaitanya Oct 23 '17 at 10:33
Your question is about how to overcome the 403 forbidden error. You can search stack overfow to learn how to treat binary content downloaded from a server. – Oguz Ozgul Oct 23 '17 at 10:38
i know hoe to convert binary to string but in this binary content is not correct as per my URL @OguzOzgul – chaitanya Oct 23 '17 at 10:42
please solve this problem. my question is i need to get URL content into string @OguzOzgul – chaitanya Oct 23 '17 at 10:47
Why a string? its a pdf - a PDF is not a string - what is the response you expect? – BugFinder Oct 23 '17 at 10:51
If you want to get the CONTENTS OF THE PDF as string, you need to use a library that converts pdf files to html / text etc. Check this open source solution for instance. https://github.com/spatie/pdf-to-text – Oguz Ozgul Oct 23 '17 at 10:55
will you please tell the where pdf content is there in which variable response,content,reader variables. @OguzOzgul – chaitanya Oct 23 '17 at 11:38
This working fine for me but it not working in IIS. will you please look into this once – chaitanya Oct 24 '17 at 04:18
the problem is solved. in hosting server they dont have permissions for Tls12 protocol. now solved thanks for helping me – chaitanya Oct 24 '17 at 04:45
You're welcome. I am glad that your problem is solved – Oguz Ozgul Oct 24 '17 at 06:45

how to get https files based on URL in asp.net c#

1 Answers1