62

Edit: I wanted to come back to note that the problem wasn't on my end at all, but rather with with code on the other company's side.

I'm trying to pull up a page using Basic Authentication. I keep getting a 404 Page not found error. I can copy and paste my url into the browser and it works fine (if I'm not logged into their site already it pops up a credential box, otherwise it opens what I want it to open). I must be getting to the right place and authenticating, because I get a 401 (not authenticated error) if I intentially put in a bad username/password and I get an internal server error 500 if I pass it a bad parameter in the query string. I've tried using Webclient and HttpWebRequest both leading to the same 404 not found error.

With Webclient:

string url = "MyValidURLwithQueryString";
WebClient client = new WebClient();
String userName = "myusername";
String passWord = "mypassword";
string credentials = Convert.ToBase64String(Encoding.ASCII.GetBytes(userName + ":" + passWord));
client.Headers[HttpRequestHeader.Authorization] = "Basic " + credentials;
var result = client.DownloadString(url);
Response.Write(result);

With HttpWebRequest

HttpWebRequest request = (HttpWebRequest)WebRequest.Create("MyValidURL");
string authInfo = "username:password";
authInfo = Convert.ToBase64String(Encoding.Default.GetBytes(authInfo));
request.Headers.Add("Authorization", "Basic " + authInfo);
request.Credentials = new NetworkCredential("username", "password");
request.Method = WebRequestMethods.Http.Get;
request.AllowAutoRedirect = true;
request.Proxy = null;
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Stream stream = response.GetResponseStream();
StreamReader streamreader = new StreamReader(stream);
string s = streamreader.ReadToEnd();
Response.Write(s);
Alexandra Damaschin
  • 761
  • 3
  • 13
  • 28
Josh Blade
  • 981
  • 2
  • 10
  • 24
  • 2
    Using [Fiddler](http://fiddler2.com/), login to your site through your browser and see what headers/body your browser sends. My guess, you should also set the `UserAgent` – I4V Apr 16 '13 at 19:29
  • The header information is correct looking in fiddler and I also tried setting UserAgent at one point from a previous post I had looked at that suggested the same thing. – Josh Blade Apr 17 '13 at 16:16
  • The 404 is not coming from WebClient - it's from the server. There's something the server doesn't like from your program, that it likes from the browser. Use Fiddler to make your WebClient look _exactly_ like the browser. – John Saunders Oct 12 '13 at 14:32
  • Refer: http://stackoverflow.com/questions/6440255/missing-basic-http-authentication-entry-in-http-request-header – Healer Oct 16 '13 at 06:01
  • Found very nice solution [here](https://www.codeproject.com/Articles/624624/Using-a-Cookie-Aware-WebClient-to-Persist-Authenti) – sairfan May 20 '18 at 01:47

4 Answers4

126
//BEWARE
//This works ONLY if the server returns 401 first
//The client DOES NOT send credentials on first request
//ONLY after a 401
client.Credentials = new NetworkCredential(userName, passWord); //doesnt work

//So use THIS instead to send credentials RIGHT AWAY
string credentials = Convert.ToBase64String(
    Encoding.ASCII.GetBytes(userName + ":" + password));
client.Headers[HttpRequestHeader.Authorization] = string.Format(
    "Basic {0}", credentials);
Jon Schneider
  • 25,758
  • 23
  • 142
  • 170
Alex from Jitbit
  • 53,710
  • 19
  • 160
  • 149
  • 2
    Wow! important note there regarding method described by @Blake doesn't work until second request. Kudos for the solution – Korayem Mar 06 '15 at 14:40
  • 2
    I'd been struggling with a similar issue and this totally fixed it. THANK YOU! – Paul Mignard Jul 27 '15 at 14:12
  • 6
    I had to use `Encoding.UTF8.GetBytes()` or it wouldn't work for special characters like é – Stefanvds Dec 09 '15 at 03:24
  • 1
    @Stefanvds good point, but this is risky. RFC does not specify which encoding should be used: https://stackoverflow.com/questions/7242316/what-encoding-should-i-use-for-http-basic-authentication – Alex from Jitbit Dec 12 '15 at 20:10
  • 1
    This is working fine for **Basic Authorization** but how can I do if I am using **Ntlm Authorization** with mandatory **domain**?? – Wajdi Chamakhi Dec 20 '18 at 11:03
  • I'm blown away. Soo many options about stuff I couldn't care less about, but not this. Thank you! – Gerard ONeill Jan 17 '20 at 05:08
31

Try changing the Web Client request authentication part to:

NetworkCredential myCreds = new NetworkCredential(userName, passWord);
client.Credentials = myCreds;

Then make your call, seems to work fine for me.

Blake
  • 1,067
  • 14
  • 25
13

This part of code worked fine for me:

        WebRequest request = WebRequest.Create(url);
        request.Method = WebRequestMethods.Http.Get;
        NetworkCredential networkCredential = new NetworkCredential(logon, password); // logon in format "domain\username"
        CredentialCache myCredentialCache = new CredentialCache {{new Uri(url), "Basic", networkCredential}};
        request.PreAuthenticate = true;
        request.Credentials = myCredentialCache;
        using (WebResponse response = request.GetResponse())
        {
            Console.WriteLine(((HttpWebResponse)response).StatusDescription);

            using (Stream dataStream = response.GetResponseStream())
            {
                using (StreamReader reader = new StreamReader(dataStream))
                {
                    string responseFromServer = reader.ReadToEnd();
                    Console.WriteLine(responseFromServer);
                }
            }
        }
thezar
  • 1,278
  • 13
  • 17
  • I am trying to do this in SSIS 2010 - // Get the URL from the variable string url = Dts.Variables["User::URL"].Value.ToString(); NetworkCredential networkCredential = new NetworkCredential( string url = Dts.Variables["User::Email"].Value.ToString(), string url = Dts.Variables["User::Password"].Value.ToString()); Can you please help with this credential code – Joeysonic Sep 03 '19 at 14:54
3

If its working when you are using a browser and then passing on your username and password for the first time - then this means that once authentication is done Request header of your browser is set with required authentication values, which is then passed on each time a request is made to hosting server.

So start with inspecting Request Header (this could be done using Web Developers tools), Once you established whats required in header then you could pass this within your HttpWebRequest Header.

Example with Digest Authentication:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Security.Cryptography;
using System.Text.RegularExpressions;
using System.Net;
using System.IO;

namespace NUI
{
    public class DigestAuthFixer
    {
        private static string _host;
        private static string _user;
        private static string _password;
        private static string _realm;
        private static string _nonce;
        private static string _qop;
        private static string _cnonce;
        private static DateTime _cnonceDate;
        private static int _nc;

public DigestAuthFixer(string host, string user, string password)
{
    // TODO: Complete member initialization
    _host = host;
    _user = user;
    _password = password;
}

private string CalculateMd5Hash(
    string input)
{
    var inputBytes = Encoding.ASCII.GetBytes(input);
    var hash = MD5.Create().ComputeHash(inputBytes);
    var sb = new StringBuilder();
    foreach (var b in hash)
        sb.Append(b.ToString("x2"));
    return sb.ToString();
}

private string GrabHeaderVar(
    string varName,
    string header)
{
    var regHeader = new Regex(string.Format(@"{0}=""([^""]*)""", varName));
    var matchHeader = regHeader.Match(header);
    if (matchHeader.Success)
        return matchHeader.Groups[1].Value;
    throw new ApplicationException(string.Format("Header {0} not found", varName));
}

private string GetDigestHeader(
    string dir)
{
    _nc = _nc + 1;

    var ha1 = CalculateMd5Hash(string.Format("{0}:{1}:{2}", _user, _realm, _password));
    var ha2 = CalculateMd5Hash(string.Format("{0}:{1}", "GET", dir));
    var digestResponse =
        CalculateMd5Hash(string.Format("{0}:{1}:{2:00000000}:{3}:{4}:{5}", ha1, _nonce, _nc, _cnonce, _qop, ha2));

    return string.Format("Digest username=\"{0}\", realm=\"{1}\", nonce=\"{2}\", uri=\"{3}\", " +
        "algorithm=MD5, response=\"{4}\", qop={5}, nc={6:00000000}, cnonce=\"{7}\"",
        _user, _realm, _nonce, dir, digestResponse, _qop, _nc, _cnonce);
}

public string GrabResponse(
    string dir)
{
    var url = _host + dir;
    var uri = new Uri(url);

    var request = (HttpWebRequest)WebRequest.Create(uri);

    // If we've got a recent Auth header, re-use it!
    if (!string.IsNullOrEmpty(_cnonce) &&
        DateTime.Now.Subtract(_cnonceDate).TotalHours < 1.0)
    {
        request.Headers.Add("Authorization", GetDigestHeader(dir));
    }

    HttpWebResponse response;
    try
    {
        response = (HttpWebResponse)request.GetResponse();
    }
    catch (WebException ex)
    {
        // Try to fix a 401 exception by adding a Authorization header
        if (ex.Response == null || ((HttpWebResponse)ex.Response).StatusCode != HttpStatusCode.Unauthorized)
            throw;

        var wwwAuthenticateHeader = ex.Response.Headers["WWW-Authenticate"];
        _realm = GrabHeaderVar("realm", wwwAuthenticateHeader);
        _nonce = GrabHeaderVar("nonce", wwwAuthenticateHeader);
        _qop = GrabHeaderVar("qop", wwwAuthenticateHeader);

        _nc = 0;
        _cnonce = new Random().Next(123400, 9999999).ToString();
        _cnonceDate = DateTime.Now;

        var request2 = (HttpWebRequest)WebRequest.Create(uri);
        request2.Headers.Add("Authorization", GetDigestHeader(dir));
        response = (HttpWebResponse)request2.GetResponse();
    }
    var reader = new StreamReader(response.GetResponseStream());
    return reader.ReadToEnd();
}

}

Then you could call it:

DigestAuthFixer digest = new DigestAuthFixer(domain, username, password);
string strReturn = digest.GrabResponse(dir);

if Url is: http://xyz.rss.com/folder/rss then domain: http://xyz.rss.com (domain part) dir: /folder/rss (rest of the url)

you could also return it as stream and use XmlDocument Load() method.

Mvg Developer
  • 183
  • 1
  • 7