Identity Framework Authenticated Request for Spidering

Question

I am working on a project to build a web spider that crawls our internal web pages. The security is built using the Identity Framework. The non-protected pages is easy by using this code:

HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create("http://www.yoursite.com/resource/file.htm");

using (StreamWriter streamWriter = new StreamWriter(webRequest.GetRequestStream(), Encoding.UTF8))
{
    streamWriter.Write(requestData);
}

string responseData = string.Empty;
HttpWebResponse httpResponse = (HttpWebResponse)webRequest.GetResponse();
using (StreamReader responseReader = new StreamReader(httpResponse.GetResponseStream()))
{
    responseData = responseReader.ReadToEnd();
}

But this code isn't using the OWIN authentication so when it hits a protected page it can't access it.

If I am logged in and try HttpContext.Request then I see through the debugger the Request is authenticated. How do I use this request object to get a new protected page to parse? I am missing a very simple method I am sure.

Update I am still struggling with this. Maybe I should ask this another way. I am calling a page from the same web application. From the calling controller this returns true which is good:

HttpContext.Current.User.Identity.IsAuthenticated

But through the debugger and check this in the receiving controller and now this is false, which is bad:

HttpContext.Current.User.Identity.IsAuthenticated

How can get the receiving controller to be authenticated?

score 0 · Answer 1 · edited May 23 '17 at 10:29

It really depends on what authentication method you implementation is being used on server side. Asp.net Identity is essentially a membership system.

For example if it's using bearer authentication tokens then you'd need to set a bearer authentication token in the Authorization header of the request you are sending. And I'd rather suggest using HttpClient for these purposes.

As you mentioned OWIN, Im assuming this is an bearer token scenario.

var httpClient = new HttpClient();
httpClient.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", "Your Oauth token");

Update

If you're using the default project template of Asp.net MVC with Asp.net Identity as a added dependency later, you can hook up a httpClient like the following one here:

class Program
{
    static void Main()
    {
        using (var client = new HttpClient())
        {
            client.BaseAddress = new Uri("http://yourapp");
            var content = new FormUrlEncodedContent(new[] 
            {
                // Push your stuff here, your username, password fields 
                // as you coded in your server
                new KeyValuePair<string, string>("", "login")
            });
            var result = client.PostAsync("login", content).Result;
            string resultContent = result.Content.ReadAsStringAsync().Result;
            Console.WriteLine(resultContent);
        }
    }
}

And you'd of course need to save the auth cookie and you can see how here and to attach it when you do the request to get the secured page, you have to do something like this

I am newbie at this and I am not totally sure how to answer your question. I am using just the standard MVC application of Indentity using the AspNet tables to store and handle user names, roles, etc... — Filjan, Aug 19 '16 at 22:37
Could you kindly at least provide me what version of Asp.net MVC you're using? — Swagata Prateek, Aug 19 '16 at 22:44
I assume this is something like [this](http://www.asp.net/identity/overview/getting-started/introduction-to-aspnet-identity) that essentially has been deployed. — Swagata Prateek, Aug 19 '16 at 22:51

score 0 · Answer 2 · answered Aug 19 '16 at 23:51

Looks like the permissions are based on roles. Check to see if the user you are logged in as has the permission to access the controller method getting hit when the user is redirected to that page.

If you have Attribute parameter for the method that specifies the role for the user required in order to hit then the logged in user should have those permissions to access the page.

As you are getting authenticated may be the issue is with authorization. The user might not have enough permissions to access the page.

Identity Framework Authenticated Request for Spidering

2 Answers2

Update