2

I am trying to read other website data using angular website with backend asp.net c#, other website has following login authentication after login need to read the data.

I have following authentical pages

  1. first page i have to enter username

https://i.stack.imgur.com/PgbLo.png

  1. second page i have to enter password then click login

https://i.stack.imgur.com/xAr4E.png

  1. after login it will redirect to home page which content information of person details

https://i.stack.imgur.com/2I4UX.png

I need to display third point page information Name, mobile, and address in my angular website.

Third page url will be looks like : www.xyz.com/1

the above 1 digit in the url is id of information base on id information of third page will appear

I found some code using c# but how to manage logins

System.Net.WebClient wc = new System.Net.WebClient();
byte[] raw = wc.DownloadData("http://www.yoursite.com/resource/file.htm");

string webData = System.Text.Encoding.UTF8.GetString(raw);

and other way

  System.Net.WebClient wc = new System.Net.WebClient();
   string webData = 
   wc.DownloadString("http://www.yoursite.com/resource/file.htm");
   System.Net.WebClient wc = new System.Net.WebClient();
coder rock
  • 113
  • 6
  • Am I right that your backend supposed to read this information and pass it to the angular frontend? – Lonli-Lokli Nov 03 '22 at 18:42
  • I know this is done from backend but how to get information because website have username and password. @Lonli-Lokli – coder rock Nov 04 '22 at 07:50
  • Please reply below questions I can help here I have done a similar thing in fast. - 1) The screenshot from 1 to 3 from your app?? 2) The username and password that you have entered are valid and can be used on the 3rd part website. 3) First we need to look at the login page of the 3rd party website, and will validate the cookie that we are getting then only we can fetch data using WebClient in ASP.NET C# backend. – Mata Prasad Chauhan Nov 05 '22 at 03:36
  • The 1 to 3 is not our app this other person website which we need to consume data and i have username and password that is valid can you share your code will help to me. @MataPrasadChauhan – coder rock Nov 15 '22 at 12:15

4 Answers4

2

I would suggest checking if the site you want to get the data from has a public API (it's easier to call, and there is a chance the site has a "No bots" policy, your IP could get banned)

That being said, if you want to automatically log in the site and then get the data, you'll need to use something like Selenium (not an expert on it, but I can give you a link with documentation)

https://www.javatpoint.com/selenium-csharp

Selenium is usually used for automation testing, but it will allow you to simulate user interaction with a site (finding the field input and typing values).

Nos for getting the information, I used HtmlAgilityPack in the past, it is a C# library, that allows you to look for specific elements in a site (using something called XPath, it's easy to learn).

https://html-agility-pack.net/

They have some great documentation, and after you get the results from the site you can just expose it using an API that you can consume from you angular app.

Sunderam Dubey
  • 1
  • 11
  • 20
  • 40
Facundo Gallardo
  • 374
  • 2
  • 8
1

Like Facundo Gallardo said the best way to get data from a website is using the site's API if it has one. If not there isn't much you can do unfortunately.

Using a web browser simulator like selenium could be an option if you were building a desktop application, but for a server it is way too resource intensive. If you were to use it (and I am not even sure you can in a server environment), your server would have to open a new browser like chrome for every user and every time you wanted to scrap data from the other website.

Any other solution that doesn't require you to open a full browser can only scrap static data that require no authentication to be seen

StyleM
  • 138
  • 1
  • 5
0

You can use python script to do so. Prepare a python script to automate the login and use the same script in c# to complete the process.

Here are the two links for reference that may be helpfull.

submitting a webform using python

running a python script in c#

Shyam Narayan
  • 1,009
  • 2
  • 14
  • 28
0

For automation purpose with some Webui you will want to use something like https://www.puppeteersharp.com/ .

Most things that you can do manually in the browser can be done using Puppeteer!

Given something like a login screen like

<!DOCTYPE html>
<html>
<body>
<form action="/action_page.php">
  <label for="name">name:</label><br>
  <input type="text" id="name" name="name"><br>
  <label for="password">password:</label><br>
  <input type="password" id="password" name="password"><br><br>
  <input type="submit" value="Submit">
</form>
</body>
</html>

The .net code for the login challenge would look like

using PuppeteerSharp;

//Setup puppeteer things
using var browserFetcher = new BrowserFetcher();
await browserFetcher.DownloadAsync();
await using var browser = await Puppeteer.LaunchAsync(new LaunchOptions { Headless = false });
await using var page = await browser.NewPageAsync();

//Using https://jsfiddle.net/hwu6bkj0/ as demo web form
await page.GoToAsync("https://jsfiddle.net/hwu6bkj0/");

//Wait for your input element to appear/get loaded. Use the F12/Dev tools of your browser to find out the right query
await Task.Delay(1000);

// JsFiddle specific - getting result iframe
var frameElement = await page.QuerySelectorAsync("iframe[name='result']");
var frame = await frameElement.ContentFrameAsync();
var frameContent = await frame.GetContentAsync();

// Setting values to the input elements
var usenanme = await frame.QuerySelectorAsync("input[name='name']");
await usenanme.TypeAsync("testUserName");

var passsword = await frame.QuerySelectorAsync("input[name='password']");
await passsword.TypeAsync("pa$$word");

var submitButton = await frame.QuerySelectorAsync("input[type='submit']");
await Task.Delay(1000);
await submitButton.ClickAsync();

await Task.Delay(5000);

Headless will be set to true and the "Task.Delay"s should be removed in a final version.

Every following form should work similar.

stefan.seeland
  • 2,065
  • 2
  • 17
  • 29