Using .Net how can I programmatically navigate to a webpage, interact with it via code, then get particular values from a newly generated page

Question

I have a scenario where I would like to automate programmatically the following process:

Currently, I have to manually

Navigate to a webpage
Enter some text (an email) in a certain field on the webpage
Press the 'Search' button, which generates a new page containing a Table with the results on it.
Manually scroll through the generated results table and extract 4 pieces of information.

Is there a way for me to do this from a Desktop WPF App using C# ?

I am aware there is a WebClient type that can download a string, presumably of the content of the webpage, but I don't see how that would help me.

My knowledge of web based stuff is pretty non-existent so I am quite lost how to go about this, or even if this is possible.

That is somewhat like a robot. What if the page has additional security like recaptcha? — Tavershima, Aug 06 '20 at 11:01
@Tavershima , the page in question that I will be using doesn't have that sort of additional security. — Cleve, Aug 06 '20 at 11:10

score 1 · Answer 1 · answered Aug 06 '20 at 10:28

1

I think a web driver is what you're looking for, I would suggest using Selenium, you can navigate to sites and send input or clicks to specific elements in them.

answered Aug 06 '20 at 10:28

Hi, thank you for the suggestion, however, I was hoping to avoid having another dependency. – Cleve Aug 06 '20 at 10:32
not sure if that's possible without a web driver but good luck – Aug 06 '20 at 10:36

score 1 · Answer 2 · answered Aug 06 '20 at 11:12

Well, I'll write the algorithm for you but you also need to some homework.

UseWebClient get the htm page with the form you want to auto fill and submit
Us regex and extract the action attribute of the form you want to auto submit. That gets you the URL you want to submit your next request to.
Since you know the fields in that form, create a class corresponding to those fields, let's call the class AutoClass
Create a new instance of your auto class and assign values you want to auto fill
Using WebClient to send your new request with the url you extracted from the form previously, attach your object which you want to send to the server either through serialization or any method.
Send the request and wait for feedback, then further action

score 1 · Answer 3 · answered Aug 06 '20 at 11:14

1

Either use a web driver like Puppeteer (Selenium is kinda dead) or use HTTPS protocol to make web requests (if you don't get stopped by bot checks). I feel like your looking for the latter method because there is no reason to use a web driver in this case when a lighter method like HTTP requests can be used.

You can use RestSharp or the built in libraries if you want. Here is a popular thread of the ways to send requests with the libraries built in to C#.

To figure out what you need to send you should use a tool like Fiddler or Chrome Dev Tools (specifically the Network tab) to see what you need to send to acheive your goal as you would in a browser.

answered Aug 06 '20 at 11:14

DudeManGuy

123
2
12

1

@Cleve This shouldn't be too hard to get the specific HTTP commands to send. All you need to do is record the network traffic your browser does or use Fiddler and press the button. Then copy what is sent when pressed the button and do the same in C#. From there its simply extracting the data and processing. Mind this will be easier if you can find an API behind it. All this is too much for a single question so go do some googling about this topic. – DudeManGuy Aug 06 '20 at 11:30
1

Thanks again. You've given me enough now to start diving into the problem. – Cleve Aug 06 '20 at 11:38

Using .Net how can I programmatically navigate to a webpage, interact with it via code, then get particular values from a newly generated page

3 Answers3