is it possible to click download button automatically in a web page using c# code

Question

I have a requirement where i have to download file automatically from a web page. There is a button for download and its redirecting to url. Can i click that button dynamically using code. The url is having a field called sid and i'm able to see sid value using firebug but the same sid is invisible in page source. How to download file automatically.

Answer 1

If the sid field is visible in firebug, yet it is not there in the source, then most probably the sid is added later via javascript. in that case, simply pulling the web source won't help much. you need a headless browser that is capable of running the javascript codes that might be needed.

i assume you are using C# to write your scraper, and unfortunately, there aren't that many headless browser the for .NET, i think your job might be easier if you are using something like node.js ( i am guessing, i never actually did any scrapping in node), but here is a SO question regarding some headless browser for .NET.

Answer 2

When Firebug is open, go and download a legitimate file. Then, in Firebug, examine where you are redirected after you click the download button. Also examine the headers/cookies. Then, simulate whatever your browser is doing in your program. It is possible to do (and relatively easy) if you know how HTTP works in general. If you don't learn about how HTTP works (it won't take long). As a rule of the thumb, mimic what you are doing (with your browser) at your code. If there are no captchas, it won't take long.