1

I'm sure this has been asked a billion and a half times, but I searched and didn't find anything on it.

I'd like to scrub some html pages, dynamically, and load the results into my database. jQuery seems like the perfect library to do this, but of course that runs in a browser and I'm looking to do this without a browser/user interaction.

What would be recommended for accomplishing this on the server-side (C# in my case)? Regex seems like overkill and not necessarily appropriate for the task.

Thanks, -Ben

UPDATE:

This library seems like what I'm after... but man I really like the way jQuery works.

http://htmlagilitypack.codeplex.com/

Ben Finkel
  • 4,753
  • 7
  • 34
  • 49

1 Answers1

4

jQuery seems like the perfect library to do this

I am not sure about this. The same origin policy will limit you to scraping pages only on the same domain as the one which served the javascript (which basically limits you to your own domain and for web scraping that's pretty limiting). It would be better to use a server side language for this task and an HTML parsing library. Depending on the server side language you are using there might be different libraries available. Html Agility Pack is very good if you are using .NET on the server. But whatever you do and whatever solution you choose one thing's for sure: don't use regular expressions for parsing HTML.

Community
  • 1
  • 1
Darin Dimitrov
  • 1,023,142
  • 271
  • 3,287
  • 2,928
  • Yea I think I understand that I would have to download the html I wanted to search/parse first. I'm okay with that part of it, it's the mechanics of parsing the page I'm curious about. – Ben Finkel Aug 07 '11 at 15:57
  • The same origin policy does NOT apply to – jfriend00 Aug 07 '11 at 15:59
  • @Ben Finkel, the answer to this is: use an HTML parsing library. – Darin Dimitrov Aug 07 '11 at 15:59
  • @jfriend00, yes it doesn't but loading anything other than javascript into a ` – Darin Dimitrov Aug 07 '11 at 16:00
  • Thanks Darin, agility pack it is. :) – Ben Finkel Aug 07 '11 at 16:03
  • Your same origin policy comment just doesn't make any sense to me period. What does it have to do with the question asked? The same origin policy limits ajax calls and cross frame access. I see nothing that has to do with the question asked. – jfriend00 Aug 07 '11 at 16:05
  • @jfriend00, my comment was indeed about HTML fetching. I just wanted to point it out as I didn't see any other tag than `jquery` and `html-parsing` on this question => I answered that this is not possible with those two techniques, because you will not be able to fetch the remote HTML. Well, of course, you can hardcode it in the javascript file but I guess that's not what's being asked here. – Darin Dimitrov Aug 07 '11 at 16:07
  • You haven't explained how the same origin policy has anything to do with HTML fetching from a server so I was trying to get you to remove that irrelevant part of your answer. The post seems to have run its course so I'll bow out now. – jfriend00 Aug 07 '11 at 16:15
  • @jfriend00, I've linked to the Wikipedia article which perfectly explains how the same origin policy has something to do with HTML fetching from a server. I am not good at explaining so I prefer to leave this to the experts and I find the Wikipedia article pretty good and easy to understand. – Darin Dimitrov Aug 07 '11 at 16:17
  • The very first line in that article is this: "In computing, the same origin policy is an important security concept for a number of `browser-side programming languages`, such as JavaScript." The subject of this question is `server-side`, not browser-side. – jfriend00 Aug 07 '11 at 17:08