0

HTML parser...My recent project needs a web spider..it automatically get web content which it gets the links recursively.... But, it needs to know its content exactly. like tag. it runs in linux and windows..do you know some opensource about this needs.. thanx or about some suggestion.

StevenWang
  • 3,625
  • 4
  • 30
  • 40

3 Answers3

3

Here is a StackOverflow question showing how to use a number of XML/HTML parsers in different languages. If you tell us what language you're using, I can be more specific, but your answer may already be in there.

Community
  • 1
  • 1
Chris Lutz
  • 73,191
  • 16
  • 130
  • 183
0

Depends what language you are developing for, trying googling:

html parser languagename

hpricot is a good one for Ruby, for example.

David Claridge
  • 6,159
  • 2
  • 27
  • 25
-1

I think the subject you need to know is Regular Expression.

Regular Expression is available on all platform and all languages (Java, PHP, Python, C#, Ruby, Javascript). Using Regular Expression, you can easily exact its content as preferred form you want. 

Pattern p = Pattern.compile("<a\\s[^>]*href=\"([^\"]+?)\"[^>]*>");
Matcher m = p.matcher(pageContent);
while( m.find() ) { 
  System.out.println( m.group(1) );
}

Above code block written in Java will extract all anchor tags in a page and extract URL into your hand.

If you don't have enough time to learn Regular Expression, the following references will help you.

http://htmlparser.sourceforge.net/

xrath
  • 834
  • 6
  • 14
  • 1
    You should never use regular expressions to parse non-regular languages. Even if this will work, what happens when your requirements change? Why not start with the right tool for the job rather than try to hack something together? (X|HT)ML parsers are avaliable in almost every modern language, and are fairly easy to work with. – Chris Lutz Sep 25 '09 at 03:12