1

I want to crawl all the wordpress blogs. Is there any quick approach to list all the secondary domain names under wordpress.com? For example http://developer.wordpress.com/

If we can get the list of all secondary domain address, it should be easy and quick to crawl all the dataset.

So can we do by 1) dns lookup for all secondary domain names for a website. 2) or by other hacking techniques?

Thank you for the answer to this question!

user1660362
  • 19
  • 1
  • 2
  • Might be a duplicate of this question - http://stackoverflow.com/questions/131989/how-do-i-get-a-list-of-all-subdomains-of-a-domain – Lawrence Oct 19 '12 at 22:01
  • yes, but it is different. Can we do other way to crawl all the pages. DNS lookup approach doesn't work since dns for the website disabled dig option for sub-domains. – user1660362 Oct 19 '12 at 22:27
  • I have no knowledge in this field so don't flame me please, but would it not be possible to compile the result programmatically from a Google search for site:wordpress.com ? – C. E. Oct 19 '12 at 23:33
  • sorry I'm just wondering how to solve the problem. The only problem for google api is that they only allow 100 queries per day. :-( – user1660362 Oct 21 '12 at 05:24
  • Google no longer provides complete results for queries such as site:wordpress.com. You might get a more complete list at http://moz.com/researchtools/ose (You might also be violating WordPress's TOS). – Michael McGinnis Jul 18 '14 at 22:24

1 Answers1

0

You can easily find the subdomain using google dorks on some online apps that do the realtime search like http://www.iseebug.com/find-sub-domain-online-search-domain/ Also the online crawler like http://iseebug.com/sandVox/

Vaibs
  • 2,018
  • 22
  • 29