Stack Overflow
Stack Overflow

Questions tagged [heritrix]

Heritrix is a web-crawler.

Heritrix is a web-crawler created by the Internet Archive for the purpose of archiving websites. It is a free software licence program written in Java.

43 questions
5
votes
2 answers

Nutch vs Heritrix vs Stormcrawler vs MegaIndex vs Mixnode

We need to crawl a large number (~1.5 billion) of web pages every two weeks. Speed, hence cost, is a huge factor for us as our initial attempts have ended up costing us over $20k. Is there any data on which crawler performs the best in a distributed…
Anakin
  • 107
  • 1
  • 5
4
votes
2 answers

How do i exclude everything but text/html from a heritrix crawl?

On: Heritrix Usecases there is an Use Case for "Only Store Successful HTML Pages" My Problem: i dont know how to implement it in my cxml File. Especially: Adding the ContentTypeRegExpFilter to the ARCWriterProcessor => set its regexp setting to…
dgAlien
  • 428
  • 1
  • 4
  • 9
3
votes
1 answer

Heritrix: Ignoring robots.txt for one site only

I am using Heritrix 3.2.0. I want to grab everything from one site, including pages normally protected by robots.txt. However, I do not want to ignore robots.txt for other sites. (Don't want Facebook or Google to get angry with us, you know) I…
Stig Hemmer
  • 2,604
  • 2
  • 11
  • 17
3
votes
2 answers

Heritrix single-site scrape, including required off-site assets

I believe need help compiling Heritrix decide rules, although I'm open to other Heritrix suggestions: https://webarchive.jira.com/wiki/display/Heritrix/Configuring+Crawl+Scope+Using+DecideRules I need to scrape an entire copy of a website (in the…
Karl M.W.
  • 728
  • 5
  • 19
2
votes
1 answer

How do I upgrade maven.xml to pom.xml?

I'm working with the 1.14.4 branch of Heritrix and I'm unfortunately for the time being stuck in that branch.. The problem I'm encountering is, its maven.xml is dependent upon Maven 1.1 which is so old I had trouble even finding the dependencies to…
synthesizerpatel
  • 27,321
  • 5
  • 74
  • 91
2
votes
1 answer

Which block represents a WARC-Block-Digest?

At Line 09 below there is this line: WARC-Block-Digest: sha1:CLODKYDXCHPVOJMJWHJVT3EJJDKI2RTQ Line 01: WARC/1.0 Line 02: WARC-Type: request Line 03: WARC-Target-URI: https://climate.nasa.gov/vital-signs/carbon-dioxide/ Line 04: Content-Type:…
user16656944
2
votes
1 answer

Is Heritrix3.2.0 able to crawl ajax-based web sites?

Is it possible to crawl ajax-based web sites using Heritrix-3.2.0?
T.Sh
  • 390
  • 2
  • 16
2
votes
1 answer

Running a web-spider on Java

Launch web spider on Windovs 8.1 64-bit. Tried not to connect additional libraries, and eventually climbs mistake. C:\Users\I>cd c:\Users\i\Desktop\heritrix-1.14.4 c:\Users\I\Desktop\heritrix-1.14.4>cd…
user3057645
  • 321
  • 1
  • 2
  • 10
2
votes
1 answer

In Heritrix crawler tool how to extract the contents from crawled urls

Am new to heritrix tool and now i am able to crawl the web pages from www and now want to extract the contents of the crawled urls. please help me any one.please.Thanks in Advance.
Dharmaraja.k
  • 523
  • 3
  • 8
  • 22
2
votes
1 answer

What is a good Java-based crawler for an academic project regarding building a search engine?

Okay, so I have been looking for the last two days for a crawler that suits my needs. I want to build a search engine and I want to do the indexing myself. This will be part of an academic project. Although I do not have the processing power to…
Marco
  • 21
  • 6
1
vote
1 answer

updating Solr from Lucene Index

I'm currently working on a web archiving project. Basically, what we try to do is archive a collection of websites (using heritrix crawler) and provide access to the archived contents through a web interface. We also offer full-text search…
user871784
  • 1,247
  • 4
  • 13
  • 32
1
vote
0 answers

Heritrix 3.2.0 can't find files and won't execute

I'm trying to use Heritrix 3.2.0 and following the steps provided here and here2. But everytime I try to execute a command like: $HERITRIX_HOME/bin/heritrix --help $HERITRIX_HOME/bin/heritrix --webui-admin PASSWORD I always get the same…
PlayHardGoPro
  • 2,791
  • 10
  • 51
  • 90
1
vote
1 answer

Heritrix Content Filtering

I have a requirement to aggregate content from several different web sites (primarily HTML pages and PDF documents). I'm currently experimenting with Heritrix (3.2.0) to see if it will meet my needs. While the documentation is pretty detailed the…
pws
  • 11
  • 2
1
vote
1 answer

Heritrix not finding CSS files in conditional comment blocks

The Problem/evidence Heritrix is not detecting the presence of files in conditional comments that open & close in one string, such as this: However…
Karl M.W.
  • 728
  • 5
  • 19
1
vote
0 answers

Heritrix3 exclude images, videos and archives from being crawled

i am using Heritrix3 we are trying to exclude images, videos and archives from the set of URIs being crawled with a MatchesListRegexDecideRule, I have set it in crawler-beans.cxml configuration file which is created at startup when job is created…
Qasim Javed
  • 27
  • 1
  • 7
1
2 3