0

I need an alternative to Google Custom Search for a website I look after, it has to be something that will crawl a website, index it, allow fiddling of priorities, and then allow search queries via REST or something similar and return XML or JSON etc. It needs to run on a Windows Server instance.

So, I'm up and running with http://www.opensearchserver.com/ and it seems to do the trick, but can't, for the life of me, work out how to get thumbnail images in the results? I've searched the documentation and read everything I could, but can't find out how to do this (or how to get my head around it).

I'm crawling standard web pages and they all have thumbnail meta data, which I'm assuming should be able to be parsed somehow for results and included in the JSON results?

Any pointers at all would be very helpful, thanks!

Ralpharama
  • 441
  • 4
  • 20

1 Answers1

1

I figured this out, in case anyone else is struggling, here's how I did it. The answer is in the documentations, it's just not that simple.

Read: http://www.opensearchserver.com/documentation/faq/crawling/how_to_extract_specific_information_from_web_pages.md - it contains the method

Assume you set up a 'web crawler' index.

Assuming you're using a meta thumbnail like this:

<meta name="thumbnail" content="http://my_cdn.com/news/images/29637.jpg">

Go into Schema / Fields. Add a new field called 'thumbnail' with index no, store yes, vector no, analyser Text, copy of blank. Save that.

Now go to schema / parser list, edit HTML parser. Go to 'field mapping', now add a new regex for the thumbnail in the html. We map from the 'htmlSource' to the thumbnail' with the matching regex.

My imperfect regex (that works though) is:

htmlSource -> linked in: thumbnail -> captured by: 

(?s)<meta name="thumbnail" content="(.*?)">

Now SAVE this and go to crawl/manual crawl, enter a url that has a thumbnail and then check if the field now appears in the list below when it's read. If not check your regex, and check you actually saved the HTML Parser changes.

To get the thumb in your results, simply add the fieldname to the JSON you send with the query:

"returnedFields": [ "
    "url",
    "thumbnail"
],
Ralpharama
  • 441
  • 4
  • 20