4

I'm working on a website that has minimal traffic at the moment. It's built using Ruby on Rails and runs on Heroku's cloud platform.

As part of the site, I have a large number of pages that need to be searchable, each of which only has a tiny amount of information on it. Think of a table of articles where each article only needs its title indexed, but there are around 8 million articles.

Postgres Search: When I first started working on this, I ran Postgres full text search, but apparently it's not optimized enough for search to handle this many indexed items, and ran dog slow. I had some searches that were timing out the database connection and taking more than 30 seconds to complete.

Websolr: I then moved onto what was then the one and only Heroku add-on for cloud search, Websolr by OneMoreCloud. Unfortunately, they charge by the number of items indexed, which is horrible for a site like mine that has no traffic but a large number of items to index, and I had performance that was arguably worse than Postgres search, which was free. Where Postgres search would timeout, and bring down the site, Websolr would return an empty or partial results set, making viewers think that the result wasn't in the database.

Index Tank: Now Heroku has added another cloud search provider, Index Tank, which is in beta still. While the beta for it is free, I'm reluctant to try them because for their non-Heroku service, which is not free, their highest plan only has 2 million documents while already costing an eye popping $500 a month.

Google Site Search: An option I'm currently looking at is moving over to Google Site Search. The Google search brand gives me confidence that I won't run into the performance issues I had in the past. Also, their pricing is extremely reasonable, and is priced by traffic. However, on the downside, it's not truly an integrated search, as it doesn't hook into the database but only looks at webpages, so there's no way as far as I can tell to specify a search where it only returns, say, articles in the Technical Articles category or something like that. Even to customize the appearance of the search results seems like it's kind of a pain, in that I'd have to parse the search results in XML and then use that to generate my search result page, and if I wanted to customize with meta data in the display, I'd have to used the parsed search results to look up all the results' rows in my database.

Are there any good options for cloud or 3rd party search providers out there that you'd recommend to the Stackoverflow community?

William Jones
  • 18,089
  • 17
  • 63
  • 98
  • why don't you use your own index, e.g Lucene? You can index exactly what you want, and on a field basis (e.g title, content) and it's easy to set up. There's various Ruby ports and/or interfaces, see this question: http://stackoverflow.com/questions/1318604/ruby-alternative-for-lucene – Richard H Dec 14 '10 at 21:21
  • @Richard If worst comes to worst, I would do that, but I'd really like to avoid it. Running my own index on Lucene would entail setting up a server, where right now I have none I need to manage myself. Also, when running Solr/Lucene on my local system to test out Websolr, it gave me a ton of configuration problems I'd like to avoid, like out of memory errors even while I had plenty of memory. – William Jones Dec 14 '10 at 21:38
  • 1
    @Richard: Lucene may be easy to get going, but it's a lot of work to get it optimised. I wouldn't take that step lightly. – skaffman Dec 14 '10 at 22:27

1 Answers1

1

Look up http://www.searchblox.com/. Another alternative, although not a cloud provider, would be to use Elastic search http://www.elasticsearch.org/. It is super simple to setup and use and generally works OOTB.

Also here is a perspective by the creator of ES himself comparing it to other providers: ElasticSearch, Sphinx, Lucene, Solr, Xapian. Which fits for which usage?

Community
  • 1
  • 1
Mikos
  • 8,455
  • 10
  • 41
  • 72