Personally, I've used both Solr and ElasticSearch with with django haystack and have been happy with the results from both. Solr seemed to have a slightly steeper learning curve, but it wasn't enough to be a deal breaker. If you have a cloud-based implementation in mind for your project, you may want to go with ElasticSearch as it features distributed indexing and some other useful features that kind of environment. I don't have any experience with Xapian, so I won't mislead you with any information there.
You may want to check out this post from the creator of ElasticSearch https://stackoverflow.com/a/2288211/1608085 (yes, it will be a bit biased)
Also check out this previous answer for a comparison of Solr and ElasticSearch
https://stackoverflow.com/a/10213568/1608085