Is it possible to boost 'newest' items using elasticsearch? (FOQElasticaBundle)

Question

I'm currently implementing elasticsearch in my Symfony2 application via the FOQElasticaBundle and so far it's been working great based on boosts applied to various fields of my "Story" entity. Here is the config:

foq_elastica:
    clients:
        default: { host: localhost, port: 9200 }

    indexes:
        website:
            client: default
            types:
                story:
                    mappings:
                        title: { boost: 8 }
                        summary: { boost: 5 }
                        text: { boost: 3 }
                        author:
                    persistence:
                        driver: orm # orm, mongodb, propel are available
                        model: Acme\Bundle\StoryBundle\Entity\Story
                        provider:
                            query_builder_method: createIsActiveQueryBuilder
                        listener:
                            service: acme_story.search_index_listener
                        finder:

However I'd like to also apply a boost based on the "published_at" date of the story, so that a story published yesterday would appear in the results before a story published 6 months ago - even if the older story had a slightly better score (obviously this will need a bit of tweaking). Is this possible?

If anyone could let me know how to achieve this using FOQElasticaBundle that would be great, but otherwise I'd appreciate it if you could let me know how to achieve this directly in elasticsearch so I can try and implement the behaviour myself and contribute to the bundle if needs be.

Thanks.

Answer 1

Whew, after much experimentation and hours of trawling the Interweb I finally managed to get the desired behavior! (Full credit goes to Clinton Gormley.)

Mapping configuration:

mappings:
    title: { boost: 8 }
    summary: { boost: 5 }
    text: { boost: 3 }
    author:
    publishedAt: { type: date }

Here is the code using the PHP client, Elastica, to dynamically build the query to boost using the original mapping AND the published date:

$query = new \Elastica_Query_Bool();
$query->addMust(new \Elastica_Query_QueryString($queryString));

$ranges = array();
for ($i = 1; $i <= 5; $i++) {
    $date = new \DateTime("-$i month");

    $currentRange = new \Elastica_Query_Range();
    $currentRange->addField('publishedAt', array(
        'boost' => (6 - $i),
        'gte' => $date->getTimestamp()
    ));

    $ranges[] = $currentRange->toArray();
}

$query->addShould($ranges);

/** @var $pagerfanta Pagerfanta */
$pagerfanta = $this->getFinder()->findPaginated($query);

And for those of you more interested in the raw elasticsearch query (only with 3 date ranges for brevity)...

curl -XPOST 'http://localhost:9200/website/story/_search?pretty=true' -d '
{
  "query" : {
    "bool" : {
      "must" : {
        query_string: {
          query: "<search term(s)>"
        }
      },
      "should" : [
        {
          "range" : {
            "publishedAt" : {
              "boost" : 5,
              "gte" : "<1 month ago>"
            }
          }
        },
        {
          "range" : {
            "publishedAt" : {
              "boost" : 4,
              "gte" : "<2 months ago>"
            }
          }
        },
        {
          "range" : {
            "publishedAt" : {
              "boost" : 3,
              "gte" : "<3 months ago>"
            }
          }
        }
      ]
    }
  }
}'

Answer 2

You can use a decay scoring function, to decrease the scoring versus time :

{
 "query": {
 "function_score": {
    "functions": [
     {
      "linear": {
        "pubdate": {
          "origin": 1398673886,
          "scale": "1h",
          "offset": 0,
          "decay": 0.1
        }
      }
    }
    ]
   }
  }
 }

Answer 3

A full elasticsearch 5 example based on function_score. See this blogpost and function_score docs for more info.

Allows for boosting more recent entries based on multiple date ranges, with varying strengths, on a gaussian curve without "hard cutoffs".

{
    "query": {
        "function_score": {

            "score_mode": "sum", // All functions outputs get summed
            "boost_mode": "multiply", // The documents relevance is multiplied with the sum

            "functions": [
                {
                    // The relevancy of old posts is multiplied by at least one.
                    // Remove if you want to exclude old posts
                    "weight": 1
                },
                {
                    // Published this month get a big boost
                    "weight": 5,
                    "gauss": {
                        "date": { // <- Change to your date field name
                            "origin": "2017-04-07", // Change to current date
                            "scale": "31d",
                            "decay": 0.5
                        }
                    }
                },
                {
                    // Published this year get a boost
                    "weight": 2,
                    "gauss": {
                        "date": { // <- Change to your date field name
                            "origin": "2017-04-07", // Change to current date
                            "scale": "356d",
                            "decay": 0.5
                        }
                    }
                }
            ],

            "query": {
                // The rest of your search here, change to something relevant
                "match": { "title": "< your search string >" }
            }
        }
    }
}