1

My Varnish server caches a maps tile server, which is updated real-time from OpenStreetMap every 1 minute. Frequently, an entire area of the map needs to be invalidated -- i.e. 10,000 or even 100,000 tiles at once. Each tile is a single URL (no variances).

Is there an efficient way to run such large scale Varnish invalidation? Ideally objects should remain in cache (so that grace period would continue to work unless a URL flag of nograce is passed in), but marked as no longer valid. Ideally this tight loop would be implemented in VCL itself.

Example URL: http://example.org/tile/10/140/11.pbf (no variance, no query portion) where the numbers are {zoom}/{x}/{y}, and the list of those numbers (i.e. 100,000 at a time) is generated externally every minute and stored in a file. BTW, most likely most of those URLs won't even be in cache.

Yuri Astrakhan
  • 8,808
  • 6
  • 63
  • 97

1 Answers1

1

The answer depends a lot on how those URLs look like. Options are:

  • Using multiple soft purges [1] (beware of the 'soft' part; you'll need the purge VMOD for that) triggered by an external loop (sorry, you cannot do that in VCL). Soft purges set TTL to 0 instead of fully removing objects from the storage.

  • Using a simple ban [2]. However, bans will completely (and lazily) remove matching objects from the storage (i.e. there is not 'soft' flavour for bans).

  • Using the xkey VMOD [3]. The VMOD provides a 'soft' invalidation option, but not sure if a surrogate index would help for your use case.

[1] https://varnish-cache.org/docs/trunk/reference/vmod_purge.html

[2] https://varnish-cache.org/docs/trunk/users-guide/purging.html#bans

[3] https://github.com/varnish/varnish-modules/blob/master/docs/vmod_xkey.rst

Carlos Abalde
  • 1,077
  • 7
  • 12
  • Thanks, I clarified above about the URL format. Will look at your links. – Yuri Astrakhan Dec 23 '19 at 20:18
  • So to set TTL=0, but allowing for grace requests to continue, I have either #1 or #3. For #3, I could simulate another key by using "lower-zoom" indexes, e.g. tile 14/16/16 corresponds to 12/4/4 (1 tile at zoom 12 corresponds to 16 tiles at z14). Using this method I could do external loop through "bigger tiles", invalidating them all. It should be ok to invalidate more than needed because of grace. Thx! – Yuri Astrakhan Dec 24 '19 at 01:21
  • Yes, you're right. Soft bans would be the ideal solution for your use case, but sadly they don't exist (yet). – Carlos Abalde Dec 24 '19 at 08:55
  • I don't understand - I thought xkey mod support soft ban? If I add a few extra xkeys to each tile, I can easily purge everything by simply iterating through all relevant xkeys. Plus xkey's softpurge() supports multiple comma/space separated keys, thus making it batch friendly. – Yuri Astrakhan Dec 25 '19 at 18:44
  • Just saying that bans would be better because you could use regexps for extra flexibility. Just that :) – Carlos Abalde Dec 25 '19 at 21:24
  • Ah, but regexes wouldn't help - the only regex i could possibly use is to ban entire zoom level - i.e. `/tiles/14/.*` to invalidate z14, and that I can already do. Anything more granular would have to be one tile at a time, and for that I could use xkey's capability. Thanks for the thorough reply! – Yuri Astrakhan Dec 25 '19 at 22:47
  • If you want to remove a range of tiles you can use [regex-engine's](https://pypi.org/project/regex-engine/) `numerical_range` method to generate a regex for that. However, my bans don't work yet: https://stackoverflow.com/questions/65632023/varnish-ban-is-added-but-old-object-is-returned – Baschdl Jan 08 '21 at 16:14