How to broadcast cache invalidate messages to all servers running a web app?

Question

I have a Java based web app hosted on AWS. It is read-mostly so it makes a lot of sense to cache objects retrieved from the database for performance.

When I do update an object, I would like to be able to broadcast to all the servers that the object was saved and it should be invalidated from all local caches.

The does not need to be real time. Stale objects are annoying and need to be flushed within about 20 seconds. Users notice if they stick around for minutes. Cache invalidation does not have to happen the millisecond that objects get saved.

What I've thought about

I've looked into broadcast technologies just as jGroups, but jGroups isn't supported on AWS.
I don't think that Amazon's SQS messaging service can be made into a broadcast service.
I'm considering using the database for this purpose: I'd write events to a database table and have each server poll this table every few seconds to get a new list items.

What's the reason jGroups isn't supported; is it multicast limitations or something else? — chrylis -cautiouslyoptimistic-, Aug 29 '13 at 13:17
That's a fairly nasty fail. Would it work to have the systems with caches each keep some sort of open connection to the database server, which could send a flush packet on updates? — chrylis -cautiouslyoptimistic-, Aug 29 '13 at 13:26
Also, have you looked at SNS? Looks like it's basically the pub-sub SQS adapter you're looking for. — chrylis -cautiouslyoptimistic-, Aug 29 '13 at 13:28
I hadn't looked into SNS. It appears to be a bit heavyweight for this purpose. The messages would get delivered through HTTP POST to each web server or each server could create a SQS queue and subscribe to have messages delivered there, then poll the queue. It looks like a possible solution though, if you write it up as an answer, it deserves an upvote. — Stephen Ostermiller, Aug 29 '13 at 13:38

score 2 · Answer 1 · answered Aug 29 '13 at 13:43

Two options come to mind. The first is to use Amazon SNS, which can use SQS as a delivery backend. This might be overkill, though, since it's designed as a frontend to lots of delivery types, including e-mail and SMS.

The approach I'd try is something along the lines of Comet-style push notifications. Have each machine with a cache open a long-lived TCP connection to the server who's responsible for handling updates, and send a compact "invalidate" message from that server to everyone who's listening. As a special-purpose protocol, this could be done with minimal overhead, perhaps just by sending the object ID (and class if necessary).

score 1 · Answer 2 · answered Dec 22 '22 at 08:21

Redis is handy solution for broadcasting a message to all subscribers on a topic. It is convenient because it can be used as a docker container for rapid prototyping, but is also offered by AWS as a managed service for multi-node clusters.

Setting up a ReactiveRedisOperations bean:

@Bean
public ReactiveRedisOperations<String, Notification> notificationTemplate(LettuceConnectionFactory lettuceConnectionFactory){
    RedisSerializer<Notification> valueSerializer = new Jackson2JsonRedisSerializer<>(Notification.class);
    RedisSerializationContext<String, Notification> serializationContext = RedisSerializationContext.<String, Notification>newSerializationContext(RedisSerializer.string())
            .value(valueSerializer)
            .build();
    return new ReactiveRedisTemplate<>(lettuceConnectionFactory, serializationContext);
}

Subscribing on a topic:

@Autowired
private ReactiveRedisOperations<String, Notification> reactiveRedisTemplate;

@Value("${example.topic}")
private String topic;

@PostConstruct
private void init() {
    this.reactiveRedisTemplate
            .listenTo(ChannelTopic.of(topic))
            .map(ReactiveSubscription.Message::getMessage)
            .subscribe(this::processNotification);
}

Publishing a message on a topic:

@Autowired
private ReactiveRedisOperations<String, Notification> redisTemplate;

@Value("${example.topic}")
private String topic;

public void publish(Notification notification) {
    this.redisTemplate.convertAndSend(topic, notification).subscribe();
}

RedisInsight is a GUI that can be used for interacting with redis.

Here is a complete sample implementation using spring-data-redis.

How to broadcast cache invalidate messages to all servers running a web app?

What I've thought about

2 Answers2

Linked