Spring Redis - Indexes not deleted after main entry expires

Question

I am saving new entries with a Spring Data Repository. I have a TTL of 10 seconds for each entry.

When I save an entry with indexes, here is what i get in Redis

127.0.0.1:6379> keys *
1) "job:campaignId:aa"
2) "job:a6d6e491-5d75-4fd0-bd8e-71692f6d18be"
3) "job:recipient:dd"
4) "job:a6d6e491-5d75-4fd0-bd8e-71692f6d18be:phantom"
5) "job:listId:cc"
6) "job:accountId:bb"
7) "job"
8) "job:a6d6e491-5d75-4fd0-bd8e-71692f6d18be:idx"

After the expiration, I still have data :

127.0.0.1:6379> keys *
1) "job:campaignId:aa"
2) "job:recipient:dd"
3) "job:listId:cc"
4) "job:accountId:bb"
5) "job"
6) "job:a6d6e491-5d75-4fd0-bd8e-71692f6d18be:idx"

Without any TTL.

Why aren't they deleting themself ? How could I do that ?

score 33 · Accepted Answer · answered Jan 17 '17 at 11:25

33

Spring Data Redis Repositories use multiple Redis features to persist domain objects in Redis.

Domain objects are stored primarily in a hash (job:a6d6e491-5d75-4fd0-bd8e-71692f6d18be). Any expiry is applied directly to the hash so Redis can expire the key. Spring Data Redis also maintains secondary indexes (job:campaignId:aa, job:recipient:dd) to provide lookup by particular field values. Individual elements inside a set cannot be expired. Only the whole data structure can expire, but that's not the thing you want to do because all non-expired elements would disappear that way.

So Spring Data Redis persists a copy of the original hash as phantom hash (job:a6d6e491-5d75-4fd0-bd8e-71692f6d18be:phantom) with a slightly longer TTL.

Spring Data Redis subscribes to key-events (with setting @EnableRedisRepositories(enableKeyspaceEvents = EnableKeyspaceEvents.ON_STARTUP) to listen to expiry events. As soon as the original hash expires, Spring Data Redis loads the phantom hash to perform cleanups (remove references from secondary indexes).

The reason why the cleanup of your data wasn't performed can have multiple reasons:

If you run a console application just to insert data and terminate, then the expiry removes the hashes but does not perform the index cleanup since your application is not running anymore. Any events published by Redis are transient, and if your application is not listening, then these events are lost
If you have enabled repository support with just @EnableRedisRepositories (without enabling keyspace-events), then the Keyspace event listener is not active, and Spring Data Redis is not subscribed to any expiry events.

answered Jan 17 '17 at 11:25

mp911de

17,546
2
55
95

Thank you ! That's the explaination I needed. Did I missed something in the documentation ? Another related question, how could I get the value of the expired key before Spring Data deletes it ? – BkSouX Jan 17 '17 at 13:41
Observe `RedisKeyExpiredEvent` in your application. It contains the key (bytes) and the expired domain object. See http://docs.spring.io/spring-data/redis/docs/current/reference/html/#redis.repositories.expirations for details on expiration. – mp911de Jan 17 '17 at 13:48
After testing a bit with @EnableRedisRepositories(enableKeyspaceEvents = EnableKeyspaceEvents.ON_STARTUP), I see that the phantom is deleted in the same time as the real key but in the documentation, it says it will be deleted 5 minutes after. Maybe there is a property or something ? – BkSouX Jan 17 '17 at 13:55
I force the TTL to 5 seconds in my model @TimeToLive public long getTimeToLive() { return 5; } – BkSouX Jan 17 '17 at 13:56
The `:phantom` TTL in Redis is 5 minutes more than the regular key TTL. – mp911de Jan 17 '17 at 13:56
That's what I see in the documentation but with my test, it's not 5 minutes after but in the same time. Im going to create a punkler – BkSouX Jan 17 '17 at 13:58
That's correct. [`RedisKeyValueAdapter`](https://github.com/spring-projects/spring-data-redis/blob/master/src/main/java/org/springframework/data/redis/core/RedisKeyValueAdapter.java#L781-L797) removes the `:phantom` key as part of the cleanup. TTL on `:phantom` is just an additional measure to make sure it expires. That's why you need to listen to `RedisKeyExpiredEvent`. – mp911de Jan 17 '17 at 14:08
Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/133362/discussion-between-bksoux-and-mp911de). – BkSouX Jan 17 '17 at 14:11
2

Why isn't enableKeyspaceEvents turned on by default? I don't see who would not want this behavior. Without it, it means your Redis is eventually going to run out of memory sooner or later. – HappyCoder86 Oct 19 '17 at 23:05
1

Redis won't run out of memory, phantom keys have a TTL set, too. `enableKeyspaceEvents` requires specific config (not applicable in several cases such as AWS, Azure or other restricted setups) and isn't reliably working on Redis Cluster because keyspace notifications are node-local. – mp911de Oct 20 '17 at 10:23

Ivan · Answer 2 · 2023-08-04T16:15:47.967

Another possible reason for indexes not being removed after the main entry expires is using a double-colon (:) in the keyspace:

@RedisHash(value = "app-name:entity-name")

According to the following topics, a double-colon in the keyspace is not supported:

You can also verify this by looking at the implementation of the keyspace event handling method org.springframework.data.redis.core.RedisKeyValueAdapter.MappingExpirationListener#onMessage (org.springframework.data:spring-data-redis:3.1.0).

Avinash · Answer 3 · 2017-01-17T10:25:49.947

-1

No key/values will be deleted automatically if you don't set Expiration Time.

So to automatically delete a data you have to set Expiration Time.

redis> SET mykey "Hello"
"OK"
redis> EXPIRE mykey 10
(integer) 1

Ref : https://redis.io/commands/expire

Below is the Spring code snippet to add a data to redis and set expiration time

@Component
public class RedisUtil {
    @Autowired
    private RedisTemplate<String, String> template;

    @Resource(name = "redisTemplate")
    ValueOperations<String, String> ops;

    public boolean addValue(String key, String value) {

        if (template.hasKey(Constants.REDIS_KEY_PREFIX + key)) {
            // key is already there
            return false;
        } else {
            ops.set(Constants.REDIS_KEY_PREFIX + key, value);
            template.expireAt(Constants.REDIS_KEY_PREFIX + key, 10);
        }
        return true;
    }
}

edited Jan 17 '17 at 10:25

answered Jan 17 '17 at 10:15

Avinash

4,115
2
22
41

ok so i have to do it manually ? I use Spring Data Redis with repositories and the key/value created for the indexes ( @Indexed in the model ) are not deleted when the main key for the job are well expired and so deleted.. – BkSouX Jan 17 '17 at 10:18
Yes you have to set the expiration time through your code for each data. – Avinash Jan 17 '17 at 10:20
well, i thought it would be simpler with spring.. Just doing a repo.save(entry) – BkSouX Jan 17 '17 at 10:23
see the updated answer, I have added a code snippet for your reference. – Avinash Jan 17 '17 at 10:26
thanks, I understand how to do it using a RedisTemplate. I was trying to use repositories http://docs.spring.io/spring-data/redis/docs/current/reference/html/#redis.repositories and annotated indexes http://docs.spring.io/spring-data/redis/docs/current/reference/html/#redis.repositories.indexes – BkSouX Jan 17 '17 at 10:29

Spring Redis - Indexes not deleted after main entry expires

3 Answers3

Linked