7

I'm running Redis server 2.8.17 on a Debian server 8.5. I'm using Redis as a session store for a Django 1.8.4 application.

I haven't changed the software configuration on my server for a couple of months and everything was working just fine until a week ago when Django began raising the following error:

MISCONF Redis is configured to save RDB snapshots but is currently not able to persist to disk.  Commands that may modify the data set are disabled.  Please check Redis logs for details...

I checked the redis log and saw this happening about once a second:

1 changes in 900 seconds.  Saving...
Background saving started by pid 22213
Failed opening .rdb for saving: Permission denied
Background saving error

I've read these two SO questions 1, 2 but they haven't helped me find the problem.

ps shows that user "redis" is running the server:

redis   26769   ...   /usr/bin/redis-server *.6379

I checked my config file for the redis file name and path:

grep ^dir /etc/redis/redis.conf =>
dir /var/lib/redis

grep ^dbfilename /etc =>
dbfilename dump.rdb

The permissons on /var/lib/redis are 755 and it's owned by redis:redis. The permissons on /var/lib/redis/dump.rdb are 644 and it's owned by redis:redis too.

I also ran strace on the server process:

ps -C redis-server  # pid = 26769
sudo strace -p 26769 -o /tmp/strace.out

But when I examine the output, I don't see any errors. In particular I don't see a "Permission denied" error as I would expect.

Also, /var/lib/redis is not an NFS directory.

Does anyone know what else could be causing this? I'd hate to have to stop using Redis. I know I can run the command "set stop-writes-on-bgsave-error yes" but that doesn't solve the problem.

This is now happening on a daily basis and the only way I can stop the error is to restart the Redis server.

Thanks.

Community
  • 1
  • 1
Jim
  • 13,430
  • 26
  • 104
  • 155
  • Is the output of this: `grep ^dir /etc/redis/redis.conf => dir /var/lib/redisredis/redis.conf` accurate? – 2ps Oct 18 '16 at 06:45
  • Also, why does your log file refer to one PID 22213, while your ps command shows another? Was this just a quirk of timing or did you restart redis in between debugging commands? – 2ps Oct 18 '16 at 06:47
  • Sorry but that was a typo in the grep which I've fixed in my question. I believe the PID 22213 refers to a new background saving process, not the Redis server process itself which has a PID of 26769. – Jim Oct 18 '16 at 15:35
  • Have you already done an `lsof /var/lib/redis/dump.rdb`? – 2ps Oct 18 '16 at 22:46

5 Answers5

10

I just had a similar issue. Despite my config file being correct, when I checked the actual dbfilename and dir in redis-client, they were incorrect.

Run redis-cli and then

CONFIG GET dbfilenamewhich should return something like 

1) "dbfilename"
2) "dump.rdb"

1) is just the key and 2) the value. Similarly then run CONFIG GET dir should return something like

1) "dir"
2) "/var/lib/redis"

Confirm that these are correct and if not, set them with CONFIG SET dir /correct/path

Hope this helps!

userman123
  • 101
  • 1
  • 4
4

If you have moved Redis to a new mounted volume: /mnt/data-01.

sudo vim /etc/systemd/system/redis.service

Set ReadWriteDirectories=-/mnt/data-01

sudo mkdir /mnt/data-01/redis

Set chown and chmod on new redis data dir and rdb file.
The permissons on /var/lib/redis are 755 and it's owned by redis:redis
The permissons on /var/lib/redis/dump.rdb are 644 and it's owned by redis:redis

Switch configurations while redis is running

$ redis-cli
127.0.0.1:6379> CONFIG SET dir /data/tmp
redis-cli 127.0.0.1:6379> CONFIG SET dbfilename temp.rdb
127.0.0.1:6379> BGSAVE
tail /var/log/redis/redis.cnf (verify saved)

holmberd
  • 2,393
  • 26
  • 30
  • 1
    I encountered this issue , during a upgrade from 14.04 to 16.04, which upgraded redis to version 3 from version 2. The data directory was non-standard and needed to be added as a ReadWriteDirectories in the /etc/systemd/system/redis.service file. Many thanks! – eodonohoe Sep 16 '17 at 22:00
4

Start Redis Server in a directory where Redis has write permissions

The answers above will definitely solve your problem, but here's what's actually going on:

The default location for storing the rdb.dump file is ./ (denoting current directory). You can verify this in your redis.conf file. Therefore, the directory from where you start the redis server is where a dump.rdb file will be created and updated.

Since you say your redis server has been working fine for a while and this just started happening, it seems you have started running the redis server in a directory where redis does not have the correct permissions to create the dump.rdb file.

To make matters worse, redis will also probably not allow you to shut down the server either until it is able to create the rdb file to ensure the proper saving of data.

To solve this problem, you must go into the active redis client environment using redis-cli and update the dir key and set its value to your project folder or any folder where non-root has permissions to save. Then run BGSAVE to invoke the creation of the dump.rdb file.

CONFIG SET dir "/hardcoded/path/to/your/project/folder"
BGSAVE

(Now, if you need to save the dump.rdb file in the directory that you started the server in, then you will need to change permissions for the directory so that redis can write to it. You can search stackoverflow for how to do that).

You should now be able to shut down the redis server. Note that we hardcoded the path. Hardcoding is rarely a good practice and I highly recommend starting the redis server from your project directory and changing the dir key back to./`. 

CONFIG SET dir "./"
BGSAVE

That way when you need redis for another project, the dump file will be created in your current project's directory and not in the hardcoded path's project directory.

Govind Rai
  • 14,406
  • 9
  • 72
  • 83
2

You can resolve this problem by going into the redis-cli

Type redis-cli in the terminal

Then write config set stop-writes-on-bgsave-error no and it resolved my problem.

Hope it resolved your problem

VIKAS KOHLI
  • 8,164
  • 4
  • 50
  • 61
0

Up to redis 3.2 it shipped with pretty insane defaults which opened the port to the public. In combination with the CONFIG SET instruction everybody can change your redis config from outside easily. If the error starts after some time, someone probably changed your config.

On your local machine check that

telnet SERVER_IP REDIS_PORT

is denied. Otherwise check your config, you should have the setting

bind 127.0.0.1

enabled.

Dependent on the user that runs redis, you should also check for damage that the intruder has done.

Krisdigital
  • 630
  • 1
  • 8
  • 16