12

We have a list of hundreds of thousands of Redis keys containing all sorts of special characters and we would like to bulk delete them. There are some great answers to a similar problem on this question: How to atomically delete keys matching a pattern using Redis

HOWEVER, I can't seem to find an answer for the case where:

  1. We have a large number of keys (hundreds of thousands)
  2. The keys have all manners of special characters like double quotes ("), backslashes (), all sorts of weird Unicode characters, etc.
  3. We are using the windows redis-cli client
  4. Bonus: Ideally we would be able to issue this command as part of a MULTI/EXEC transaction so we can also delete a SET atomically along with the keys.

I would LOVE if we could just do something like the below, but have it handle keys with all of the special characters that give Redis problems:

redis-cli SMEMBERS "myGiganticListOfKeys" | xargs --delim='\n' redis-cli DEL

Unfortunately this just gives the below error:

"C:/Program Files (x86)/Git/bin/xargs.exe": redis-cli: Bad file number

I think this would otherwise work if we didn't have special characters in the keys.

Thanks so much in advance.

Community
  • 1
  • 1
jakejgordon
  • 4,008
  • 7
  • 36
  • 45
  • I'm not intimate with the Windows fork, but if its redis-cli supports the --pipe switch, you can prepare a text file with all the `DEL ` commands and pipe it in directly. – Itamar Haber Oct 01 '15 at 17:31
  • @RyanVincent We actually have the exact list of keys we want to delete -- they are in a set. The problem is taking that set (E.g. SMEMBERS "myKeyName") and deleting all of them when there are a large number, and the keys happen to have special characters. – jakejgordon Oct 01 '15 at 18:38
  • Thanks for the idea @ItamarHaber, but even if that did work -- which I'm guessing it wont as I doubt it's any different from just having the list of keys available from SMEMBERS mykeyName -- it wouldn't be terribly practical as we are trying to write a reusable PowerShell script to delete all of the keys that are in a specified set. Creating a temporary file as part of this process and then cleaning it up wouldn't be ideal. Thanks though. – jakejgordon Oct 01 '15 at 18:41
  • @RyanVincent No problem, I changed the example above to use SMEMBERS instead of the wildcard example so as to clarify the actual problem we're trying to solve. – jakejgordon Oct 01 '15 at 20:10
  • What redis problem do you mean with *all of the special characters that give Redis problems* ? – thepirat000 Oct 01 '15 at 23:35
  • @thepirat000 That is definitely part of the problem -- I can't find documentation regarding which characters are problematic, and hence I'm not sure how to escape them. For example, if I do ```redis-cli SMEMBERS "myGiganticListOfKeys" | xargs --delim='\n' redis-cli DEL``` it errors out, presumably because there are special characters in some of the keys and redis-cli doesn't know what to do with them. It's as if I have to write my own logic to escape the characters. – jakejgordon Oct 02 '15 at 11:37
  • You should try creating an application using a client of your favorite language (see the [client list](http://redis.io/clients)). The redis-cli is a very basic command-line utility intended just to hack/play with Redis. – thepirat000 Oct 02 '15 at 14:11
  • @thepirat000 We are using the .NET ServiceStack client, so writing a little C# program is probably our best bet here. In our particular case, I think we are going to change our entire implementation to store key/value pairs inside of hashes instead of managing independent keys and tracking them instead of separate sets. This was a bad implementation on our part initially and if we switch to this model, we can delete all keys just by deleting the hash. Unfortunately that leaves this question open without a real answer other than "use a different client.". If you write that answer I can accept. – jakejgordon Oct 02 '15 at 15:26

2 Answers2

3

Here's how I solved it, this works for zillions of records without stressing redis.

WARNING: PLEASE DO NOT TRY THIS AT HOME UNATTENDED AND MAKE SURE TO WEAR 
ALL SAFETY EQUIPMENT NECESSARY FOR THE TASK.

Step 1. Dump all the keys you need from redis to a file, lets call this file YES_WE_CAN.sh

redis-cli KEYS "StartsWith*" > YES_WE_CAN.sh

Step 2: Open the file YES_WE_CAN.sh with vi or vim and press : character, then type the following to replace the special character ':

:%s/'/'"'"'/g

This will replace all ' characters with '"'"' escape sequence. (Trust me, this works, keep going!)

Step 3: Prepend to each string redis-cli DEL (dont forget the space at the end):

:%s/^/redis-cli DEL /g

Step 4: Append at the end of each line the ' character:

:%s/$/'/g

Step 5: Save the file and quit using :wq

Step 6: Change the file YES_WE_CAN.sh to executable mode:

chmod +x YES_WE_CAN.sh

Step 7: Run the file:

./YES_WE_CAN.sh

Enjoy your coffee while the script deletes the millions of keys you requested.

Basil Musa
  • 8,198
  • 6
  • 64
  • 63
2

You should try creating an application using a more robust client. See the client list. The redis-cli is a very basic command-line utility intended just to hack/play with Redis.

I agree with you that the best will be to re-design your key/value store.

Consider using a tagging mechanism if you need to invalidate multiple keys: Use a hash to group keys by tag when the keys are added, and then invalidate the entire tag removing all the keys in the hash.

thepirat000
  • 12,362
  • 4
  • 46
  • 72