14

I would like to export all keys and values from a memcached server, using python-memcache. There is no such function in that module. How to do it then?

Perhaps something more complicated involving the "socket" module would be needed.

kenorb
  • 155,785
  • 88
  • 678
  • 743
Falken
  • 2,528
  • 2
  • 17
  • 10

7 Answers7

27

This will get you all the keys on a memcached server, you can use any memcached client library to get the value for each key.

import telnetlib

def get_all_memcached_keys(host='127.0.0.1', port=11211):
    t = telnetlib.Telnet(host, port)
    t.write('stats items STAT items:0:number 0 END\n')
    items = t.read_until('END').split('\r\n')
    keys = set()
    for item in items:
        parts = item.split(':')
        if not len(parts) >= 3:
            continue
        slab = parts[1]
        t.write('stats cachedump {} 200000 ITEM views.decorators.cache.cache_header..cc7d9 [6 b; 1256056128 s] END\n'.format(slab))
        cachelines = t.read_until('END').split('\r\n')
        for line in cachelines:
            parts = line.split(' ')
            if not len(parts) >= 3:
                continue
            keys.add(parts[1])
    t.close()
    return keys
lrd
  • 271
  • 3
  • 4
  • A colleague is getting "zero length field name in format" at the line "t.write('stats cachedump...". Probably for earlier versions of Python you need "stats cachedump {0} 200000". – Mike Stoddart Apr 06 '16 at 14:01
  • It works for me in local server *BUT* in production environment it is not usable because it gets too slow (you can wait for many minutes) – jobima Jan 31 '19 at 10:59
15

use the memdump and memcat utilities from the libmemcached suite. They can't guarantee you'll get all the data but they're easy to use.

Note: On ubuntu/debian you can get these by intstalling the libmemcached-tools package and they are called memcdump and memccat.

dump all keys:

memcdump --servers=localhost

dump all values:

memccat --servers=localhost `memcdump --servers=localhost`

of course you still have to match up the keys and values - I'd suggest dumping the keys to a file and then using that as the input for memcat (this ensures consistency). Then of course you need to split the values - a full stop is the delimiter I believe - and then pair the keys and values sequentially. There's probably a script out there somewhere...

Sandeep
  • 28,307
  • 3
  • 32
  • 24
scytale
  • 12,346
  • 3
  • 32
  • 46
7

There is no way to do that. Memcache protocol does not define any command to iterate over keys. You have to know the key to retrieve value.

vartec
  • 131,205
  • 36
  • 218
  • 244
  • Still, via telnet you can run "stat items" and then "stats cachedump" Too bad the functions do not exist in python-memcache – Falken Apr 21 '11 at 13:21
  • @Falken: true, but that doesn't give you information about individual items, just aggregated stats. – vartec Apr 21 '11 at 13:26
  • 2
    @Falken: As for "cachedump" it's not guaranteed to work, it's meant for debugging only, return is limited to 1MB – vartec Apr 21 '11 at 13:34
1

The easiest way is to use python-memcached-stats package, https://github.com/abstatic/python-memcached-stats

The keys() method should get you going.

abhishek_M
  • 1,110
  • 11
  • 19
  • With Python3.8 that failed with an error at `from memcached_stats import MemcachedStats`. It's a pity as I was wanting a Pythonic way of doing this. – Keeely Feb 15 '21 at 13:04
  • Above doesn't support python3+, feel free to fork and update or raise a PR! – abhishek_M Feb 15 '21 at 16:28
  • OK, when I've finished porting your code to Python3, I'll come back here and up-vote this :). – Keeely Feb 15 '21 at 19:10
0

As mentioned by others in many places, in the general case, there is no way to list all the keys that stored in a memcached instance. E.g., Memcached: List all keys, Couldn't retrieve all the memcache keys via telnet client

You can, however, list something like the first 1Meg of keys, which is usually enough to have an idea about what are stored in memcache server during development. Basically, you can have two option to extract items from memcache server:

(1) To retrieve a subset of keys and values, you can the method introduced above by use @lrd

However, when the data is very large (e.g., millions of records), this method can be very time-consuming. What's more, this method can only retrieve only a subset of the keys & values.

(2) In cases where you would like to iterate all the items of the memcached server, logging the keys when one adds/set/cas items to memcache server is a much cheaper solution. Then you can read through the log file to get all the keys and get the values from memcache server. As discussed in this mailing list: List all objects in memcached

Community
  • 1
  • 1
Good Will
  • 1,220
  • 16
  • 10
0

In Bash, this script may help:

while read -r key; do
    memccat --servers=localhost $key > "$key.dump";
done < <(memcdump --server localhost)

Instead of memccat, you can also connect to port directly and send get SOME-KEY command (example using Netcat: echo "get $key" | nc localhost 11211).

Related:

kenorb
  • 155,785
  • 88
  • 678
  • 743
-3

You're looking for the 'flush_all' memcache command: http://code.google.com/p/memcached/wiki/NewCommands#flush_all

With python-memcached, it looks something like this:

>>> import memcache
>>> c = memcache.Client(('127.0.0.1:11211',))
>>> c.flush_all()
Paul
  • 791
  • 5
  • 8