8

In this answer I found a way to get a reference count of objects in Python.

They mentioned using sys.getrefcount(). I tried it, but I'm getting an unexpected result. When there is 1 reference, it seems as though the count is 20. Why is that?

I looked at the documentation but it doesn't seem to explain the reason.

enter image description here

Community
  • 1
  • 1
  • 1
    Try `sys.getrefcount(257)` and it will probably drop markedly. – Padraic Cunningham Jul 07 '16 at 19:34
  • Try `sys.getrefcount(None)` for an interesting number. – cdarke Jul 07 '16 at 19:35
  • 1
    Thats really strange, i never created a var with 257. Why does it return 3? –  Jul 07 '16 at 19:35
  • @cdarke 4534!? why is that :) –  Jul 07 '16 at 19:36
  • 1
    Try `sys.getrefcount("Valerie")` also. (To be in sync with the other commenters here) – Bhargav Rao Jul 07 '16 at 19:36
  • 1
    Again an implementation detail but small ints from -5 to 257 are interned so the ref count for that range will be much higher – Padraic Cunningham Jul 07 '16 at 19:36
  • @BhargavRao haha guess what, it was 3 again :) –  Jul 07 '16 at 19:37
  • 1
    Just goes to show there is only one `None` object, that LOTS (4533) of things use. Demonstrates that variable names are just references. You got 3 for 257 because 2 other things in python use it (`sys.getrefcount` itself adds one to the reference count). – cdarke Jul 07 '16 at 19:38
  • 2
    Thanks everyone, just learned a lot about python. I guess these are the kinds of things they skip over in the Hello World tutorials :) –  Jul 07 '16 at 19:39
  • @Valerie: in the same spirit, look at `id`. In the C implementation it returns a virtual address. The value is not useful, but compare the address of objects. For example: `a=42`, `b=42`, compare `id(a)` and `id(b)`. Now change one of them and do it again. Now try the same with two lists, and do `list1 = list2` and look at the `id` of each. – cdarke Jul 07 '16 at 19:43
  • Your title was misleading: it wasn't just any object, it was a small integer. – smci Jul 30 '16 at 21:27
  • How many times each integer from -5 to 256 is referenced [link](https://imgur.com/a/dob1VD4) (I have `gc` imported, which may slightly modify the results) – Doot Oct 15 '20 at 15:55

3 Answers3

8

There a bunch of reasons for having many references to an object. Tracking down which one can be difficult, and deciding if it is worth it may bypass your level of interest. Reference counting is of primary interest to developers of debugging applications and python variants.

  • Python tries to keep only one actual value to each reference. So, the 100 you have in your example would be the same 100 that is some internal limit on recursion calls or the same 100 as a current loop index.
  • Python keeps extra references to some common objects, including low integers. The reference count to 1,234,567 is different than the count to 20.
  • Many functions memoize, and keep references, to recent arguments.
  • Some interpreters keep references to recent values and values referenced on recent lines. For example, the previous return value is stored in "_". This means running in the interpreter and running from the command line will give different answers.
  • Like all reference counting schemes, there are mistakes. For example, PyTuple_GetItem() has had some questionable choices.

The exact reference counts and meanings of those counts will be different in PyPi versus C-Python versus IPython. Reference counting is rarely a good tool for finding odd behavior in Python.

Charles Merriam
  • 19,908
  • 6
  • 73
  • 83
5

This object happens to have 20 references to it at the time of the first sys.getrefcount call. It's not just the references you created; there are all sorts of other references to it in other modules and in the Python internals, since (this is an implementation detail) the standard implementation of Python only creates one 100 object and uses it for all occurrences of 100 in a Python program.

user2357112
  • 260,549
  • 28
  • 431
  • 505
  • 1
    More relevantly, if you create your own objects (lists, tuples, class instances), you'll get the expected behavior. – alexis Jul 07 '16 at 19:38
4

It's fun to read the source code for Python2.7 which is very well written and clear to read. (I'm referring to version 2.7.12 if you want to play along at home.) A good place to start in understanding the code is the excellent series of lectures: C Python Internals which starts from a beginner's perspective.

The critical code (written in C) relevant to us appears in the file 'Objects/intobject.c' (I've removed some #ifdef code and slightly modified the creation of a new Integer object for clarity):

    #define NSMALLPOSINTS           257
    #define NSMALLNEGINTS           5
    static PyIntObject *small_ints[NSMALLNEGINTS + NSMALLPOSINTS];

    PyObject *
    PyInt_FromLong(long ival)
    {
        register PyIntObject *v;
        if (-NSMALLNEGINTS <= ival && ival < NSMALLPOSINTS) {
            v = small_ints[ival + NSMALLNEGINTS];
            Py_INCREF(v);
            return (PyObject *) v;
        }
        /* Inline PyObject_New */
        v = (PyIntObject *)Py_TYPE(v);
        PyObject_INIT(v, &PyInt_Type);
        v->ob_ival = ival;
        return (PyObject *) v;
    }

So essentially it creates a preset array containing all the numbers from -5 to 256 inclusive and uses those objects (increasing their reference counts using the Py_INCREF macro) if possible. If not, it will create a brand new PyInt_Type object, which is initialised with a reference count of 1.

The mystery of why every number seems to have a reference count of 3 (actually almost any new object) is only revealed when you look at the byte code that Python generates. The Virtual Machine operates with a value stack (a bit like in Forth), and each time an object is put on the value stack it increments the reference count.

So what I suspect is happening is that your code is itself providing all 3 of the references you see, since for numbers not in the list of small values, you should get a unique object. The first reference is presumably in the value stack for the caller of getrefcount when it makes the call; the second is in the local variable list for the getrefcount frame; the third is likely on the value stack in the getrefcount frame while it looks up its reference count.

A useful tool if you want to dig further into the issue are the 'compile' command, and the 'dis' command (disassemble) which is in the 'dis' module, which will together allow you to read the actual byte code generated by any piece of Python code, and should help you find out exactly when and where the third reference is being created.

As for the higher reference counts for small values, when you start Python, it automatically loads the entire Standard Library and runs quite a lot of Python module initialisation code before you get to start interpreting your own code. These modules are holding their own copies many of the small integers (and the None object which is also unique).

QuesterZen
  • 121
  • 5