1

I use collections.Counter to count the words in a certain string:

s = """Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book. It has survived not only five centuries, but also the leap into electronic typesetting, remaining essentially unchanged. It was popularised in the 1960s with the release of Letraset sheets containing Lorem Ipsum passages, and more recently with desktop publishing software like Aldus PageMaker including versions of Lorem Ipsum."""
lorem = s.lower().split()

Note how this is smaller than the real string I've attempted on, but the conclusion/results are generalizable.

%%timeit
dcomp = Counter(lorem)

# 8 µs ± 329 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

If I use this (which is same as part of source code of cpython/Lib/collections/init.py)

%%timeit
d = dict()
get = d.get
for w in lorem:
    d[w] = get(w, 0) + 1

# 15.4 µs ± 1.61 µs per loop (mean ± std. dev. of 7 runs, 100000 loops each)

EDIT: Use function:

def count():
    d = dict()
    get = d.get
    for w in lorem:
        d[w] = get(w, 0) + 1
    return d

%%timeit
count()
# Still significantly slower. function definition not in timeit loop.
# 14 µs ± 763 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

For a much larger string, the results are similar, which the latter process take around 1.8-2 times as long as the first one.

The part of the source code that works is here:

def _count_elements(mapping, iterable):
    'Tally elements from the iterable.'
    mapping_get = mapping.get
    for elem in iterable:
        mapping[elem] = mapping_get(elem, 0) + 1

Of which mapping is an instance of itself super(Counter, self).__init__() -> dict(). The same speed persisted after I put all of the latter try into a function and called that function. I can't understand where this speed difference originated from. Is there special treatment that python lib gets? Or some caveats I overlooked.

Rocky Li
  • 5,641
  • 2
  • 17
  • 33
  • 1
    Is https://stackoverflow.com/questions/11241523/why-does-python-code-run-faster-in-a-function relevant to your question? – Thierry Lathuille Apr 04 '19 at 19:30
  • Beaten by about 5 seconds by @ThierryLathuille. That was my exact stab in the dark. – roganjosh Apr 04 '19 at 19:31
  • @ThierryLathuille I tried putting the latter in a function and the speed difference persisted. Also tried across different machines, this holds true. – Rocky Li Apr 04 '19 at 19:31

1 Answers1

2

Look more closely at the code for collections/__init__.py. It does define _count_elements as you show, but then it attempts to do from _collections import _count_elements. This is an indication that it is importing from a C library, which is much more optimised and therefore quicker. The Python implementation is only used if the C version is not found.

Daniel Roseman
  • 588,541
  • 66
  • 880
  • 895