Python string join performance

Question

There are a lot of articles around the web concerning Python performance. The first thing you read is concatenating strings should not be done using '+'; avoid s1 + s2 + s3, and instead use str.join

I tried the following: concatenating two strings as part of a directory path: three approaches:

1. '+' which I should not do
2. str.join
3. os.path.join

Here is my code:

import os, time

s1 = '/part/one/of/dir'
s2 = 'part/two/of/dir'
N = 10000

t = time.clock()
for i in xrange(N):
    s = s1 + os.sep + s2
print time.clock() - t

t = time.clock()
for i in xrange(N):
    s = os.sep.join((s1, s2))
print time.clock() - t

t = time.clock()
for i in xrange(N):
    s = os.path.join(s1, s2)
print time.clock() - t

Here the results (Python 2.5 on Windows XP):

0.0182201927899
0.0262544541275
0.120238186697

Shouldn't it be exactly the other way around?

Answer 1

Most of the performance issues with string concatenation are ones of asymptotic performance, so the differences become most significant when you are concatenating many long strings.

In your sample, you are performing the same concatenation many times. You aren't building up any long string, and it may be that the Python interpreter is optimizing your loops. This would explain why the time increases when you move to str.join and path.join - they are more complex functions that are not as easily reduced. (os.path.join does a lot of checking on the strings to see if they need to be rewritten in any way before they are concatenated. This sacrifices some performance for the sake of portability.)

By the way, since file paths are not usually very long, you almost certainly want to use os.path.join for the sake of the portability. If the performance of the concatenation is a problem, you're doing something very odd with your filesystem.

Answer 2

The advice is about concatenating a lot of strings.

To compute s = s1 + s2 + ... + sn,

1. using +. A new string s1+s2 is created, then a new string s1+s2+s3 is created,..., etc, so a lot of memory allocation and copy operations is involved. In fact, s1 is copied n-1 times, s2 is copied n-2 time, ..., etc.

2. using "".join([s1, s2, ..., sn]). The concatenation is done in one pass, and each char in the strings is copied only once.

In your code, join is called on each iteration, so it's just like using +. The correct way is collect the items in an array, then call join on it.

Answer 3

It is true you should not use '+'. Your example is quite special. Try the same code with:

s1 = '*' * 100000
s2 = '+' * 100000

Then the second version (str.join) is much faster.

Answer 4

Shouldn't it be exactly the other way round?

Not necessarily. I don't know the internals of Python well enough to comment specifically, but some common observations are that your first loop uses a simple operator + which is probably implemented as a primitive by the runtime. In contrast, the other loops first have to resolve a module name, resolve the variable/class found there and then call a member function on that.

Another note is that your loop might simply be too small to yield significant numbers. Considering your low overall running time, this probably makes your tests useless.

Also, your test case is highly specialized on two short strings. Such cases never give a clear picture of edge case performance.

Answer 5

String concatenation (+) has an optimized implementation on CPython. But this may not be the case on other architectures like Jython or IronPython. So when you want your code to perform well on these interpreters you should use the .join() method on strings. os.path.join() is specifically meant to join filesystem paths. It takes care of different path separators, too. This would be the right way to build a file name.

Answer 6

Trying to do the same test again in 2020 with Python 3.9, join remains plenty fast but plain concatenation has seen some improvement too:

from io import StringIO
from array import array

loop_count = 10000
strings = [str(num) for num in range(loop_count)]
bytestrings = [b'%d' % num for num in range(loop_count)]

# 1.1453419709578156 seconds for 1000 repetitions (fastest of 5)
def concat():
  out = ''
  for s in strings:
    out += s
  return out

# 1.468063525040634 seconds for 1000 repetitions
# Removing decode() does not make it faster
def bytearray():
  out = array('b')
  for b in bytestrings:
    out.frombytes(b)
  return out.tobytes().decode()

# 0.9110414159949869 seconds for 1000 repetitions
def join():
  # I am rebuilding the list on purpose: I don't want to include
  # the overhead of printing numbers, but I do want to include
  # the overhead of building the list
  str_list = []
  for s in strings:
    str_list.append(s)
  return ''.join(str_list)

# 1.174764319031965 seconds for 1000 repetitions
def stringio():
  io = StringIO()
  for s in strings:
    io.write(s)
  return io.getvalue()

Answer 7

Here is a link to the Python wiki, where there are notes regarding string concatenation and also that "this section is somewhat wrong with Python 2.5. Python 2.5 string concatenation is fairly fast".

I believe that string concatenation had a big improvement since Python 2.5, and that, although str.join is still faster (specially for big strings), you will not see as much improvement as in older Python versions.

String Concatenation