How to speed-up string formatting?

Question

I want to create a long string from a list of objects that contain smaller strings. A simplified example is a chat log:

class Line:
    def __init__(self, user, msg):
        self.user = user
        self.msg = msg

Now I try to create a log:

log = []
for line in lines:
    log.append("{0}: {1}".format(log.user, log.msg))
log_str = "\n".join(log)

On a fast machine I get only around 50000 lines per second (according to tqdm).

Alternatives I tried are just concatenating the string:

log.append(log.user + ": " + log.msg + "\n")

or directly appending it to log_str and both are slower.

As far as I know, concatenation is faster with "\n".join(string_list), but how can I speed up creating the lines?

Answer 1

Do you know about F Strings? they're part of python 3.6+

name = "example"
print(f"any string {name}") 

F strings are evaluated at run time so should be faster than concatenations

You can read more about them here https://stackabuse.com/string-formatting-with-python-3s-f-strings/

Answer 2

Options for representing Line, from fastest to slowest:

• As a tuple ("user", "message").

• As a namedtuple:

  import collections
  Line = collections.namedtuple("Line", "user, msg")
  line = Line("myuser", "mymsg")
  

• With __slots__ and a regular class:

  class Line:
    __slots__ = ("user", "msg")
  
    def __init__(self, user, msg):
        self.user = user
        self.msg = msg
  

Fastest way to create the string using the fastest line representation (tuple or namedtuple)¹:

log_str = "".join([f"{user}: {message}\n" for user, message in lines])

I don't think you'll be able to go faster than that without resorting to Cython or running on PyPy.

Keep in mind your largest bottleneck is the attribute access and not the string formatting. Attribute access in Python is slow.

---

^{1 Yes, the list comprehension is required, and is faster than a generator expression.}