4

How to write a function that takes n (where n > 0) and returns the list of all combinations of positive integers that sum to n? This is a common question on the web. And there are different answers provided such as 1, 2 and 3. However, in the answers provided, they use two functions to solve the problem. I want to do it with only one single function. Therefore, I coded as follows:

def all_combinations_sum_to_n(n):
    from itertools import combinations_with_replacement

    combinations_list = []

    if n < 1:
        return combinations_list

    l = [i for i in range(1, n + 1)]
    for i in range(1, n + 1):
        combinations_list = combinations_list + (list(combinations_with_replacement(l, i)))

    result = [list(i) for i in combinations_list if sum(i) == n]
    result.sort()
    return result

If I pass 20 to my function which is all_combinations_sum_to_n(20), the OS of my machine kills the process as it is very costly. I think the space complexity of my function is O(n*n!). How do I modify my code so that I don't have to create any other function and yet my single function has an improved time or space complexity? I don't think it is possible by using itertools.combinations_with_replacement.

UPDATE

All answers provided by Barmar, ShadowRanger and pts are great. As I was looking for an efficient answer in terms of both memory and runtime, I used https://perfpy.com and selected python 3.8 to compare the answers. I used six different values of n and in all cases, ShadowRanger's solution had the highest score. Therefore, I chose ShadowRanger's answer as the best one. The scores were as follows:

enter image description here

plpm
  • 539
  • 3
  • 12
  • 1
    The problem isn't the time complexity, it's the space complexity. All those intermediate lists that you're creating in `combinations_list` take up an enormous amount of space. – Barmar Feb 01 '23 at 15:49
  • @Barmer You are right. I just modified my question. – plpm Feb 01 '23 at 15:50
  • 1
    You can save some space by using `combinations_list += ...` instead of `combinations_list = combination_list + ...` because this extends the list in place rather than copying the list. – Barmar Feb 01 '23 at 15:51
  • @Barmar Good suggestion. But after modifying my code the problem is still there. – plpm Feb 01 '23 at 15:53
  • You shouldn't need two functions. Just use recursion in this function. For each number `i` in the range, append it to all the results of `all_combinations_sum_to(n-i)`. – Barmar Feb 01 '23 at 15:54
  • @Barmar I am not sure if I understood your hint. Did you mean `for i in range(1, n + 1):combinations_list.append(all_combinations_sum_to_n(n-i))` as it doesn't work? – plpm Feb 01 '23 at 15:59
  • @plpm: OOC, how did you come up with those combined scores? I made [my own benchmark](https://perfpy.com/219) that: 1) Won't run above `n == 20` or so (it dies with an error for `n` of `25` or higher anyway), and 2) Gives a slight advantage to pts's answer on speed for `n` of `20` (not huge, but mine took ~30% longer), though at the cost of using a *lot* more memory. I also tweaked Barmar's code a bit to remove unnecessary sorts in the recursive case, though that makes little difference; the code is ~1000x slower than the other two solutions, even if it's relatively memory-efficient. – ShadowRanger Feb 01 '23 at 20:41
  • @ShadowRanger I did it differently. I put each solution in a separate code box while leaving the setup box empty. – plpm Feb 01 '23 at 23:07
  • Why do you want to restrict yourself to defining only one function? – Kelly Bundy Feb 02 '23 at 02:04

3 Answers3

3

You've got two main problems, one causing your current problem (out of memory) and one that will continue the problem even if you solve that one:

  1. You're accumulating all combinations before filtering, so your memory requirements are immense. You don't even need a single list if your function can be a generator (that is iterated to produce a value at a time) rather than returning a fully realized list, and even if you must return a list, you needn't generate such huge intermediate lists. You might think you need at least one list for sorting purposes, but combinations_with_replacement is already guaranteed to produce a predictable order based on the input ordering, and since range is ordered, the values produced will be ordered as well.

  2. Even if you solve the memory problem, the computational cost of just generating that many combinations is prohibitive, due to poor scaling; for the memory, but not CPU, optimized version of the code below, it handles an input of 11 in 0.2 seconds, 12 in ~2.6 seconds, and 13 in ~11 seconds; at that scaling rate, 20 is going to approach heat death of the universe timeframes.

Barmar's answer is one solution to both problems, but it's still doing work eagerly and storing the complete results when the complete work might not even be needed, and it involves sorting and deduplication, which aren't strictly necessary if you're sufficiently careful about how you generate the results.

This answer will fix both problems, first the memory issue, then the speed issue, without ever needing memory requirements above linear in n.

Solving the memory issue alone actually makes for simpler code, that still uses your same basic strategy, but without consuming all your RAM needlessly. The trick is to write a generator function, that avoids storing more than one results at a time (the caller can listify if they know the output is small enough and they actually need it all at once, but typically, just looping over the generator is better):

from collections import deque  # Cheap way to just print the last few elements
from itertools import combinations_with_replacement  # Imports should be at top of file,
                                                     # not repeated per call
def all_combinations_sum_to_n(n):
    for i in range(1, n + 1):  # For each possible length of combinations...
        # For each combination of that length...
        for comb in combinations_with_replacement(range(1, n + 1), i):
            if sum(comb) == n:    # If the sum matches...
                yield list(comb)  # yield the combination

# 13 is the largest input that will complete in some vaguely reasonable time, ~10 seconds on TIO
print(*deque(all_combinations_sum_to_n(13), maxlen=10), sep='\n')

Try it online!

Again, to be clear, this will not complete in any reasonable amount of time for an input of 20; there's just too much redundant work being done, and the growth pattern for combinations scales with the factorial of the input; you must be more clever algorithmically. But for less intense problems, this pattern is simpler, faster, and dramatically more memory-efficient than a solution that builds up enormous lists and concatenates them.

To solve in a reasonable period of time, using the same generator-based approach (but without itertools, which isn't practical here because you can't tell it to skip over combinations when you know they're useless), here's an adaptation of Barmar's answer that requires no sorting, produces no duplicates, and as a result, can produce the solution set in less than a 20th of a second, even for n = 20:

def all_combinations_sum_to_n(n, *, max_seen=1):
    for i in range(max_seen, n // 2 + 1):
        for subtup in all_combinations_sum_to_n(n - i, max_seen=i):
            yield (i,) + subtup
    yield (n,)


for x in all_combinations_sum_to_n(20):
    print(x)

Try it online!

That not only produces the individual tuples with internally sorted order (1 is always before 2), but produces the sequence of tuples in sorted order (so looping over sorted(all_combinations_sum_to_n(20)) is equivalent to looping over all_combinations_sum_to_n(20) directly, the latter just avoids the temporary list and a no-op sorting pass).

ShadowRanger
  • 143,180
  • 12
  • 188
  • 271
  • Thanks for your contribution. Very helpful! – plpm Feb 01 '23 at 17:04
  • @plpm: Well, I just made a better one in an edit, so take a look at that. :-) – ShadowRanger Feb 01 '23 at 17:08
  • Thanks again. I chose your answer as the best one and I added an explanation about the comparison of all three answers in the update part of my question. – plpm Feb 01 '23 at 18:43
  • @plpm: Huh, I'll have to look at the benchmark when I get home (firewalls block perfpy for some reason). Weird that I tie with precisely one of the other answers (not the same one though) for each `n` from `15` up. I expect some of this may vary by Python version too (e.g. as of 3.11, the repeated indexing in pts's answer is more heavily optimized; prior to 3.11, `list` indexing was one of the things in CPython that had the worst overhead:actual-work ratio). – ShadowRanger Feb 01 '23 at 18:50
  • I chose python 3.8 on perfpy. – plpm Feb 01 '23 at 18:58
  • Your 1. makes it sound like their `sort()` has no effect, but without it, for example for n=4 they'd get: [[4], [1, 3], [2, 2], [1, 1, 2], [1, 1, 1, 1]]. It seems you're talking about sorting the inner lists, but they're sorting the outer list. – Kelly Bundy Feb 02 '23 at 02:21
  • In 2., what scaling are you thinking of? From 11 to 12 you got factor 13.0, from 12 to 13 you got factor 4.2. if we assume factor 5 for each further step, then n=20 would take only 10 days. I don't think heat death is that imminent. – Kelly Bundy Feb 02 '23 at 02:36
  • @KellyBundy: I think (and I haven't bothered to work it out) that the scaling factor for their original code is factorial based (stuttering a bit due to how `combinations*` functions work). As for the sorting, I didn't mean it has no effect, only that there are ways (as in my final code block) to produce the sorted output without needing to fully realize the entire inputs first (if nothing else, using the original algorithm, the various length `combinations_with_replacement` could be fed to `heapq.merge` to form a combined iterator that only needs one output from each of them at a time). – ShadowRanger Feb 02 '23 at 02:58
  • I agree the scaling is probably not quite bad enough to hit heat death levels by `n = 20` (forgive some hyperbole), but it's silly to accept even minute long runtimes and superlinear memory requirements when algorithmic improvements can keep runtime in millisecond range for awhile, and memory usage linear or better. – ShadowRanger Feb 02 '23 at 03:04
  • Ok. I might try computing the costs for a better scaling estimate. Playing with your final code now. – Kelly Bundy Feb 02 '23 at 03:11
  • The `if n >= max_seen` irritates me. If that's false, i.e., `n < max_seen`, it seems the call should've never happened in the first place. With your code, I'm having difficulty removing it, but I got a [`while` version](https://tio.run/##fY5BDoMgEEX3nGKW0uLCdNGkKWchWjGS4GAAUz09rSPVrpzl/zNv3rjE3uEtpVZ3UFurXm5oDNbROAwqTIOKTmGBAi4ChnpWQWuUFX8w@E5nfIgg94JCrylDKLeewndvrM77T0k7G4IwzkOYmjiNYPDEYr36syAaPzDrLEbbFgqqBIdr5h6vyOAqodoj8i1/SQag4IyxVWw@d7pngdEbjMXMU/oA) that doesn't have that `if`. I find it easier to understand and it seems factor 1.4 faster. – Kelly Bundy Feb 02 '23 at 04:06
  • Ooohhh... I couldn't make sense of your range's upper limit `n - max_seen + 1`. Now I know why: it just doesn't make any. Should be `n//2 + 1`. And then you can also do the `yield (n,)` unconditionally, without that `if`. And it becomes about as fast as mine. – Kelly Bundy Feb 02 '23 at 20:57
  • @KellyBundy: Thanks. That did bother me as well, and I tried to cut it down (originally the loop went up to `n`, not `n - max_seen + 1`). And yeah, had the same annoyance, because I was pretty sure it should be possible to remove the `if`, but was missing the cause of the problem. Thanks for finding the logic flaw there, I've updated. That last tweak let it pull ahead of pts's solution for the `n = 20` case too. :-) – ShadowRanger Feb 02 '23 at 21:02
2

Use recursion instead of generating all combinations and then filtering them.

def all_combinations_sum_to_n(n):
    combinations_set = set()
    for i in range(1, n):
        for sublist in all_combinations_sum_to_n(n-i):
            combinations_set.add(tuple(sorted((i,) + sublist)))

    combinations_set.add((n,))

    return sorted(combinations_set)

I had a simpler solution that didn't use sorted() and put the results in a list, but it would produce duplicates that just differed in order, e.g. [1, 1, 2] and [1, 2, 1] when n == 4. I added those to get rid of duplicates.

On my MacBook M1 all_combinations_sum_to_n(20) completes in about 0.5 seconds.

Barmar
  • 741,623
  • 53
  • 500
  • 612
  • Great solution. I only need to add `if i == n: combinations_set.add(tuple((n,)))` inside the outer for loop to also include `n` itself in the final answer. Thank you for your help! – plpm Feb 01 '23 at 17:07
  • @plpm (and Barmar I guess): This solution forces all results to include a `1` (which omits some valid outputs). Your fix fixed some things, but causes other problems (adds elements which include `0`). I wrote a fix for both (that actually simplifies the code): [Try it online!](https://tio.run/##fY/NDoIwDMfvPEWPXZxG9GbCs5AhQ5uMjmzdwadHQJaoUXpYsv4/9tvwkLvn8zi2tgPjXH31fUNshDzHOqa@Fl8zsroUMA11wFBVUL6u8wQrKTAgllppVSz7zxIrUMF0olrEzgcgIIZg@GanGOT2rMbUOIoyezaQ9vQW@/XqwbQtShqcxeiD2BaRtIJd7lfqD@4SRNZZX7@4lny7J9MMPa@3iU/HFXgIxLL0qHF8Ag "Python 3 – Try It Online") – ShadowRanger Feb 01 '23 at 17:13
  • It's still slower than [my updated answer](https://stackoverflow.com/a/75313464/364696) (that generates the results precisely once, in the correct order, via a generator, no sorting required, taking a fraction of the time this one needs), but it's correct, pretty straightforward, and scales well enough. – ShadowRanger Feb 01 '23 at 17:15
  • @ShadowRanger Thank, I added your fix to my code. – Barmar Feb 01 '23 at 17:20
  • @Barmar: Cool, one last simplification I forgot to include: The addition of `combinations_set.add((n,))` and restricting the `range` to stop before `n`, rather than `n + 1`, means the `n == 1` case need not be special cased; you can delete `if n == 1: return ((1,),)`, and all it will do is the trivial work of creating a `set`, an empty `range`, `add`ing the single element to the `set` and sorting the one element `set`, which is pretty minor (in exchange, you avoid a test per recursive call). Might be a *tiny* bit slower, but simpler code without corner cases is always satisfying. – ShadowRanger Feb 01 '23 at 17:29
  • Right, because the loop is empty in that case. – Barmar Feb 01 '23 at 17:31
  • 1
    The fixes make it slower, but that's the price we pay for getting the right answer. :) – Barmar Feb 01 '23 at 17:33
  • @Barmar: C'est la vie. :-) You might recover some of the cost by reverting to `return combinations_set` and leaving it to the caller to call `sorted` on the end result (and/or wrap it in an outer function that simplifies to `return sorted(_inner_func(n))`). All the calls to `sorted` in recursive case are unnecessary after all (since the results will be individually concatenated and added to another `set` and lose ordering anyway). Bad TIO timings indicate it might shave maybe 20% off the runtime to `return combination_set` directly and wrap the call to `all_combinations_sum_to_n` in `sorted`. – ShadowRanger Feb 01 '23 at 17:41
2

Here is a fast iterative solution:

def csum(n):
  s = [[None] * (k + 1) for k in range(n + 1)]
  for k in range(1, n + 1):
    for m in range(k, 0, -1):
      s[k][m] = [[f] + terms
                 for f in range(m, (k >> 1) + 1) for terms in s[k - f][f]]
      s[k][m].append([k])
  return s[n][1]

import sys
n = 5
if len(sys.argv) > 1: n = int(sys.argv[1])
for terms in csum(n):
  print(' '.join(map(str, terms)))

Explanation:

  • Let's define terms as a non-empty, increasing (can contain the same value multiple times) list of postitive integers.

  • The solution for n is a list of all terms in increasing lexicographical order, where the sum of each terms is n.

  • s[k][m] is a list of all terms in increasing lexicographical order, where the sum of each terms in n, and the first (smallest) integer in each terms is at least m.

  • The solution is s[n][1]. Before returning this solution, the csum function populates the s array using iterative dynamic programming.

  • In the inner loop, the following recursion is used: each term in s[k][m] either has at least 2 elements (f and the rest) or it has 1 element (k). In the former case, the rest is a terms, where the sum is k - f and the smallest integer is f, thus it is from s[k - f][f].

This solution is a lot faster than @Barmar's solution if n is at least 20. For example, on my Linux laptop for n = 25, it's about 400 times faster, and for n = 28, it's about 3700 times faster. For larger values of n, it gets much faster really quickly.

This solution uses more memory than @ShadowRanger's solution, because this solution creates lots of temporary lists, and it uses all of them until the very end.

How to come up with such a fast solution?

  • Try to find a recursive formula. (Don't write code yet!)

  • Sometimes recursion works only with recursive functions with multiple variables. (In our case, s[k][m] is the recursive function, k is the obvious variable implied by the problem, and m is the extra variable we had to invent.) So try to add a few variables to the recursive formula: add the minimum number of variables to make it work.

  • Write your code so that it computes each recursive function value exactly once (not more). For that, you may add a cache (memoization) to your recursive function, or you may use dynamic programming, i.e. populating a (multidimensional) array in the correct order, so that what is needed is already populated.

pts
  • 80,836
  • 20
  • 110
  • 183
  • 1
    Nice. Similar speed to my updated solution. (I'm testing on TIO, so the run to run variation is pretty nasty, but both of them seem to do the job for `n = 20` in about 0.05 seconds). – ShadowRanger Feb 01 '23 at 17:23