50

I tested sys.getsize('') and sys.getsize(' ') in three environments, and in two of them sys.getsize('') gives me 51 bytes (one byte more than the second) instead of 49 bytes:

Screenshots:

Win8 + Spyder + CPython 3.6:

sys.getsizeof('') == 49 and sys.getsizeof(' ') == 50

Win8 + Spyder + IPython 3.6:

sys.getsizeof('') == 51 and sys.getsizeof(' ') == 50

Win10 (VPN remote) + PyCharm + CPython 3.7:

sys.getsizeof('') == 51 and sys.getsizeof(' ') == 50

First edit

I did a second test in Python.exe instead of Spyder and PyCharm (These two are still showing 51), and everything seems to be good. Apparently I don't have the expertise to solve this problem so I'll leave it to you guys :)

Win10 + Python 3.7 console versus PyCharm using same interpreter:

enter image description here

Win8 + IPython 3.6 + Spyder using same interpreter:

enter image description here

Nicholas Humphrey
  • 1,220
  • 1
  • 16
  • 33
  • 15
    My burning question is "why does it matter?". But anyway, Spyder will also be throwing that into a shared namespace – roganjosh Dec 22 '18 at 23:06
  • 6
    @roganjosh Actually I think it doesn't matter because my job as a data analyst doesn't ask me to dig deep into the object model, but I'm scratching my head to understand the why behind this. I wish I have other OS e.g. Linux to test this. BTW does this have something to do with the "shared namespace" you said? – Nicholas Humphrey Dec 22 '18 at 23:09
  • 2
    My job is also data scientist/data analyst. This behaviour is inconsequential, but I don't want to invalidate your question (curiosity is fine). Spyder has a complex namespace, you must have observed how things are available in the console from your main scripts... – roganjosh Dec 22 '18 at 23:12
  • @roganjosh thanks! btw the same happens in PyCharm. I'll test it in main scripts. – Nicholas Humphrey Dec 22 '18 at 23:14
  • 1
    @NicholasHumphrey in the terminal, on OSX and ubuntu with python 3.6: `sys.getsizeof('')` yields 49, with python 2.7, it yields 37. – BiBi Dec 22 '18 at 23:16
  • @BiBi thanks! I'm not even sure if I should restart my computer (the Python environments are clean start though, but both my PC and remote VPN PC have been staying awake for weeks) and test again. Pretty weird as it appears in both IPython 3.6 and CPython 3.7 – Nicholas Humphrey Dec 22 '18 at 23:21
  • 5
    @AndreyTyukin No I just want to see if someone else has encountered this weird thing before, and more importantly, if an empty string does have 1 more byte than a string with one char, it means that my understanding of the string object could be totally wrong. If you think this is normal, then sorry, because I'm not professional software developer and this is indeed weird to me. For now I'm settled with this issue as a second test with Python.exe console shows 49. – Nicholas Humphrey Dec 22 '18 at 23:43
  • 1
    It could be due to a compression used in the implementation, some compression algorithms may have such a weird behaviour: shorter input lead to larger output, sometimes. – Alexandre Fenyo Dec 22 '18 at 23:48
  • @AlexandreFenyo Thanks, could be the reason. This is way over my expertise and I now know 51 is abnormal so my understanding of the empty string object was OK. I agree with previous comments that it's probably nothing and I'm fine with it. – Nicholas Humphrey Dec 22 '18 at 23:56
  • @AndreyTyukin Thanks for the help! Good to know the details of the implementation. I'll maybe come back after I have more understanding of the field (gc and compiler theory are both way over my need for the job) – Nicholas Humphrey Dec 23 '18 at 00:08
  • 2
    The most likely candidate seems to be that strings cache a version encoded with UTF-8 when first required. – Davis Herring Dec 23 '18 at 00:20
  • 1
    Seems more likely to be the wstr representation than the UTF-8 representation - the UTF-8 would be one byte (just the null terminator), and also, the UTF-8 would most likely not be stored separately, since an empty string is probably in "compact ASCII" representation. – user2357112 Dec 23 '18 at 04:07

2 Answers2

33

This sounds like something is accessing the deprecated Py_UNICODE API.

As of CPython 3.7, the way the CPython Unicode representation works out, an empty string is normally stored in "compact ASCII" representation, and the base data and padding for a compact ASCII string on a 64-bit build works out to 48 bytes, plus one byte of string data (just the null terminator). You can see the relevant header file here.

For now (this is scheduled for removal in 3.12), there is also a deprecated Py_UNICODE API that stores an auxiliary wchar_t representation of the string. On a platform with 2-byte wchar_t, the wchar_t representation of an empty string is 2 bytes (just the null terminator again). The Py_UNICODE API caches this representation on the string object on first access, and str.__sizeof__ accounts for this extra data when it exists, resulting in a 51-byte total.

(If you need a wchar_t representation of a string, the non-deprecated way to get one is to use PyUnicode_AsWideChar or PyUnicode_AsWideCharString. These functions are not scheduled for removal, and do not attach any data to the string object.)

user2357112
  • 260,549
  • 28
  • 431
  • 505
  • 2
    the getsizeof() does refer to `__sizeof__` internally. This is the correct answer – Abhishek Dujari Dec 23 '18 at 04:26
  • Thanks! Although I don't get the full pic this seems to explain the source of 51-byte. Just curious, why does Spyder with CPython give me 49-byte but shows 51-byte for IPython on the same PC? I figured from your answer that it has something to do with the size of wchar, and in turn the IDEs get 2 bytes because it's OS-specified, but I think it should be the same for all interpreters? Anyway I may misunderstand your answer... – Nicholas Humphrey Dec 23 '18 at 04:56
  • 2
    @NicholasHumphrey: Something is retrieving the wchar representation in your IPython tests. (Also, your IPython tests are also using CPython; CPython is the interpreter implementation IPython runs on.) – user2357112 Dec 23 '18 at 04:59
  • 9
    This is largely unrelated to the question, but seeing a reference to a "[Python] 4.0" is giving me anxiety... – Mike Caron Dec 23 '18 at 05:25
  • 1
    @MikeCaron and others: fear not. References to '4.0' means 'some future release after 2.7 support ends (2020 Jan). Some post-deprecation removals have been delayed to make migration easier for those who prefer smaller steps. Things deprecated in 3.3, about 6 years ago, might otherwise have gone away in 3.5. We no longer allow new references to a fictional '4.0'. I just suggested in https://bugs.python.org/issue35616 that we 'backport' this policy to older notices, precisely to avoid 'anxieties' that no one needs. – Terry Jan Reedy Dec 30 '18 at 04:59
  • I don't understand the last part. If it's 48 bytes plus 2 bytes null terminator, what is that extra 1 byte to reach 51? – Kelly Bundy Mar 30 '22 at 00:50
  • 1
    @KellyBundy: 48 bytes, plus 1 byte ASCII null terminator for the ASCII representation, plus 2 byte wchar_t null terminator for the wchar_t representation. – user2357112 Mar 30 '22 at 00:56
  • Ah, ok. Didn't realize the wchar_t representation was *in addition* to the compact one, thought it was used instead. Btw this is indeed what I see on Google Colab, where the empty string has 53 bytes. Its `compact` bit is set, there's a zero-byte after the 48 bytes, and the `wchar_t*` is non-zero and points to four zero-bytes. – Kelly Bundy Mar 30 '22 at 01:08
4

https://docs.python.org/3.5/library/sys.html#sys.getsizeof

sys is system specific so it can easily differ. This is often overlooked by everyone. All system specific stuff in python has been dumped in the sys package for years. For e.g sys.getwindowsversion() is not portable by definition but it's there. It like the bottomless pit of rejects in the perfect world of cross platform coding. What you see is one of the interesting nuggets of Python.

from getsizeof docs:

Only the memory consumption directly attributed to the object is accounted for, not the memory consumption of objects it refers to. getsizeof() calls the object’s __sizeof__ method and adds an additional garbage collector overhead if the object is managed by the garbage collector.

When Garbage collection is in use the OS will add those extra bits. If you read Python and GC Q & A When are objects garbage collected in python? the folks have gone into excruciating detail expounding the GC and how it will affect the memory/refcount and bits blah blah.

I hope that explains where this coming from. If you don't use system level attributes but more pythonic attributes then you will get consistent sizes.

TrebledJ
  • 8,713
  • 7
  • 26
  • 48
Abhishek Dujari
  • 2,343
  • 33
  • 43
  • 3
    It's not GC data. String objects are never tracked by the GC; they don't have that data. Also, the same objects would have GC data on all the configurations the questioner tested. – user2357112 Dec 23 '18 at 04:21
  • 1
    Then I stand to be corrected. it may not be GC. However the difference in representation still applies and is system specific. It could be OS+runtime – Abhishek Dujari Dec 23 '18 at 04:24