0

I am building a regression for my log converter application, which is written in Python. I found a very weird situation that when converting by running Python script directly, "Python myConverter.py -i <input> -o <output>" directly, the output usually have slightly different in the order, however the content is the same. Like the elements order in a json string, or the order of log entries.

However, if I compile the Python and run the .exe, this is not happening.

E.g.
(log1)
foo={bar1:a, bar2:b, bar3:c, bar4:d}
(log2)
foo={bar3:c, bar4:d, bar1:a, bar2:b}

E.g.2
(log1)
line1
line2
line3
line4
line5
(log2)
line2
line1
line3
line5
line4

My environment setting:

  • Python 2.6

  • It's a single process application, I didn't explicit use any parallel processing technic.

  • For the json serialize/deserialize, I use "import json".

It feels to me that the Python interpreter yields different output for different run. Any idea on how this could happen?



Correction:

After running the executable 5 times, I found it yields different output too!! So it seems like the comment mentioned, the dict could cause this issue.

Stan
  • 37,207
  • 50
  • 124
  • 185
  • 3
    This is probably due to dictionnary you are using in your code. Dictionarys are unordored. – FunkySayu Aug 14 '15 at 07:35
  • Could you post your code to test it...? – wenzul Aug 14 '15 at 07:48
  • I would like too, however it's proprietary. I also think about posting a simplified version just for demonstration, though the code structure is quite complicated... – Stan Aug 14 '15 at 07:52
  • @FunkySayu I read this SO (http://stackoverflow.com/questions/526125/why-is-python-ordering-my-dictionary-like-so). It says "The ordering of python dictionaries is arbitrary but deterministic". So my output should always the same, not varies every time. Thanks for pointing this out, though this is probably not the cause of issue. – Stan Aug 14 '15 at 07:58
  • 2
    I am unable to tell you why it is not the case when running an executable, and this is why i didn't answer :) But yes, order of keys may vary on different runs, and maybe you didn't make enough test to see that happening with the executable ? Try using `collections.OrderedDict` – FunkySayu Aug 14 '15 at 07:58
  • @Stan deterministic doesn't necessarily mean consistent. Also, json dictionaries aren't ordered either. – Falmarri Aug 14 '15 at 08:00
  • 1
    So the question now ist: Why the order varies? Is there a varing paramter in the hash function? – wenzul Aug 14 '15 at 08:10
  • How could I simply reproduce this dict arbitrary order in a short test script? Any idea? – Stan Aug 14 '15 at 08:11
  • http://stackoverflow.com/questions/9793922/order-of-keys-in-a-different-python-dict – wenzul Aug 14 '15 at 08:13
  • Can you be specific about the process you are using to make the exe please?I am aware of a few. When you say python26 do you mean the standard cpython implementation? – Danny Staple Aug 14 '15 at 08:15
  • @DannyStaple we use "PyInstaller 2.1" to make the exe. For the interpreter, it's CPython interpreter. – Stan Aug 14 '15 at 08:26
  • 1
    You are using `json.dumps` and `load`? If yes, use it with `sort_keys=True`. – wenzul Aug 14 '15 at 08:35

1 Answers1

1

This piece of code illustrates that the order of a dictionary depends on order of insertion:

>>> a = dict()
>>> b = dict()
>>> a[1] = None; b[9] = None
>>> a[9] = None; b[1] = None
>>> a
{1: None, 9: None}
>>> b
{9: None, 1: None}

Order of keys in a different Python dict()

Community
  • 1
  • 1
wenzul
  • 3,948
  • 2
  • 21
  • 33
  • In my case, I am sure the insertion order is always the same. Just the output order varies on each run. – Stan Aug 14 '15 at 08:22