3

My system consists on a C++ main part, which acts as core for a complex process. This core is in charge of executing python scripts which perform several tasks and then displaying those on a Graphical User Interface based on HTML technology. This HTML document is loaded using WTL, but it is not that relevant since my problem arises before the information generated on python reach that point.

import json
#my_python_dict is the Python dictionary information I want to send to C++ -> Javascript.
json_str = json.JSONEncoder().encode(my_python_dict)

Then I send this information to C++ using appropriate API calls: API_Call is the method which executes and returns the string generated in python. 

PyObject *json_string;
json_string = API_Call();
char * c_str;
c_str = PyString_AsString(json_string);

If I execute those lines I get weird characters and broken data, whereas if I substitute the original code in python for this:

import json
#my_python_dict is the Python dictionary information I want to send to C++ -> Javascript.
#json_str = json.JSONEncoder().encode(my_python_dict).
json_str = "This is a dummy test... of random lenght..."

It works just perfectly, so I dare to say that the problem is probably on json.JSONEncoder() or maybe a coding problem within json.JSONEncoder and PyString_AsString().

Any ideas?

====================== P.D. ====================================

I have been trying out a few ideas in order to try to solve this issue.

  • The most shocking one is this: If I copy the encoded string just from the debug console, paste it into the dummy string and then return that string, everything works fine! But I do need the output from the json.JSONEncoder obviously.
Pablo Stark
  • 682
  • 10
  • 34
  • Have you tried logging `json_str` (or `my_python_dict`) in the Python code? – Some programmer dude Sep 04 '14 at 07:48
  • Yes I did. I even tried to print the json_string from the python console and the substitute the dummy test string for the output in the console and that works perfectly. – Pablo Stark Sep 04 '14 at 07:50
  • https://docs.python.org/2/library/json.html#json.JSONEncoder has some arguments about escaping non-ascii chars. Have you tried altering these options? Also have you tried to use json.dumps(my_python_dict) ? – gosom Sep 04 '14 at 07:52
  • Yes, I have tried those options, and by default the json.JSONEncoder().ensure_ascii element is set to True on the __init__ method, which I already checked using the debugger. – Pablo Stark Sep 04 '14 at 07:55
  • 1
    try this https://github.com/simplejson/simplejson just to see if there is a difference. also check that http://stackoverflow.com/questions/1446347/how-to-find-out-if-python-is-compiled-with-ucs-2-or-ucs-4 – gosom Sep 04 '14 at 07:59
  • 2
    When do you inspect the contents of `c_str`? When you use the result of `json.JSONEncoder().encode()`, is there any chance the Python object has been deallocated or otherwise modified before you use `c_str`? – Warren Weckesser Sep 04 '14 at 08:10
  • 2
    What are the "weird characters and broken data"? Give us a small example dict, what the JSON looks like when you log it from Python, and what the bytes are that you get in your C code. – abarnert Sep 04 '14 at 08:11
  • Actually @WarrenWeckesser that was the problem, I was using Py_DECREF to the variable without copying the c_str into another buffer. Thank you very much. If you answer the question I will accept your answer. – Pablo Stark Sep 04 '14 at 08:28
  • Thank you @abarnert too, I was about to provide you with such data, just before testing for deallocation. – Pablo Stark Sep 04 '14 at 08:29

1 Answers1

1

Converted comment to an answer...

When do you inspect the contents of c_str? Be sure that the Python string object created by json.JSONEncoder().encode() has not been deallocated or otherwise modified before you use the contents of c_str.

Warren Weckesser
  • 110,654
  • 19
  • 194
  • 214