6

I've been looking for some time for a way to send JSON over a tcp socket and to de-serialize it into an object. The problem that I'm having is that when my message is too big, TCP chunks it. By doing this, my clients are receiving only a piece of the JSON and the parser sends that part and the following as malformed JSONs.

I'm trying to look for a tool that will allow me to feed it with this partial views of my JSON. The first time, my buffer contains only a partial JSON, I send it to the parser and it returns something that indicates that the JSON is not finished. Next time, when I append the new information to the buffer, the parser continues from the last point of where it was and finds the end of the JSON message (Or waits in case is needed and the JSON is coming in more than two chunks) and return something with the information of that JSON content, and removes it from the buffer, where the next information coming form the stream will be appended.

So far, I found that some other ideas but for other languages (Java in this one: Is there a streaming API for JSON?)

Flashito
  • 332
  • 1
  • 4
  • 16
  • An easy way is to have a header of the size of the whole packet... read first the header and then the body... – Elvis Dukaj May 23 '18 at 10:27
  • This would break the idea of having a normal JSON being sent and then it would be no worth using JSON at all anymore. I would be building a protocol on top of TCP – Flashito May 23 '18 at 11:21
  • Tool recommendations are off-topic, sorry. But you're on the correct path; a streaming JSON parser is much preferred over handrolled hacks. That said, have you considered HTTP? Still a protocol on top of TCP, but not handrolled. – MSalters May 23 '18 at 14:04
  • Thanks for letting me know. I was not aware that this would be such a off-topic case. I'll keep it in mind for the next time. We can close this then, since I have a solution "hand-made" already in place. – Flashito May 24 '18 at 14:18

2 Answers2

1

You have some alternatives... One is to read data until some special sequence of characters (for example https uses \r\n\r\n). If you use boost C++ library you can look at boost::asio::async_read_until or boost::asio::read_until. Or you can implement you read routine that stops with a sentinel value.

The second alternative is to have a header file of your packet... in the header, you can specify for example the length of the message. So you must first read the header and then the whole body.

Elvis Dukaj
  • 7,142
  • 12
  • 43
  • 85
  • I thought about this too, forcing all JSONs to be one line and separated by a new line character. However, it won't be optimal, since in order to do that, I would have to go through the JSON file more than one time in order to be able to extract it and to send it to the parser – Flashito May 23 '18 at 11:18
1

I once had a similar problem.

To fix it, I determined the maximum size of one chunk and split my message manually to a lower size before sending it. Also attached an index and max index to the "manual chunk" to make an assembly on the receiver side.

Not sure if this is the best approach but it did the job

MauriceRandomNumber
  • 362
  • 4
  • 16
  • I would have to go over the JSON file several times in order to be able to parse merge all chunks manually and parse it in the end. Also, after having them merged, then I would have to go once more over the JSON to actually parse it – Flashito May 23 '18 at 11:14