Sending data using socket from Python(client) to C++(server) arrays

Question

I have been attempting to send an array from Python to C++ using a socket however have kept running into issues.

Python side there are issues sending an array directly such as pickle not being compatible with C++, as such the only semi-reliable method I have found is sending it as a string:

import socket
import sys
import random

# Create a TCP/IP socket
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

server_address = ('localhost', 5555)
print >>sys.stderr, 'connecting to %s port %s' % server_address
sock.connect(server_address)

# Message to be sent to C++
# message = [random.randint(1, 10),random.randint(1, 10),random.randint(1, 10)]
i = 0

while i < 5:
    a_converted = (random.randint(1,255), random.randint(1,255), random.randint(1,255))
    #a_converted = 'words'

    print a_converted
    # Sending message to C++
    sock.sendall(str(a_converted))
    i += 1

sock.close()

The issue with sending it as a string is that I actually require it as an double style array on the other side. The C++ code I have is currently the following:

#include "stdafx.h"
#include <iostream>
#include <winsock2.h>
#include <WS2tcpip.h> //for SOCKET communication
#include <sstream>
#include <stdlib.h>

//Linking Ws2_32.lib, Mswsock.lib, Advapi32.lib
#pragma comment (lib, "Ws2_32.lib")
#pragma comment (lib, "Mswsock.lib")
#pragma comment (lib, "AdvApi32.lib")
#pragma warning(disable:4996)//to disable warning message

using namespace std;

int main()
{
    WSADATA WSAData;

    SOCKET server, client;

    SOCKADDR_IN serverAddr, clientAddr;

    WSAStartup(MAKEWORD(2, 0), &WSAData);
    server = socket(AF_INET, SOCK_STREAM, 0);

    serverAddr.sin_addr.s_addr = INADDR_ANY;
    serverAddr.sin_family = AF_INET;
    serverAddr.sin_port = htons(5555);
    ::bind(server, (SOCKADDR *)&serverAddr, sizeof(serverAddr));
    listen(server, 0);

    cout << "Listening for incoming connections..." << endl;

    string a_converted[1000];

    int clientAddrSize = sizeof(clientAddr);
    if ((client = accept(server, (SOCKADDR *)&clientAddr, &clientAddrSize)) != INVALID_SOCKET)
    {
        cout << "Client connected!" << endl;
        // Loop 
        int i = 0;

        while (i<5) {

            recv(client, (char*)a_converted, sizeof(a_converted), 0);

            char char_array[sizeof(a_converted)];

            strcpy(char_array, (char*)a_converted);
            memset(a_converted, 0, sizeof(a_converted));
            cout << "Client says: " << char_array << endl; 
            cout << endl;

            i = i++;

        }
        closesocket(client);
        WSACleanup();
        cout << "Client disconnected." << endl;
    }
    cout << "Press Enter to continue" << endl;
    getchar();
}

The information is received and correct but I have been unable to correctly convert the data. I have tried to use atof and similar functions to convert on the C++ side but the presence of commas and brackets from the Python side seem to result in it erroring and giving zeros and I've had little luck trying to remove them from the string.

I can't help but think there must be a better way of doing this but I am really new to coding so would not be surprised if I am overlooking something. I would appreciate either an explanation of how to send an array directly that C++ can read from Python or a way to convert the string it sends in C++.

Why don't you use some standard format for serialized data, like JSON? Pickle of course won't work in your case because it's Python specific. — ElmoVanKielmo, Feb 19 '18 at 14:26

MauriceRandomNumber · Answer 1 · 2018-02-19T15:20:51.360

This sounds like a serialization problem. As you want to connect two different languages, I suggest a standardized, well known format like JSON or XML. There are lots of libraries for converting JSON or XML into objects and vice versa.

Ultimate, dead-end solution is, to pack the data into a binary file and send this over the socket. Try external libraries for JSON or XML first

Edit:

JSON describes (in a very simple way), how objects can be saved as text (serialization). It is as simple as

{
  "firstName": "John",
  "lastName": "Smith",
  "isAlive": true,
  "age": 27,
  "address": {
    "streetAddress": "21 2nd Street",
    "city": "New York",
    "state": "NY",
    "postalCode": "10021-3100"
  },
  "phoneNumbers": [
    {
      "type": "home",
      "number": "212 555-1234"
    },
    {
      "type": "office",
      "number": "646 555-4567"
    },
    {
      "type": "mobile",
      "number": "123 456-7890"
    }
  ],
  "children": [],
  "spouse": null
}

(taken from https://en.wikipedia.org/wiki/JSON)

You can imagine, it is a very straight forward process to read from the text to an object again (deserialization). There are libraries there which will do the job for you. This means for your project:

find a python JSON library which serializes your objects into JSON format
find a c++ JSON library which deserializes your data to an object

I have seen this in use, as every data type is treated as a string to send it, and on the deserialization side, you need to decice for each case which data type it really is (converting the string to double or int or float or bool etc..)

Are there any examples or guides for how JSON is implemented? I noticed on their website there are multiple versions. — SarahB, Feb 19 '18 at 15:11
https://stackoverflow.com/questions/17549906/c-json-serialization is a good source of other links. JSON is just how an object is described as text. Follow this link https://en.wikipedia.org/wiki/JSON . Updated my answer — MauriceRandomNumber, Feb 19 '18 at 15:17

score 1 · Accepted Answer · answered Feb 19 '18 at 18:28

The most straight-forward way to accomplish this is to employ python's struct module to encode your array into a binary format that will be convenient to receive in C++.

For example, to send an array of 32-bit integers, you might do something like this:

import struct

def encode_int_array(int_list):
    buf = struct.pack("!I" + "I" * len(int_list), len(int_list), *int_list)
    return buf

Explanation: The ! character specifies the byte-ordering to be used in the encoding (here, big-endian / "network" order), the I character is used here for the array length, then again once for each integer to be encoded. The actual array length and each integer is then packed.

So, if you called this function with the list [1, 2, 3], the format string given to pack will be "!IIII" and the the remaining arguments will be 3, 1, 2, 3 (the first '3' being the array length to encode). The end result is a bytes string containing the encoded 32-bit (4-byte) integers:

|ArrayLen|Integer0|Integer1|Integer2|....

Use the above along with sendall to transmit the resulting buffer:

sock.sendall(encode_int_array(
    [random.randint(1,255), random.randint(1,255), random.randint(1,255)]))

On the C++ side, first read 4 bytes (to get the array length), convert the array length to native byte ordering, then read an additional 4 * array-length bytes to get all the integers; then convert each of those to native byte order. You should be careful never to assume that recv will receive all of the data you want. The SOCK_STREAM semantics do not guarantee that. So you need to ensure you receive exactly the number you expected.

The C++ side might look something like this:

#include <cstdint>  // uint32_t et al definitions

// Function to receive exactly "len" bytes.
// Returns number of bytes received, or -1 on EOF or error
int recv_all(int sock, char *buf, unsigned int len)
{
    unsigned int n = 0;
    int status;
    while (n < len) {
        status = recv(sock, buf + n, len - n);
        if (status == 0) {
            // Unexpected End of File
            return -1; // Or whatever
        } else if (status < 0) {
            // Error
            return -1; // Need to look at errno to find out what happened
        } else {
            n += status;
        }
     }
     return (int)n;
}

...
int status;
// Receive array length from client
uint32_t array_len;
status = recv_all(client, reinterpret_cast<char *>(&array_len), sizeof array_len);
if (status < 0) {
    // handle error
}

array_len = ntohl(array_len); // Convert length to native byte order

// Receive array contents from client
uint32_t int_array[array_len];
status = recv_all(client, reinterpret_cast<char *>(&int_array[0]), sizeof int_array);
if (status < 0) {
    // handle error
}
for (unsigned int n = 0; n < array_len; ++n)
    int_array[n] = ntohl(int_array[n]); // Convert to native byte order

(If you only wanted to send single byte integers, substitute 'B' for 'I' in the pack calls above, and the C++ below will also need to be adjusted accordingly -- with uint8_t in place of uint32_t, say.)

Apart from some issues with getting the array_len to be accepted in array_int (mentions that it cannot be treated as a constant) it transmits as I wished to. Very clear explanation of the process. — SarahB, Feb 20 '18 at 10:15

Sundar Rajan · Answer 3 · 2018-02-19T14:52:37.000

0

You need to convert String in python to Bytes while

sock.sendall

Refer Python: convert string to byte array

Based on your python syntax, it should be python 2.

Python 3 can easily find and insist to convert into Bytes while sendall.

In python 2, you can use bytearray like

sock.sendall (bytearray(str(a_converted)))

In python 3, you can call encode which defaults to UTF-8 while converting to bytes.

sock.sendall (str(a_converted).encode())

edited Feb 19 '18 at 14:52

answered Feb 19 '18 at 14:43

Sundar Rajan

556
4
25

Do you know how to get C++ to receive that correctly, as the recv function will only accept char*? – SarahB Feb 19 '18 at 14:53
I can find this works like this. Client says: (97, 12, 127)(187, 67, 80)(235, 177, 44)(91, 215, 50)(10, 80, 31)╠╠╠╠╠╠╠╠╠╠╠ – Sundar Rajan Feb 19 '18 at 14:54
as mentioned by @MauriceRandomNumber , You should rely on JSON or XML to parse through standard format if you do not bother to write your own parsing logic. The Symbols after 31 are attempt to print unallocated bytes after receiving. You can fix it by looking for a delimiter. – Sundar Rajan Feb 19 '18 at 14:57

Sending data using socket from Python(client) to C++(server) arrays

3 Answers3