0

Key points:

  • I need to send roughly ~100 float numbers every 1-30 seconds from one machine to another.
  • The first machine is catching those values through sensors connected to it.
  • The second machine is listening for them, passing them to an http server (nginx), a telegram bot and another program sending emails with alerts.

How would you do this and why? Please be accurate. It's the first time I work with sockets and with python, but I'm confident I can do this. Just give me crucial details, lighten me up! Some small portion (a few rows) of the core would be appreciated if you think it's a delicate part, but the main goal of my question is to see the big picture.

La faguette
  • 541
  • 4
  • 11

2 Answers2

2

Main thing here is to decide on a connection design and to choose protocol. I.e. will you have a persistent connection to your server or connect each time when new data is ready to it.

Then will you use HTTP POST or Web Sockets or ordinary sockets. Will you rely exclusively on nginx or your data catcher will be another serving service.

This would be a most secure way, if other people will be connecting to nginx to view sites etc.

Write or use another server to run on another port. For example, another nginx process just for that. Then use SSL (i.e. HTTPS) with basic authentication to prevent anyone else from abusing the connection.

Then on client side, make a packet every x seconds of all data (pickle.dumps() or json or something), then connect to your port with your credentials and pass the packet.

Python script may wait for it there.

Or you write a socket server from scratch in Python (not extra hard) to wait for your packets. The caveat here is that you have to implement your protocol and security. But you gain some other benefits. Much more easier to maintain persistent connection if you desire or need to. I don't think it is necessary though and it can become bulky to code break recovery. No, just wait on some port for a connection. Client must clearly identify itself (else you instantly drop the connection), it must prove that it talks your protocol and then send the data. Use SSL sockets to do it so that you don't have to implement encryption yourself to preserve authentication data. You may even rely only upon in advance built keys for security and then pass only data.

Do not worry about the speed. Sockets are handled by OS and if you are on Unix-like system you may connect as many times you want in as little time interval you need. Nothing short of DoS attack won't inpact it much.

If on Windows, better use some finished server because Windows sometimes do not release a socket on time so you will be forced to wait or do some hackery to avoid this unfortunate behaviour (non blocking sockets and reuse addr and then some flo control will be needed).

As far as your data is small you don't have to worry much about the server protocol. I would use HTTPS myself, but I would write myown light-weight server in Python or modify and run one of examples from internet. That's me though.

Dalen
  • 4,128
  • 1
  • 17
  • 35
0

The simplest thing that could possibly work would be to take your N floats, convert them to a binary message using struct.pack(), and then send them via a UDP socket to the target machine (if it's on a single LAN you could even use UDP multicast, then multiple receivers could get the data if needed). You can safely send a maximum of 60 to 170 double-precision floats in a single UDP datagram (depending on your network).

This requires no application protocol, is easily debugged at the network level using Wireshark, is efficient, and makes it trivial to implement other publishers or subscribers in any language.

John Zwinck
  • 239,568
  • 38
  • 324
  • 436
  • 1
    Oh, yes, and lose whole data packet or implement whole layer to ensure that it arrives. How practical. Note that this is not video nor audio so that you can just discard something. – Dalen May 22 '17 at 15:07
  • Notot mention one would hve a minimal protocol and some code even to encode/decode an arbitrary number of floats using `struct` - as it requires a struct description with a fixed lenght for the data already. – jsbueno May 22 '17 at 18:16
  • @jsbueno: No, it does not. A UDP datagram knows its own length, so e.g. a 32 byte datagram contains 4 x 8-byte floats. – John Zwinck May 23 '17 at 02:52
  • @Dalen: I guess you are being sarcastic, which is not appreciated. OP did not say that losing messages is unacceptable--on the contrary OP said this is for a sensor network, and in real-world sensor networks losing a few messages is often perfectly acceptable. – John Zwinck May 23 '17 at 02:54
  • I am not talking of UDP datagrams, I am talking about Python struct.pack - and no, it does not know its own lenght - one will have to either encode its lenght at one edge of the data, therefore having to defien mini-protocol, or fetch the length from the UDP headers to decode - but still will have to manipulate the format string to both encode and decode the struct. – jsbueno May 23 '17 at 07:28
  • @jsbueno not big problem, you do: fmt = "%if" % len(packet); struct.pack(" – Dalen May 23 '17 at 08:40
  • @JohnZwinck Yes, a bit. Not meant to be offending. Sorry. As OP didn't specify that data loss is acceptable your A should mention it and give some more options. You are writing as everything is just puff. Fingerclick and done. For instance multicast doesn't have to be supported by LAN router or a switch even. Your solution is limiting as well. Harder to expand if needed etc. Notice though that you won't get -1 from me. Your A is nevertheless useful. – Dalen May 23 '17 at 08:49
  • @JohnZwinck Ow, I forgot. What if order of arrival is switched? So additional timestamp will have to be included too. I don't think that switching measurement results in time would be desirable. I wouldn't rely on 1 second time interval as a method against late packet arrival. – Dalen May 23 '17 at 09:00
  • @Dalen the only sensor that need to be checked every second is a ultrasonic proximity sensor. I guess instead of sending the data every second I could make an exception for that one and keep it controlled locally. If and only if there's an alert, then the alert will be sent. This way you can assume I need the data to be sent every 10-30 seconds. – La faguette May 26 '17 at 07:01
  • @Lafaguette That's very good idea. (To send alerts only.) If you need to send only few bytes every second and you need speedy receival, then it would be better to use TCP sockets with perhaps struct packing them. HTTP would require big headers in both directions comparing to sent data. I assumed you would be sending info from whole sensor network each sec. Then already established protocol makes more sense. – Dalen May 27 '17 at 09:35
  • Note: I made a mistake when explaining to @jsbueno easy use of struct. Correct way is : fmt = "%if" % len(packet); struct.pack(("<%is"+fmt) % (len(fmt)+2), "<"+fmt+":", *packet) then on receiving end you do fmt, packet = packet.split(":", 1); packet = struct.unpack(fmt, packet)[0] - I forgot to denote length of string when packing format in and to use asterisk to unpack the (packet) list/tuple into struct.pack()'s arguments. Sorry. – Dalen May 27 '17 at 09:39
  • " I made a mistake when explaining to @jsbueno easy use of struct. "... which just means you are assembling a mini protocol that is prone to errors, and therefore is better avoided (altogether with raw sockets, in favor of a higher level protocol for the whole data exchange) – jsbueno May 27 '17 at 19:28
  • @jsbueno Sorry, but nothing that dramatic happened. Python would raise an exception saying that format string is incorrect. A person that would try my suggestion will go and read the __doc__ string of struct.pack() and set it right or send me to hell. And everything works fine. Using this "mini protocol" over TCP for non-complex data is perfectly acceptable. However, I agree that it is best to use "higher level protocols" when possible. – Dalen May 30 '17 at 00:12