15

Azure IoT Hub Supports AMQP, MQTT, HTTP protocols. In order to customize these protocols we have Azure IoT protocol gateway. I can find good samples on MQTT protocol customization. I need some sample codes for TCP based protocol customization using Azure IoT Protocol Gateway.

EDIT (in order to get an answer): what the OP was asking, is an example using the Azure Protocol Gateway to support a proprietary TCP-based protocol. Currently the IoT hub only supports AMQP, MQTT and HTTP. While those protocols actually rely on TCP, the hub doesn't support direct TCP connection without the extra layer of AMQP, MQTT or HTTP. As explained here, we need a basic example of a custom TCP based protocol.

Imagine a basic device that can only send some proprietary payload through TCP on a given IP address/port: we need an example of a gateway customization that allows this device to send data to the hub.

The current code of the protocol gateway is poorly designed, as it heavily relies on MQTT.

Adding some bounty too.

ken2k
  • 48,145
  • 10
  • 116
  • 176
arun thatham
  • 500
  • 1
  • 4
  • 13
  • 1
    TCP is the *underlying* communication protocol for all of these *message* protocols. What are you trying to do? What protocol do you want to implement? Why not use one of the available protocols? – Panagiotis Kanavos Mar 16 '17 at 16:29
  • Besides, you already found the Azure IoT Gateway and samples. The MQTT protocol itself was added as a customization using the gateway framework – Panagiotis Kanavos Mar 16 '17 at 16:33
  • 1
    My device supports only TCP communication. It sends a sync message over TCP to a static IP and port. I want this device to be virtualized in Azure IoT Hub. Something which is described here. https://github.com/Azure/azure-iot-protocol-gateway/issues/44 – arun thatham Mar 16 '17 at 16:35
  • That doesn't explain anything. You can use HTTP over TCP - in fact, you are doing so right now. The same with MQTT, AMQP. You just have to right the appropriate client code, or use a library. All of these work over TCP. There are mqtt clients for Arduino for example, like [this client for ESP8266](https://github.com/marvinroger/async-mqtt-client) – Panagiotis Kanavos Mar 16 '17 at 16:39
  • I agree with you that appropriate client code will work. But the problem here is that the device which i'm holding currently doesn't allow me to update its firmware. It is fully abstracted and it has some configuration done in the past which can support only TCP communication. I cannot add any library or upload any sketch to that. Hence i wanted to do something out of the device – arun thatham Mar 16 '17 at 16:49
  • I don't think you understand that TCP communication is the underlying protocol of ALL other messaging protocols. When you make an HTTP request, that ends up as a TCP connection and a bunch of TCP packets. It's like insisting that you need a car with wheels. Besides, how are you going to do *anything* if your device can't be programmed? – Panagiotis Kanavos Mar 16 '17 at 17:03
  • What device are you talking about anyway? If the firmware can't be changed, or a program uploaded, how are you even going to set the cloud provider's IP? It sounds like you are trying to modify a device hardwired to work with only the manufacturer's service – Panagiotis Kanavos Mar 16 '17 at 17:04
  • Anyway, it's impossible to answer your question without *specifics* - device, model, protocol you want to use. Azure IoT already supports TCP, just as cars support wheels – Panagiotis Kanavos Mar 16 '17 at 17:05
  • 1
    @Panagiotis Kanavos Maybe the OP wasn't clear enough, but the question is a real, unanswered question. The OP sure understands how TCP works. To make it more clear, see my edit. In a few words: we need an example for a proprietary TCP-based protocol. IoT hub doesn't allow any TCP-based protocol to communicate with it: only MQTT, AMQP and HTTP. We need a low-level example of a basic TCP-based protocol. Take [STOMP](https://en.wikipedia.org/wiki/Streaming_Text_Oriented_Messaging_Protocol) as an example if you want. – ken2k Feb 06 '18 at 14:29
  • I’m trying to make apple to Apple comparison...AWS has a TCP load balancer...I need something similar to this in IOT hub in Azure – arun thatham Feb 06 '18 at 14:31
  • @arunthatham I've modified your question and added a bounty so we can get an answer. Everybody on the github project needs this example but it's still not provided after all this time.Not sure about the load balancer though. – ken2k Feb 06 '18 at 14:33
  • The answer I heard in past is that use service fabric...and build ur cluster to ingest the data through TCP...no support on IOT hub – arun thatham Feb 06 '18 at 14:35
  • @ken2k it is the *OP's* responsibility to clean up the question and explain what he/she wants. The OP still doesn't improve the answer although the comments show that no TCP example is needed. Instead, it's a question about *load balancing*. – Panagiotis Kanavos Feb 06 '18 at 14:40
  • @arunthatham Well that's one solution for sure, but you'll lose the benefits of the IoT hub (scalability, device registry, security...). I've managed in the past to port some proprietary TCP-based protocol using the Azure Protocol Gateway. But it was a real pain in the ass, as the current gateway code is poorly structured and heavily relies on MQTT. It was a 30 man-days effort to get it working right. Now I need something similar, and I think everyone should benefit from a basic TCP example. – ken2k Feb 06 '18 at 14:40
  • @arunthatham update your question explaining what you want. *Don't* add explanations in the comments. You didn't ask about load balancing. And anyway, *ingres* scalability isn't load balancing. AWS doesn't use load balancers for this, it uses Kinesis. For Azure it's Event Hub and lately Event Grid – Panagiotis Kanavos Feb 06 '18 at 14:41
  • 1
    @PanagiotisKanavos Forget the load balancer. The question is pretty clear as it is now, we need an example of a TCP-based proprietary protocol. Not some MQTT tweak the Azure Protocol Gateway currently provide. – ken2k Feb 06 '18 at 14:42
  • @ken2k a basic TCP example of what? Ingres is handled by event hub, event grid. No load balancers are involved here *or* in Amazon Web Services – Panagiotis Kanavos Feb 06 '18 at 14:42
  • @ken2k no, it's even more confuse. You said `TCP example` the OP said `load balancer`. It was unclear a year ago, it's unclear now. In fact, if you are correct, it's out-of-scope too. Implementing a custom TCP protocol ins't `basic` or `simple` and can't be covered in a single answer – Panagiotis Kanavos Feb 06 '18 at 14:43
  • @PanagiotisKanavos It _really_ was clear a year ago if you are familiar with IoT hub and the protocol gateway MS provides. Now the "load balancer" stuff the OP needs today is not clear, but the original question really is. – ken2k Feb 06 '18 at 14:45
  • @PanagiotisKanavos The [github link](https://github.com/Azure/azure-iot-protocol-gateway/issues/44) the OP provided explained the issue clearly to me. I could write a new question, but everything I would write is already written here. The github issue is the most active one for this repo, and this question got 3 upvotes: we need an example of a TCP-based protocol that is not tied to MQTT. The MS documentation suggest you can use the Azure Protocol Gateway for "custom protocols" which is not true at all currently. – ken2k Feb 06 '18 at 14:51
  • @ken2k Good. Then everyone else on SO is dense and couldn't understand this for an entire year. Edit the question. Asking for an entire implementation is out-of-scope anyway. It can't fit in a single answer. The problem is interesting, hence no downvotes. *Answering* it in a Q&A site is somewhat difficult – Panagiotis Kanavos Feb 06 '18 at 14:52
  • @PanagiotisKanavos If it was asked yesterday without success doesn't mean it won't today. That's what the "draw attention bounties" are for. We don't need a full implementation but either a link to an existing work, or some guidelines from MS folks that are monitoring SO tags. – ken2k Feb 06 '18 at 14:56
  • @ken2k You said you did the same thing in the past. Why don't you post the answer? Whatever that is, prevents everyone else from answering too – Panagiotis Kanavos Feb 06 '18 at 14:59
  • @PanagiotisKanavos What we did previously was a 30 man-days effort. It probably wasn't a good implementation because we had no clue about how to do it. As MS advertises, the protocol gateway should easily allow custom protocol implementation. We need guidelines about how to do it properly. Even an official answer such as "you can't, it really is a 30 man-days effort" would be fine. – ken2k Feb 06 '18 at 15:02
  • 1
    @ken2k this is SO, not Github or MSDN or UserVoice. If *you* can't post your solution as an answer, who can and what do you expect it to look like? BTW if that someone comes this way, he should be able to read all the required info in the *question*, not the comments. Mentioning STOMP or whatever protocol you used will help – Panagiotis Kanavos Feb 06 '18 at 15:22

2 Answers2

3

The default Protocol Gateway sample are indeed somewhat confusing because of all the MQTT code. The protocol gateway works by 'simulating' a IoTHub connection for each custom protocol device you connect to the gateway.

To do this translation from the TCP device to an IoTHub device you first need to have a connection to the IoTHub on behalf of the device. This is the gateway part. Below is the core essentials for this IoTHubConnection.

namespace GatewayTest
{
    using System;
    using System.Text;
    using System.Threading;
    using System.Threading.Tasks;
    using DotNetty.Buffers;
    using Microsoft.Azure.Devices.ProtocolGateway.Identity;
    using Microsoft.Azure.Devices.ProtocolGateway.IotHubClient;
    using Microsoft.Azure.Devices.ProtocolGateway.Messaging;

    public class IoTHubConnection : IMessagingChannel<IMessage>
    {
        private readonly string iotHubHostName;
        private readonly Func<IDeviceIdentity, Task<IMessagingServiceClient>> deviceClientFactory;
        private readonly Func<string, Task> onMessage;
        private IMessagingServiceClient deviceClient;
        private IDeviceIdentity deviceIdentity;

        public IoTHubConnection(
            string iotHubHostName,
            Func<IDeviceIdentity, Task<IMessagingServiceClient>> deviceClientFactory,
            Func<string, Task> onMessage)
        {
            this.iotHubHostName = iotHubHostName;
            this.deviceClientFactory = deviceClientFactory;
            this.onMessage = onMessage;
        }

        public event EventHandler CapabilitiesChanged;

        public async Task OpenAsync(string deviceId, string deviceKey)
        {
            this.deviceIdentity = this.GetDeviceIdentity(deviceId, deviceKey);
            if (this.deviceIdentity != UnauthenticatedDeviceIdentity.Instance)
            {
                this.deviceClient = await this.deviceClientFactory(this.deviceIdentity);
                this.deviceClient.BindMessagingChannel(this);
            }
        }

        public async Task CloseAsync()
        {
            await this.deviceClient.DisposeAsync(null);
            this.deviceClient = null;
        }

        public void Handle(IMessage message)
        {
            var messageBody = message.Payload.ToString(Encoding.UTF8);

            this.onMessage(messageBody);

            this.deviceClient.CompleteAsync(message.Id);
        }

        public Task SendMessage(string message)
        {
            var buffer = Unpooled.WrappedBuffer(Encoding.UTF8.GetBytes(message));
            var deviceMessage = this.deviceClient.CreateMessage($"devices/{this.deviceIdentity.Id}/messages/events", buffer);
            return this.deviceClient.SendAsync(deviceMessage);
        }

        protected virtual void OnCapabilitiesChanged(EventArgs e)
        {
            this.CapabilitiesChanged?.Invoke(this, e);
        }

        private IDeviceIdentity GetDeviceIdentity(string userName, string deviceKey)
        {
            IotHubDeviceIdentity ideviceIdentity;
            if (!IotHubDeviceIdentity.TryParse($"{this.iotHubHostName}/{userName}", out ideviceIdentity))
            {
                return UnauthenticatedDeviceIdentity.Instance;
            }

            ideviceIdentity.WithDeviceKey(deviceKey);
            return ideviceIdentity;
        }
    }
}

The deviceClientFactory callback method should be implemented as shown below and in this line in the ProtocolGateway repo in Github.

deviceClientFactory = IotHubClient.PreparePoolFactory(
    "IotHubConnectionString",
    400,
    TimeSpan.FromMinutes(3),
    iotHubClientSettings,
    PooledByteBufferAllocator.Default,
    new ConfigurableMessageAddressConverter("TopicNameConversion"));

When a Tcp Device connects to the protocol, you should create an instance of this IoTHubConnection and send messages from the Device to the IoTHubConnection and vica versa. The code below shows a very simple version of how this should be done.

private const int BufferSize = 1024;
private byte[] buffer = new byte[BufferSize];
private IoTHubConnection ioTHubConnection;
private NetworkStream stream;

private async Task Start()
{
    listener = new TcpListener(IPAddress.Any, port);
    listener.Start();

    var client = await listener.AcceptTcpClientAsync();
    ioTHubConnection = new IoTHubConnection("IoTHubName", deviceClientFactory, OnIoTHubMessage);
    stream = client.GetStream();

    // Read DeviceId and DeviceKey from some sort of StartConnection-message send by the TcpClient.
    await ioTHubConnection.OpenAsync("DeviceId", "DeviceKey");

    stream.BeginRead(buffer, 0, BufferSize, ReadTcpStreamCallback, null);
}

private void ReadTcpStreamCallback(IAsyncResult ar)
{
    var bytesRead = stream.EndRead(ar);

    if (bytesRead > 0)
    {
        var message = System.Text.Encoding.ASCII.GetString(result);

        ioTHubConnection.SendMessage(message);

        // Read again.
        stream.BeginRead(buffer, 0, BufferSize, ReadTcpStreamCallback, null);
    }
}

private async Task OnIoTHubMessage(string message)
{
    // Potentially do some translation on the IoTHub message
    // and send it to the Device

    var byteData = Encoding.UTF8.GetBytes(message);
    stream.BeginWrite(byteData, 0, byteData.Length, SendTcpCallback, null);
}

private void SendTcpCallback(IAsyncResult ar)
{
    stream.EndWrite(ar);
}
Mikael Falkvidd
  • 413
  • 6
  • 13
  • 1
    Was anyone for sure about to get this working? I'm attempting to build out a custom solution of my own and am getting a bit overwhelmed by all of the MQTT code in the example. I'm not completely clear (given the above example) where to plug this in relative to all of the other code/instructions/hosts/etc. – Brandon Avant Apr 09 '19 at 18:22
  • @BrandonAvant Have you had any luck with this implementation? – Mohammed Sohail Ebrahim Oct 24 '19 at 08:46
  • 1
    @MohammedSohailEbrahim , unfortunately no. We ended up having to create a web socket proxy that converts IoT Hub Traffic to Websockets and vice versa. – Brandon Avant Oct 26 '19 at 00:59
0

I know I am late to this conversation. However I have interesting add on or might be a solution for some.

Azure IoT Gateway is now known as Azure IoT Edge, this is clearly mentioned in the following Azure github repo

enter image description here

https://github.com/Azure/iot-edge-modbus.git

On the other hand, Azure IoT Edge supports TCP for some protocols which can be found in the following links

  1. https://learn.microsoft.com/en-us/azure/iot-edge/deploy-modbus-gateway
  2. https://learn.microsoft.com/en-us/azure/iot-edge/iot-edge-as-gateway
Jawad Sabir
  • 167
  • 2
  • 12