Mqtt Last Will & Testament (LWT) message - issues with the timing of the LWT message

Question

I have a Windows Service running on multiple boxes, and publishing messages to the HiveMq Mqtt Broker. Good so far.

However, I noticed that when I manually STOP one of those Win Services - my browser Mqtt client IMMEDIATELY gets the last Last Will message (which of course I'm subscribing to).

Shouldn't I be receiving the LWT message according to the configured number of seconds in the keepAlive period, and not immediately ? Does it have to do with the retained flag perhaps ?

I feel that I have something wrong in my LWT configuration.

Here is a snippet of my C# code (in my Win Svc) -

public ManagedMqttClientOptions WsSecureClientOptions(PublisherSubscriber pubSubType) {
  // Build LWT message, then Client Options
  MqttModel lastWill = new MqttModel();
  lastWill.message = "BMAZZO box is OFFLINE";
  lastWill.datestamp = DateTime.Now;
  lastWill.status = ConnectionStatus.Offline;
  LastWillMsgJson = JsonConvert.SerializeObject(lastWill, Formatting.Indented);

 
  clientOptionsBldr = new MqttClientOptionsBuilder()
        .WithClientId(".netWinSvc-BMAZZO-Pub")
        .WithProtocolVersion(MqttProtocolVersion.V500)
        .WithWebSocketServer("<broker-url>:8884/mqtt")
        .WithCredentials("myUser", "myPswd")                                            
        .WithCleanSession(true)
        .WithWillQualityOfServiceLevel(2)
        .WithWillTopic('myApp\myLAN\Box\BMAZZO\Status')   // LAST WILL TOPIC
        .WithWillRetain(true)                           // RETAIN
        .WithWillPayload(LastWillMsgJson)               // WILL PAYLOAD
        .WithKeepAlivePeriod(TimeSpan.FromSeconds(30))         // KEEP ALIVE, 30 SECS
        .WithTls(
            new MqttClientOptionsBuilderTlsParameters()
            {
                UseTls = true,
                SslProtocol = System.Security.Authentication.SslProtocols.Tls12,
                Certificates = new List<X509Certificate2>() { x509Cert }
            });
            
    return managedClientOptions;
 }
          
 public async Task Publish(string messageIn, string topic, IManagedMqttClient pubClient = null, ConnectionStatus status = ConnectionStatus.Unknown)
    {            
        MqttModel message = new MqttModel();
        message.message = messageIn;
        message.datestamp = DateTime.Now;
        message.status = status;
        var payload = JsonConvert.SerializeObject(message, Formatting.Indented);

        var send = new MqttApplicationMessageBuilder()
            .WithTopic("myApp\myLAN\Box\BMAZZO\Status")
            .WithPayload(payload)
            .WithMessageExpiryInterval(86400)   // seconds per day
            .WithQualityOfServiceLevel(2)       // QOS
            .WithRetainFlag(true)               // retain = true
            .Build();


                applog.Debug($"Mqtt Publish() to broker - message = {messageIn} / {topic} ");
                await this.managedMqttClientPublisher.EnqueueAsync(send);
                
       await this.managedMqttClientPublisher.EnqueueAsync(send);

    }

Example: I stopped the Win Svc at 2023-04-25 17:30:00, and immediately received a message in my browser that the Service was OFFLINE minutes earlier at 2023-04-25T17:26:43

 {message: 'Offline BMAZZO 10.2.23.62', status: 1, source: 'BMAZZO', datestamp: '2023-04-25T17:26:43.7074826-04:00'}

Another example: I just stopped the Service at 17:58, and the LWT message appears to show one of the previous LWT messages (retained in the broker?).

  {message: 'Offline BMAZZO 10.2.23.62', status: 1, source: 'BMAZZO', datestamp: '2023-04-25T17:47:32.4730311-04:00'}

Perhaps the WithWillRetain flag should be set to false ?

Any thoughts or suggestions are appreciated.

Bob

Answer 1

Firstly a note - in the comments you reference the v3 spec ("3.1.2.5 Will Flag specs at docs.oasis-open.org/mqtt/mqtt/v3.1.1/mqtt-v3.1.1.html"). You are using MQTT v5 (.WithProtocolVersion(MqttProtocolVersion.V500)) and this is one area where the specs vary considerably. The below is based upon my understanding of the spec (so may contain errors!).

What you are seeing appears in line with the MQTT v5 spec which says:

The Server delays publishing the Client’s Will Message until the Will Delay Interval has passed or the Session ends, whichever happens first

So the question here is "when does the session end"; I believe the relevant section of the spec is:

The session can continue across a sequence of Network Connections. It lasts as long as the latest Network Connection plus the Session Expiry Interval.

Your code does not use the SessionExpiryInterval() option meaning that the session will expire immediately upon disconnection

If the Session Expiry Interval is absent the value 0 is used.

So when your client disconnects the session ends and the Will is published.

With reference to keep alives; these are a mechanism to enable the client to detect a half-open connection. Often (especially when a service is cleanly shutdown) the other side of the connection will be notified when the connection is closed and the keep alive is irrelevant.

Note that another factor may be coming into play (as mentioned by @Aaron Franz):

when I manually STOP one of those Win Services

Manually stopping the services may be resulting in a clean disconnection (i.e. a DISCONNECT packet is sent). Disconnecting cleanly ("Normal disconnection") deletes the will but you may also opt to send it (and this may change the session expiry interval).

Additional comments (added later in response to comments)

I'm not sure that I need both SessionExpiryInterval AND WithWillDelayInterval since it will delay publishing..

Correct; WithWillDelayInterval is really only useful when it's less than SessionExpiryInterval (and really only with WithCleanSession(false)). The broker and client can retain session information (and messages published) while the connection is down meaning that intermittent connection loss does not result in any loss of QOS1+ messages. As such you might not be bothered by a 1 minute outage but do want to know if the connection is down more than 10 minutes (times will depend upon your use-case - e.g. SessionExpiryInterval = 1 month, WithWillDelayInterval = 10 minutes).

Answer 2

Glad to hear implementation stories for the HiveMQ Broker!

One detail I wanted to clarify - when this service is terminated, is there any current functionality implemented to send a DISCONNECT packet when the service has been terminated? The current functionality seems to indicate that a graceful disconnect is being triggered at the time of connection closure. Depending on the version of the HiveMQ broker being used, it may be worthwhile to implement a custom listener to log onDisconnect or onServerInitiatedDisconnect reason codes for more specific diagnostics. More information regarding implementation can be found here : https://docs.hivemq.com/hivemq/4.11/extensions/registries.html#on-server-disconnect.

Also, I encourage anyone with questions regarding HiveMQ to reach out on our community forum at https://community.hivemq.com/!

Best,

Aaron from the HiveMQ Team