After a while on our WildFly 18 server, in production, we encountered this error:
[org.xnio.listener] (default I/O-1) XNIO001007: A channel event listener threw an exception:
java.lang.OutOfMemoryError: Direct buffer memory
at java.base/java.nio.Bits.reserveMemory(Bits.java:175)
at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:118)
at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:317)
at org.jboss.xnio@3.7.3.Final//org.xnio.BufferAllocator$2.allocate(BufferAllocator.java:57)
at org.jboss.xnio@3.7.3.Final//org.xnio.BufferAllocator$2.allocate(BufferAllocator.java:55)
at org.jboss.xnio@3.7.3.Final//org.xnio.ByteBufferSlicePool.allocateSlices(ByteBufferSlicePool.java:162)
at org.jboss.xnio@3.7.3.Final//org.xnio.ByteBufferSlicePool.allocate(ByteBufferSlicePool.java:149)
at io.undertow.core@2.0.27.Final//io.undertow.server.XnioByteBufferPool.allocate(XnioByteBufferPool.java:53)
at io.undertow.core@2.0.27.Final//io.undertow.server.protocol.framed.AbstractFramedChannel.allocateReferenceCountedBuffer(AbstractFramedChannel.java:549)
at io.undertow.core@2.0.27.Final//io.undertow.server.protocol.framed.AbstractFramedChannel.receive(AbstractFramedChannel.java:370)
at io.undertow.core@2.0.27.Final//io.undertow.websockets.core.AbstractReceiveListener.handleEvent(AbstractReceiveListener.java:38)
at io.undertow.core@2.0.27.Final//io.undertow.websockets.core.AbstractReceiveListener.handleEvent(AbstractReceiveListener.java:33)
at org.jboss.xnio@3.7.3.Final//org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
at io.undertow.core@2.0.27.Final//io.undertow.server.protocol.framed.AbstractFramedChannel$FrameReadListener.handleEvent(AbstractFramedChannel.java:950)
at io.undertow.core@2.0.27.Final//io.undertow.server.protocol.framed.AbstractFramedChannel$FrameReadListener.handleEvent(AbstractFramedChannel.java:931)
at org.jboss.xnio@3.7.3.Final//org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
at org.jboss.xnio@3.7.3.Final//org.xnio.conduits.ReadReadyHandler$ChannelListenerHandler.readReady(ReadReadyHandler.java:66)
at org.jboss.xnio.nio@3.7.3.Final//org.xnio.nio.NioSocketConduit.handleReady(NioSocketConduit.java:89)
at org.jboss.xnio.nio@3.7.3.Final//org.xnio.nio.WorkerThread.run(WorkerThread.java:591)
We checked a JVM dump through jxray, and it seems websockets are the culprit:
Fact is that our websocket is kind of straightforward:
@ApplicationScoped
@ServerEndpoint(value = "/ws/messenger/{accountId}")
public class MessengerSocket implements Serializable
{
private static final long serialVersionUID = -3173234888004281582L;
@Inject
private Logger log;
@Inject
private MessengerHandler handler;
@OnOpen
public void onOpen(@PathParam("accountId") String accountId, Session session, EndpointConfig config)
{
log.debug("Opening for {}", accountId);
handler.subscribeSocket(session, UUID.fromString(accountId));
}
@OnClose
public void onClose(@PathParam("accountId") String accountId, Session session, CloseReason closeReason)
{
log.debug("Closing {}", accountId);
handler.unsubscribeSocket(session, UUID.fromString(accountId));
}
}
It's coupled with a simple handler, managing a map of users sessions:
@ApplicationScoped
public class MessengerHandler
{
@Inject
private Logger log;
// key: Account id
private Map<UUID, AccountMessengerSessions> sessions;
public void init()
{
sessions = new ConcurrentHashMap<>();
}
public void subscribeSocket(Session session, UUID accountId)
{
// build and store the account messenger session if new
AccountMessengerSessions messenger = sessions.getOrDefault(accountId, new AccountMessengerSessions(accountId));
messenger.getWsSessions().add(session);
sessions.putIfAbsent(accountId, messenger);
log.debug("{} has {} messenger socket session(s) (one added)", messenger.getAccountId(), messenger.getWsSessions().size());
}
/**
* Unsubscribes the provided WebSocket from the Messenger.
*/
public void unsubscribeSocket(Session session, UUID accountId)
{
if (!sessions.containsKey(accountId))
{
log.warn("Ignore unsubscription from {} socket, as {} is unknwon from messenger", session.getId(), accountId);
return;
}
AccountMessengerSessions messenger = sessions.get(accountId);
messenger.getWsSessions().remove(session);
log.debug("{} has {} messenger socket session(s) (one removed)", messenger.getAccountId(), messenger.getWsSessions().size());
if (!messenger.getWsSessions().isEmpty())
{
return;
}
// no more socket sessions, fully remove
sessions.remove(messenger.getAccountId());
}
}
Client side, we have a bit of javascript called when a page is loaded, again, nothing fancy:
var accountId = // some string found in DOM
var websocketUrl = "wss://" + window.location.host + "/ws/messenger/" + accountId;
var websocket = new WebSocket(websocketUrl);
websocket.onmessage = function (event) {
var data = JSON.parse(event.data);
// nothing fancy here...
};
Our users don't use a lot the feature offered by the websocket (an instant messenger), so what is really happening in production is basically websockets opening and closing at each page, with very few messages sent through.
Where could we get it wrong and create this buffer leak? Did we forget something critical?
