We're running JBoss EAP 7.4.11 (at OpenJDK 11.0.20) configured with Elytron (via the docs/examples/enable-elytron-se17-domain.cli script) in domain mode.
Some applications call others via remote EJB. For that purpose there's an authentication-configuration and authentication-context configured in the elytron subsystem configuration and this authentication context is then referenced in remote-outbound-connection in the remoting subsystem configuration.
It's basically configured almost identically to https://docs.wildfly.org/27/Developer_Guide.html#EJB_invocations_from_a_remote_server_instance (which is probably the best documentation currently available).
This appears to work as expected except for one case and that is "local" call within the same host. In such cases JBoss uses the special JBOSS-LOCAL-USER authentication mechanism which causes unexpected troubles JBREM000308: Authentication failed (no mechanisms left), tried: JBOSS-LOCAL-USER: org.wildfly.security.auth.AuthenticationException: JBREM000304: Server rejected authentication
After enabling some tracing at the called instance (e.g. the one acting as the EJB server) the following can be seen:
2023-08-01 19:59:24,020 TRACE [org.jboss.remoting.remote] (default I/O-2) Received authentication request, id b4e72a33, mech JBOSS-LOCAL-USER
2023-08-01 19:59:24,020 TRACE [org.jboss.remoting.remote] (default I/O-2) Received authentication request for ID b4e72a33, mech JBOSS-LOCAL-USER
...
2023-08-01 19:59:24,020 TRACE [org.wildfly.security] (default task-4) Handling SocketAddressCallback
2023-08-01 19:59:24,020 TRACE [org.wildfly.security] (default task-4) Handling SocketAddressCallback
2023-08-01 19:59:24,020 TRACE [org.wildfly.security] (default task-4) Handling MechanismInformationCallback type='SASL' name='JBOSS-LOCAL-USER' host-name='server.domain.loc' protocol='remote'
2023-08-01 19:59:24,020 TRACE [org.wildfly.security] (default task-4) Handling MechanismInformationCallback type='SASL' name='JBOSS-LOCAL-USER' host-name='server.domain.loc' protocol='remote'
2023-08-01 19:59:24,020 TRACE [org.wildfly.security] (default task-4) Creating SaslServer [org.wildfly.security.sasl.localuser.LocalUserServer@58583e7f] for mechanism [JBOSS-LOCAL-USER] and protocol [remote]
2023-08-01 19:59:24,020 TRACE [org.wildfly.security] (default task-4) Created SaslServer [org.wildfly.security.sasl.util.SecurityIdentitySaslServerFactory$1@66bb693a->org.wildfly.security.sasl.util.AuthenticationTimeoutSaslServerFactory$DelegatingTimeoutSaslServer@14bb462f->org.wildfly.security.sasl.util.AuthenticationCompleteCallbackSaslServerFactory$1@c74c779->org.wildfly.security.sasl.localuser.LocalUserServer@58583e7f] for mechanism [JBOSS-LOCAL-USER]
2023-08-01 19:59:24,021 TRACE [org.jboss.remoting.remote] (default task-4) Sending authentication challenge for ID b4e72a33
...
2023-08-01 19:59:24,021 TRACE [org.wildfly.security] (default task-4) Handling NameCallback: authenticationName = $local
2023-08-01 19:59:24,021 TRACE [org.wildfly.security] (default task-4) Principal assigning: [$local], pre-realm rewritten: [$local], realm name: [local], post-realm rewritten: [$local], realm rewritten: [$local]
2023-08-01 19:59:24,021 TRACE [org.wildfly.security] (default task-4) PropertiesRealm: identity [$local] does not exist
2023-08-01 19:59:24,021 TRACE [org.wildfly.security] (default task-4) Authorization failed - realm identity does not exists
2023-08-01 19:59:24,021 TRACE [org.wildfly.security] (default task-4) Handling AuthorizeCallback: authenticationID = $local authorizationID = $local authorized = false
2023-08-01 19:59:24,021 TRACE [org.jboss.remoting.remote] (default task-4) Authentication failed at response evaluation: javax.security.sasl.SaslException: ELY05014: Authentication mechanism authorization failed: "$local" running as "$local"
at org.wildfly.security.elytron-private@1.15.16.Final-redhat-00001//org.wildfly.security.sasl.localuser.LocalUserServer.evaluateMessage(LocalUserServer.java:262)
at org.wildfly.security.elytron-private@1.15.16.Final-redhat-00001//org.wildfly.security.sasl.util.AbstractSaslParticipant.evaluateMessage(AbstractSaslParticipant.java:219)
at org.wildfly.security.elytron-private@1.15.16.Final-redhat-00001//org.wildfly.security.sasl.util.AbstractSaslServer.evaluateResponse(AbstractSaslServer.java:82)
at org.wildfly.security.elytron-private@1.15.16.Final-redhat-00001//org.wildfly.security.sasl.util.AuthenticationCompleteCallbackSaslServerFactory$1.evaluateResponse(AuthenticationCompleteCallbackSaslServerFactory.java:58)
at org.wildfly.security.elytron-private@1.15.16.Final-redhat-00001//org.wildfly.security.sasl.util.AuthenticationTimeoutSaslServerFactory$DelegatingTimeoutSaslServer.evaluateResponse(AuthenticationTimeoutSaslServerFactory.java:110)
at org.wildfly.security.elytron-private@1.15.16.Final-redhat-00001//org.wildfly.security.sasl.util.SecurityIdentitySaslServerFactory$1.evaluateResponse(SecurityIdentitySaslServerFactory.java:59)
at org.jboss.remoting@5.0.27.Final-redhat-00001//org.jboss.remoting3.ConnectionImpl.lambda$receiveAuthResponse$3(ConnectionImpl.java:274)
at org.jboss.remoting@5.0.27.Final-redhat-00001//org.jboss.remoting3.EndpointImpl$TrackingExecutor.lambda$execute$0(EndpointImpl.java:993)
at org.jboss.threads@2.4.0.Final-redhat-00001//org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
at org.jboss.threads@2.4.0.Final-redhat-00001//org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1990)
at org.jboss.threads@2.4.0.Final-redhat-00001//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
at org.jboss.threads@2.4.0.Final-redhat-00001//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377)
at org.jboss.xnio@3.8.9.Final-redhat-00001//org.xnio.XnioWorker$WorkerThreadFactory$1$1.run(XnioWorker.java:1282)
at java.base/java.lang.Thread.run(Thread.java:829)
Which appears as if it tries to authenticate using the $local user which doesn't exist. Moreover this occurs randomly, e.g. in some cases we can see that the expected user (the one configured via authentication-configuration and context) is used, but mostly not.
We could have something misconfigured (though the related configuration is very simple and based on the default one with just a few things related to the remote EJB added), but we suspect this is more a JBoss issue. Could someone confirm please?
Note: As a dirty workaround we added the user $local to the application-users.properties file (we added it manually because the add-user.sh script doesn't allow the $ character). With this the authentication appears to succeed even when the $local user is used, but we don't think this is the right thing to do.
Related parts of the configuration:
<subsystem xmlns="urn:wildfly:elytron:13.0" final-providers="combined-providers" disallowed-providers="OracleUcrypto">
<authentication-client>
<authentication-configuration name="MH-ejb-auth-config" authentication-name="mhejb">
<credential-reference clear-text="password"/>
</authentication-configuration>
<authentication-configuration name="TF-ejb-auth-config" authentication-name="tfejb">
<credential-reference clear-text="password"/>
</authentication-configuration>
<authentication-context name="MH-ejb-auth-context">
<match-rule match-port="8080" authentication-configuration="MH-ejb-auth-config"/>
</authentication-context>
<authentication-context name="TF-ejb-auth-context">
<match-rule match-port="8100" authentication-configuration="TF-ejb-auth-config"/>
</authentication-context>
</authentication-client>
...
<subsystem xmlns="urn:jboss:domain:remoting:4.0">
<http-connector name="http-remoting-connector" connector-ref="default" sasl-authentication-factory="application-sasl-authentication"/>
<outbound-connections>
<remote-outbound-connection name="MH-remote-ejb-connection-1" outbound-socket-binding-ref="MH-remote-ejb-1" authentication-context="MH-ejb-auth-context">
<properties>
<property name="SASL_POLICY_NOANONYMOUS" value="false"/>
<property name="SSL_ENABLED" value="false"/>
<property name="READ_TIMEOUT" value="120000"/>
</properties>
</remote-outbound-connection>
<remote-outbound-connection name="TF-remote-ejb-connection-1" outbound-socket-binding-ref="TF-remote-ejb-1" authentication-context="TF-ejb-auth-context">
<properties>
<property name="SASL_POLICY_NOANONYMOUS" value="false"/>
<property name="SSL_ENABLED" value="false"/>
<property name="READ_TIMEOUT" value="120000"/>
</properties>
</remote-outbound-connection>
</outbound-connections>
</subsystem>
<socket-binding-group name="GS-ha-sockets" default-interface="public">
<socket-binding name="ajp" port="${jboss.ajp.port:8009}"/>
<socket-binding name="http" port="${jboss.http.port:8080}"/>
<socket-binding name="https" port="${jboss.https.port:8443}"/>
<socket-binding name="jgroups-mping" interface="private" multicast-address="${jboss.default.multicast.address:231.0.40.4}" multicast-port="45740"/>
<socket-binding name="jgroups-tcp" interface="private" port="7600"/>
<socket-binding name="jgroups-tcp-fd" interface="private" port="57600"/>
<socket-binding name="jgroups-udp" interface="private" port="55200" multicast-address="${jboss.default.multicast.address:231.0.40.4}" multicast-port="45728"/>
<socket-binding name="jgroups-udp-fd" interface="private" port="54200"/>
<socket-binding name="modcluster" multicast-address="${jboss.modcluster.multicast.address:225.0.41.105}" multicast-port="23404"/>
<socket-binding name="txn-recovery-environment" port="4712"/>
<socket-binding name="txn-status-manager" port="4713"/>
<outbound-socket-binding name="MH-remote-ejb-1">
<remote-destination host="server.domain.loc" port="8080"/>
</outbound-socket-binding>
<outbound-socket-binding name="TF-remote-ejb-1">
<remote-destination host="server.domain.loc" port="8100"/>
</outbound-socket-binding>
</socket-binding-group>
