3

To improve troubleshooting on our services, we implemented a mechanism of "Request IDs", in which a unique ID is assigned to each request and then when a service calls another while servicing a particular request, it sends the request ID as an HTTP header, so that if an error occurs in the called service, we can easily find out what was the original request. To implement this feature, we've created two filters:

  • a ContainerRequestFilter which reads the ID from the request headers and sets it in the ContainerRequestContext
  • a ClientRequestFilter that adds the Request ID to all requests issued by a Client

For this to work, we are registering the ClientRequestFilter on every request, like so:

public class MyServiceClient {

    private WebTarget target;

    @Inject
    public MyServiceClient(Client client, @Context ContainerRequestContext ctx) {
        // This is called on every request, with a new context
        this.target = client
                .target("/target/endpoint")
                .register(new RequestIdFilter(ctx)); // custom ClientRequestFilter
    }

    public void doSomething() {
        this.target.path("resource")
                .request()
                .get();
    }
}

This works fine most of the time, but we've encountered sporadic errors which seem to be caused by thread-safety issues. What happens is that if two threads call MyServiceClient::doSomething at the same time, the WebTarget instance will eventually call ClientConfig::initRuntime which at some point collects all Binders configured and calls Binder::bind on them. Now, AbstractBinder::bind is clearly not thread-safe, so sometimes requests fail with:

java.lang.IllegalArgumentException: Recursive configuration call detected.
        at org.glassfish.hk2.utilities.binding.AbstractBinder.bind(AbstractBinder.java:179)
        at org.glassfish.jersey.model.internal.CommonConfig.configureBinders(CommonConfig.java:676)
        at org.glassfish.jersey.model.internal.CommonConfig.configureMetaProviders(CommonConfig.java:641)
        at org.glassfish.jersey.client.ClientConfig$State.configureMetaProviders(ClientConfig.java:372)
        at org.glassfish.jersey.client.ClientConfig$State.initRuntime(ClientConfig.java:405)
        at org.glassfish.jersey.client.ClientConfig$State.access$000(ClientConfig.java:90)
        at org.glassfish.jersey.client.ClientConfig$State$3.get(ClientConfig.java:122)
        at org.glassfish.jersey.client.ClientConfig$State$3.get(ClientConfig.java:119)
        at org.glassfish.jersey.internal.util.collection.Values$LazyValueImpl.get(Values.java:340)
        at org.glassfish.jersey.client.ClientConfig.getRuntime(ClientConfig.java:733)
        at org.glassfish.jersey.client.ClientRequest.getConfiguration(ClientRequest.java:286)
        at org.glassfish.jersey.client.JerseyInvocation.validateHttpMethodAndEntity(JerseyInvocation.java:135)
        at org.glassfish.jersey.client.JerseyInvocation.<init>(JerseyInvocation.java:105)
        at org.glassfish.jersey.client.JerseyInvocation.<init>(JerseyInvocation.java:101)
        at org.glassfish.jersey.client.JerseyInvocation.<init>(JerseyInvocation.java:92)
        at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:420)
        at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:316)

According to Is java Jersey 2.1 client thread safe?, this problem could have been caused by a non-thread-safe Binder registered by someone else (perhaps JacksonBinder, from Dropwizard). What do you think?

If the approach is wrong, do you know of a better way of implementing this "Request ID" functionality?

Thanks!

Edit 1: Here is the code for the RequestIdFilter

public class RequestIdFilter implements ClientRequestFilter {

    private ContainerRequestContext context;

    public RequestIdFilter(ContainerRequestContext context) {
            this.context = context;
        }

    @Override
    public void filter(ClientRequestContext clientCtx) throws IOException {
            String requestId = (String) context.getProperty(X_REQUEST_ID); // This is set by a ContainerRequestFilter
            clientCtx.getHeaders().putSingle(X_REQUEST_ID, requestId);
        }
}

Edit 2: The issue has been (temporarily) fixed by passing the Request ID as a query parameter when creating the WebTarget in the MyServiceClient constructor. This seems to be thread-safe and avoids registering a new filter on every request.

Community
  • 1
  • 1
Luís Pureza
  • 841
  • 1
  • 7
  • 13
  • You should post your `RequestIdFilter` code also. – zloster Apr 21 '17 at 07:16
  • I experience the same. Everything works fine with dropwizard 1.1.1, but does not with 1.1.2. I have an integration test, that fails constantly with this exception, but I am not able to provide a minimal working example. – scho Oct 16 '17 at 11:40
  • @scho did you make any progress with solving? We're running into the same issue with our clients as well. – jmif Oct 31 '17 at 23:07
  • Hi, OP here. I don't know what you're doing with your filters, but I'd like to mention that the temporary fix I suggested at the end of my post (registering the filters only once on startup and passing any parameters as query parameters) worked well. – Luís Pureza Nov 01 '17 at 09:47
  • I ended up registering the `HttpAuthenticationFeature` feature on the client and setting the properties on `Invocation.Builder` when doing the actual request. – scho Nov 02 '17 at 10:02

0 Answers0