How to check the connection status of a Kinesis client?

Question

I run a fleet of web services behind a load balancer. The lb periodically requests a status endpoint to verify that the particular server is operational. This service endpoint checks connectivity to all external services (sql, redis, kinesis etc.)

To check the kinesis connectivity I originally used kinesis.listStreams() but as we are increasing the number of servers we are running into the 5 req/sec limit of the listStreams command.

What is the best/cheapest way to verify that kinesis is available?

We are using the Java client, but I imagine it would be the same for all clients.

Answer 1

AWS support answered in this thread https://forums.aws.amazon.com/thread.jspa?messageID=743149&#743149

Kinesis does not directly support this.

There are two options:

1) Perform the rate limited request and treat the rate limit exception as a success as it being returned by the server indicates that the connection is working.

The rate limiting in this case is actually done on the server side. So you will either get a valid ListStreams response (HTTP code 200), or you'll get a LimitExceededException (HTTP code 400). Both responses imply that you have connectivity to the Kinesis endpoint, since the response is coming from the service. If connectivity is broken, you would see timeouts or other errors with response code 5xx (things like ServiceUnavailableException).

2) Don't use the kinesis library, but use regular network techniques to test connectivity to the kinesis rest endpoint

Unfortunately (and for obvious reasons), all our APIs are rate limited and we don't expose any APIs suitable for this usage pattern. However, it seems like you just want to verify the connectivity between your service and the Kinesis endpoint. You don't really need to use the AWS SDK for this, you can probably use standard Java libraries to perform something like a "ping" to our endpoint. Preferred Java way to ping an HTTP URL for availability has some options