25

This morning the following exception has started occuring on every API request to my Google Cloud Endpoint from my Android app:

com.google.api.server.spi.auth.GoogleIdTokenUtils verifyToken: verifyToken: Signature length not correct: got 256 but was expecting 128

The call still works perfectly from my javascript web clients. I have changed nothing on the server side code or client code.

Has anything changed with the service recently that might make this occur?

UPDATE: The first occurrence of this appears to have been at 11:17:07 UTC

UPDATE: Things that don't work include generating a new Client ID for android & updating to App Engine SDK 1.9.22

Tom
  • 664
  • 6
  • 21
  • so it's not just me... looking into it – beetstra Jun 11 '15 at 13:19
  • 1
    This just happened with me... was working fine 3 hours ago. – Sanket Berde Jun 11 '15 at 13:46
  • We are not alone The link to the official google community for downtime issues. https://groups.google.com/forum/#!topic/google-appengine/B5RmtfLS6HM – Sanket Berde Jun 11 '15 at 14:07
  • I got the same issue. This problem is causing thousands of android users not to be able to access our system. – Mael Jun 11 '15 at 15:37
  • We are having the same issue if someone gets a solution – ndgreen Jun 11 '15 at 15:56
  • Thank God i am not alone :) happened to me after lunch break :( – Muhammad Haris Altaf Jun 11 '15 at 15:59
  • 4
    We contacted Google Enterprise Support team. They told they are already working to fix this issue. – Mael Jun 11 '15 at 16:34
  • Thanks @Mael - I've been trying to work out a way of getting in touch with them! Can you keep us updated please? – Tom Jun 11 '15 at 16:35
  • 2
    They told they will provide further update at 10:00 US/Pacific. As soon as I receive any response back from them I post here. – Mael Jun 11 '15 at 16:52
  • Any updates? The only thing I can find is https://code.google.com/status/appengine which seems to show some problem with the java and python gae. – ndgreen Jun 11 '15 at 18:40
  • 3
    The issue should only affect the mixed-key-length key combinations. The current sets of keys being served only include 2048-length keys. You may still be experiencing outage due to cached key sets, but likely if you re-start your application the issue will have been resolved. – breno Jun 11 '15 at 18:59
  • 3
    Thanks seems to be good now! That as a nice 7 hour panic :) Thanks everyone for updates – ndgreen Jun 11 '15 at 19:06
  • @breno restarting changes nothing to me, still the same issue... – splinter123 Jun 11 '15 at 19:59
  • 1
    @breno I still have the same problem as well – Tom Jun 11 '15 at 20:07
  • Hi all, please let us know if you are still dealing with issues. – saiyr Jun 12 '15 at 10:39
  • It all seems to be working for me now, thanks sorting it out! @saiyr Could someone post something that I can mark as the correct answer for this? – Tom Jun 12 '15 at 10:43

2 Answers2

6

The causes

  • RSA has variable length signatures, depending on the key size.
  • Google updated the key pairs it uses for signing, and now one of the key pairs generates a different length signature from the other
  • java.security.Signature.verify(byte[] signature) throws an exception if a signature of the wrong length is passed (instead of returning false which is normally done when a signature does not match the key)

For me the solution was to wrap the verify call (try...catch), and return false instead. You could also do an early check on the public key yourself, checking if the length of the signature matches the length of the public key modulus.

If you use a library to check the signature, make sure you use the latest version.

Looking at the example code on http://android-developers.blogspot.nl/2013/01/verifying-back-end-calls-from-android.html, you would have to change this:

GoogleIdToken token = GoogleIdToken.parse(mJFactory, tokenString);

to

JsonWebSignature jws = JsonWebSignature.parser(mJFactory).setPayloadClass(Payload.class).parse(tokenString);
GoogleIdToken token = new GoogleIdToken(jws.getHeader(), (Payload) jws.getPayload(), jws.getSignatureBytes(), jws.getSignedContentBytes()) {
   public boolean verify(GoogleIdTokenVerifier verifier)
  throws GeneralSecurityException, IOException {
       try {
           return verifier.verify(this);
       } catch (java.security.SignatureException e) {
           return false;
       }
   }
};

I unfortunately don't have an exact setup to test this.

For those using Google Cloud Endpoint, like the question states, I think there was very little you could do except wait until Google fixes it. Luckily it's fixed now. (Technically, you could argue changing the keys, as is done now, is a workaround, and the library Google provides needs to be fixed. But it works, so that's a good start)

beetstra
  • 7,942
  • 5
  • 40
  • 44
  • Thanks for the info. Any idea where to start trying to fix it? – Tom Jun 11 '15 at 13:44
  • @SanketBerde Google generates the keys, and then they just sign the message... if Google generates new keys, maybe it will help, but for us that's no solution – beetstra Jun 11 '15 at 13:52
  • This is definitely not a problem from our end right ? Did we do something wrong with the config ? – Sanket Berde Jun 11 '15 at 13:55
  • @beetstra where did you wrap the verify call? I'm using Endpoints annotations to set up my API so I think that call may be hidden away somewhere in the app engine sdk – Tom Jun 11 '15 at 13:55
  • @Tom, I don't use the app engine sdk, so that makes it a bit different... Do you have some control over which public key is used to verify the signature? – beetstra Jun 11 '15 at 13:57
  • 1
    @beetstra Do you mean on the server side or android side? This is a play store deployed application so I can't quickly change anything on the client side as it will take hours to push through to users. – Tom Jun 11 '15 at 14:13
  • @Tom, on the server side – beetstra Jun 11 '15 at 14:14
  • @beetstra Yes, I can change the client id for android applications. I'm going to try generating a new client Id and pushing a new version of the server. – Tom Jun 11 '15 at 14:16
  • @Tom, the client id is not related to the public key... The public key is provided by Google. But pushing a new version of the server without changing anything might solve it simply by updating the Google Libraries. – beetstra Jun 11 '15 at 14:22
  • 1
    I Tried generating new Android client ID and Audience. Didn't work, Same error. What i've noticed is that the Endpoints calls made from the Google Endpoints explorer work. So this is a problem with the Client ids only. – Sanket Berde Jun 11 '15 at 14:30
  • @SanketBerde Same here, also my javascript clients are still working. – Tom Jun 11 '15 at 14:35
  • Do we know any big products that work on endpoints ? if they are failing too, we are experiencing a once-in-a-lifetime phenomena. – Sanket Berde Jun 11 '15 at 14:53
  • @Tom, does your server side contain code from the example I just added? Could you test my suggestion? – beetstra Jun 11 '15 at 15:04
  • @beetstra Sorry, I can't verify that works. My API methods are all automagically created by the SDK - I only declare them with the API annotation and add authentication as described here https://cloud.google.com/appengine/docs/java/endpoints/auth – Tom Jun 11 '15 at 15:16
  • The verification is supposed to happen automatically behind the scenes. thats what they say when they advertise endpoints. – Sanket Berde Jun 11 '15 at 15:19
  • I tired this solution but In my case **return verifier.verify(this);** is never called.. – Muhammad Haris Altaf Jun 11 '15 at 16:38
  • I tested the java code posted to replace the creation of the GoogleIdToken object. Unfortunately, I get the exact same exception as before!! – splinter123 Jun 11 '15 at 17:32
  • The issue should only affect the mixed-key-length key combinations. The current sets of keys being served only include 2048-length keys. You may still be experiencing outage due to cached key sets, but likely if you re-start your application the issue will have been resolved. – breno Jun 11 '15 at 19:13
0

Same problem here, as far as I can tell the public cert URL (now? I guess this wasn't the case before or the order changed) returns two keys:

https://www.googleapis.com/oauth2/v1/certs

inspecting those, the first has a 1024 bit key and the second a 2048 bit key. I believe my incoming tokens from android clients were signed by the second cert with the 2048 bit key, hence the "Signature length not correct: got 256 but was expecting 128".

Looking at the Google verifier source (GoogleTokenVerifier.java) it does appear to iterate multiple keys:

// verify signature
for (PublicKey publicKey : publicKeys.getPublicKeys()) {
  if (googleIdToken.verifySignature(publicKey)) {
    return true;
  }
}

assuming the keys got parsed correctly (that code looks reasonable but haven't actually inspected the results).

As beestra pointed out the, this code expects false to be returned in case it could not be verified but instead it's throwing an exception. Ideally it should keep iterating after a failure and use the second public key to verify, which should work.

To fix this there appears to be two options:

  1. Fork the google api client library and fix
  2. Duplicate the verification (GoogleIdTokenVerifier.verify(GoogleIdToken)) in (your) calling code

I don't know how realistic 2. is, some super functionality is used and a lot of internal state is private, would have to duplicate all of it. Busy investigating...

UPDATE: Ok, looks to be fixed in my tests using production data, though haven't deployed it to production just yet. Here's the Scala

  val jsonFactory = new JacksonFactory()
  val transport = new NetHttpTransport()
  val googleIdTokenVerifier = new GoogleIdTokenVerifier(transport, jsonFactory)

  class DuplicateVerifier(builder: GoogleIdTokenVerifier.Builder) extends IdTokenVerifier(builder)
  val topIdTokenVerifier = new DuplicateVerifier(new GoogleIdTokenVerifier.Builder(transport, jsonFactory))
  val publicKeysManager = new GooglePublicKeysManager(transport, jsonFactory)
  def duplicateGoogleVerify(token: GoogleIdToken): Boolean = {
    // check the payload
    if (!topIdTokenVerifier.verify(token)) {
      false
    } else {
      // verify signature
      import scala.collection.JavaConverters._
      publicKeysManager.getPublicKeys.asScala.map { k =>
        Try(token.verifySignature(k))
      }.foldLeft(false)((c, x) => c || x.getOrElse(false))
    }
  }

Just to be clear if it isn't obvious, using this method instead of Google's:

// if (googleIdTokenVerifier.verify(token)) {
if (duplicateGoogleVerify(token)) {

I'll try to write the Java equivalent later if anyone needs it.

Peter vR
  • 111
  • 6
  • 1
    I think you're right about the public cert URL - as of March 15th 2015 it had 2 certificates of the same size https://web.archive.org/web/20150315022911/https://www.googleapis.com/oauth2/v1/certs – Tom Jun 11 '15 at 15:35
  • please provide JAVA equivalent if it is successful.. i need it!! – Muhammad Haris Altaf Jun 11 '15 at 16:02