1

While I was deploying my google app engine project (Java) using gradle appengineDeploy my internet connection was interrupted.

I re-deployed the project. Although the console said BUILD SUCCESSFUL, the app engine instance no longer works. No matter how many times I re-deploy or update my application, the logs show nothing but errors:

java.lang.NoClassDefFoundError: com/google/appengine/api/ThreadManager at com.google.api.control.extensions.appengine.GoogleAppEngineControlFilter.createClient (GoogleAppEngineControlFilter.java:61) at com.google.api.control.ControlFilter.init (ControlFilter.java:141) at org.eclipse.jetty.servlet.FilterHolder.initialize (FilterHolder.java:139) at org.eclipse.jetty.servlet.ServletHandler.initialize (ServletHandler.java:873) at org.eclipse.jetty.servlet.ServletContextHandler.startContext (ServletContextHandler.java:349) at org.eclipse.jetty.webapp.WebAppContext.startWebapp (WebAppContext.java:1406) at com.google.apphosting.runtime.jetty9.AppEngineWebAppContext.startWebapp (AppEngineWebAppContext.java:175) at org.eclipse.jetty.webapp.WebAppContext.startContext (WebAppContext.java:1368) at org.eclipse.jetty.server.handler.ContextHandler.doStart (ContextHandler.java:778) at org.eclipse.jetty.servlet.ServletContextHandler.doStart (ServletContextHandler.java:262) at org.eclipse.jetty.webapp.WebAppContext.doStart (WebAppContext.java:522) at com.google.apphosting.runtime.jetty9.AppEngineWebAppContext.doStart (AppEngineWebAppContext.java:120) at org.eclipse.jetty.util.component.AbstractLifeCycle.start (AbstractLifeCycle.java:68) at com.google.apphosting.runtime.jetty9.AppVersionHandlerMap.createHandler (AppVersionHandlerMap.java:240) at com.google.apphosting.runtime.jetty9.AppVersionHandlerMap.getHandler (AppVersionHandlerMap.java:178) at com.google.apphosting.runtime.jetty9.JettyServletEngineAdapter.serviceRequest (JettyServletEngineAdapter.java:120) at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.dispatchServletRequest (JavaRuntime.java:747) at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.dispatchRequest (JavaRuntime.java:710) at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.run (JavaRuntime.java:680) at com.google.apphosting.runtime.JavaRuntime$NullSandboxRequestRunnable.run (JavaRuntime.java:872) at com.google.apphosting.runtime.ThreadGroupPool$PoolEntry.run (ThreadGroupPool.java:270) at java.lang.Thread.run (Thread.java:748) Caused by: java.lang.ClassNotFoundException: com.google.appengine.api.ThreadManager at java.net.URLClassLoader.findClass (URLClassLoader.java:381) at com.google.apphosting.runtime.ApplicationClassLoader.findClass (ApplicationClassLoader.java:135) at java.lang.ClassLoader.loadClass (ClassLoader.java:424) at java.lang.ClassLoader.loadClass (ClassLoader.java:357)

Is there some way for me to clear out the partial build (or whatever there may be) in google app engine?

I tried deleting all versions and instances (you cannot delete them all, it won't let you delete the serving one).

I tried increasing the version number in appengine-web.xml.

I tried gradle clean.

I tried disabling and enabling the application in the google cloud console.

No luck.

One idea I have is maybe if I can somehow force all the files for the app to be uploaded again? Because with gradle appengineDeploy only the changed files get uploaded.

EDIT:

I managed to fix one part of this by upgrading classpath 'com.google.cloud.tools:appengine-gradle-plugin:2.2.0' in my gradle to the latest version. (Felt kind of hackish to do that and not reliable in the future if this happens again.) Maybe this flushed whatever was in app engine or something by doing that. I still have one problem remaining:

I am using the firebase admin SDK to send firebase cloud messages (authenticated with a .json credentials file) like this:

FirebaseMessaging.getInstance().send(msg);

When I do I am getting this error:

com.google.firebase.messaging.FirebaseMessagingException: Unexpected HTTP response with status: 401; body: null

Which is strange because other parts of the firebase admin SDK are working (like writing/reading from firestore).

So there is still something weird going on and I think it has to do with the fact that as I was uploading my google app engine project the connection was interrupted.

EDIT 2: Here is something interesting: When I deploy my application using gcloud app deploy appengine-web.xml the application again will not work. Deploying with gradle appengineDeploy does though. I also noticed that the size of the application is shown as much smaller in the GCP console when deploying with gcloud app deploy appengine-web.xml. So something is messed up here. I tried looking up some sort of gcloud command to clear cache? Or something like that but no luck.

EDIT 3: Additional Info: My app was already deployed and working before the failed attempt. I changed one small piece of code in a function and upon uploading the app, the connection was interrupted because the internet went out.

I am on app engine standard environment. I am deploying my application from the macOS terminal and using android studio to develop.

I have tried the stopPreviousVersion promote and version configs in gradle (actually the first two are true by default and version gets auto-generated if you do not set it).

Running gcloud app deploy appengine-web.xml --verbosity=debug shows a lot of sensitive information but one thing I am seeing is all the files in WEB-INF are being skipped:

DEBUG: Skipping upload of [WEB-INF/.... INFO: Incremental upload skipped 100.0% of data DEBUG: Uploading 0 files to Google Cloud Storage DEBUG: Using [16] threads

So perhaps files are not all being uploaded? This SO post raises a similar problem but has no solution: stackoverflow.com/q/42137452/3075340

It's weird but when I do gcloud app deploy, the app won't work at all. There are run-time errors all over the place. Doing gradle appengineDeploy fixes that but I still am having the firebase-admin issue.

Micro
  • 10,303
  • 14
  • 82
  • 120
  • The error you are getting is pretty strange, could it be that your plugin was updated in a bad way ? if you try to deploy on a new gpc project just for testing does it work ? I usually use maven but I think you can "uninstall" the appengine plugin and retry again? – Pievis Nov 27 '19 at 17:38
  • @Pievis If i deploy on a new gpc project just for testing it works. What do you mean "uninstall" the appengine plugin? like in the gcp console? – Micro Nov 28 '19 at 00:58
  • I feel like I need a way to force deploy all files. Because when I run "gradle appengineDeploy" only the changed files get uploaded. Is there a way to do that? – Micro Nov 28 '19 at 01:00
  • I was talking about whatever dependencies the gradle appengineDeploy command was using. But since it works on another project it feels like its related to the gcp project as you mentioned. This is really strange though, have you tried contacting support ? – Pievis Nov 28 '19 at 08:22
  • @Pievis I managed to fix it partially, see my edit. There just has to be someway to clean install the app or something, otherwise I am stuck in this state where half the files are being held somewhere and I can't get rid of them. I'll try contacting support if no one else responds in a few days. – Micro Nov 28 '19 at 08:45
  • Was your app already deployed and working before the failed deployment? `gcloud app deploy` is an atomic operation, meaning that each deployment using this command does not read files from previous failed deployments. From where exactly are you deploying your application? Have you tried to use the [stopPreviousVersion and version properties](https://cloud.google.com/appengine/docs/standard/java/tools/gradle-reference#appenginedeploy) respetively when running appengineDeploy? Are you using App Engine Standard or Flex environment? – Daniel Ocando Nov 29 '19 at 12:31
  • Also have you tried to use the [--verbosity=debug](https://cloud.google.com/sdk/gcloud/reference/#--verbosity) when running `gcloud app deploy` in order to further troubleshoot the issue? – Daniel Ocando Nov 29 '19 at 12:33
  • @DanielOcando I answered all your questions in my edit at the bottom. Thank you for your help. – Micro Nov 29 '19 at 16:03
  • Try creating an `ignoretest` file to override your `.gcloudignore` file (all the relevant information about this last file is [here](https://cloud.google.com/sdk/gcloud/reference/topic/gcloudignore)) on the top level directory where your appengine-web.xml is located. This file should contain only an entry with the name of the file (ignoretest) and nothing else. Run the deploy command with the relevant flag: `gcloud app deploy appengine-web.xml --ignore-file=ignoretest` to ensure that all the files are uploaded. Notice this could make the deployments very slow since everything is uploaded. – Daniel Ocando Dec 01 '19 at 23:30
  • @DanielOcando I get `ERROR: (gcloud.app.deploy) Cannot have both an ignore file ignoretest and skip_files defined in the same application. We recommend you translate your skip_files ignore patterns to your ignoretest file.` when trying to do that. I do not have a `skip_files` anywhere (nor was there any `.gcloudignore` in my project). – Micro Dec 02 '19 at 06:56
  • @DanielOcando My project is in Java by the way. – Micro Dec 02 '19 at 07:22
  • Not even as a [hidden file](https://stackoverflow.com/questions/29135878/what-is-the-quickest-way-to-toggle-hide-show-hidden-files-on-a-mac-os-x-yosemite)? – Daniel Ocando Dec 02 '19 at 13:39
  • @DanielOcando Not even as a hidden file. I made sure. I tried creating a `.gcloudignore` file as well and I got the same error as my pervious comment. `skip_files` applied to python app engine projects from what I understand (as this user had a similar issue: https://stackoverflow.com/q/52101736/3075340) - however, mine is a gradle java project. – Micro Dec 02 '19 at 15:13
  • Can you try to deploy with another bucket, it should reupload all the files, i believe.`gcloud app deploy appengine-web.xml --bucket ` – Emil Gi Dec 03 '19 at 13:56
  • @EmilGi I made a new bucket and did what you said but with `--verbosity=debug` the console said only 15 files (out of hundreds) were uploaded.. And the app is in the same broken state as if using the default bucket. Very weird that it wouldn't re-upload all the files. Thank you for the suggestion though. – Micro Dec 03 '19 at 15:35
  • Just for clarification, can you make sure you don't have `skip_files` entry in your app.yaml? Cause it is actually supported in java. Otherwise this is really weird, app deploy definitely skipping files somehow. – Emil Gi Dec 03 '19 at 16:46
  • @EmilGi I do not even have an `app.yaml` file. Deploying with `gcloud app deploy` deploys only a few files. However `gradle appengineDeploy` deploys all (so it seems). However, firebase-admin-sdk is still broken - so something is weird. Either way, `gcloud app deploy` should upload all files somehow. – Micro Dec 04 '19 at 14:52
  • @EmilGi Also when I look at the size of the app in the GCP console, the app deployed with `gcloud` is much smaller than the one deployed with `gradle appengineDeploy` - which is odd. – Micro Dec 04 '19 at 15:10
  • @Micro may be try this `gcloud config set gcloudignore/enabled false` from https://cloud.google.com/sdk/gcloud/reference/topic/gcloudignore#DESCRIPTION – Emil Gi Dec 04 '19 at 15:50
  • @EmilGi that had no effect – Micro Dec 05 '19 at 03:12
  • This issue might be caused by outdated SDK libraries. Can you try `gcloud components update` if you are not on the latest version. – Emil Gi Dec 05 '19 at 08:37
  • @EmilGi Tried that now and before. All components are up to date it says. – Micro Dec 05 '19 at 12:13
  • What does 'api_version' state after deployment in your auto-generated yaml configuration. You can check it in GCP console -> app engine -> versions. It should be '1'. And also can you check if you have `appengine-api-1.0-sdk-1.9.77.jar` uner `WEB-INF/lib/`? – Emil Gi Dec 05 '19 at 15:06
  • @EmilGi In the generated yaml the `api_version` is `user_defined`. However, in the GCP console the version is an auto-generated `20191206t492842`. I do indeed have that exact `.jar` file in `WEB-INF/lib/` – Micro Dec 05 '19 at 17:02
  • @Micro from what i understand, this 'user_defined' is telling the app to use local jar file uploaded during deploy. In your case it doesn't work properly and continues to use runtime-provided jar missing your code i believe. Can you check that the jar file under `WEB-INF/lib/` contains your code/classes? – Emil Gi Dec 06 '19 at 08:29
  • @EmilGi The jar files in the build folder under `WEB-INF/lib/` contain .jar files from all of my gradle imports (I have a Java project with gradle). However, there aren't any .jar files of my written classes in `WEB-INF/lib/`. My written classes are only contained in `WEB-INF/classes` and as `.class` files. – Micro Dec 07 '19 at 09:50

2 Answers2

0

To actually answer your question, GAE uses a staging bucket to cache files.

You can delete this bucket, and it will get recreated next time you try to deploy.

The bucket is always named staging.[PROJECT-ID].appspot.com. It should show on the buckets overview page

Just to be clear, I'm not anywhere near sure that this will actually resolve your issue, but this will most definitely clear whatever files GAE cached

Edo Akse
  • 4,051
  • 2
  • 10
  • 21
  • I tried uploading to a brand new bucket as well as deleting everything from the staging bucket. Using `gradle appengineDeploy` many (all?) files get uploaded but `gcloud app deploy` only uploads a few. So something is off there. Either way, something is off because with `gradle appengineDeploy` the firestore-admin-sdk is broken. – Micro Dec 04 '19 at 14:57
  • from this we can determine that the cache from GAE is not the issue – Edo Akse Dec 05 '19 at 14:23
  • hmm then what is going on? – Micro Dec 05 '19 at 15:59
  • If `gcloud app deploy` only uploads a few files, then it looks like you didn't actually empty the staging bucket. Emptying the staging bucket should cause all your files to be uploaded. – new name Dec 08 '19 at 18:25
0

Here there was a discussion on a, more or less, similar issue on App Engine from last year.

You can use it as a source of inspiration to fix your current issue, because I think they are really similar as a concept. From there I think it may worth trying to put the appengine-api-1.0-sdk-1.9.63.jar ( or whatever your version is ) file under WEB-INF/lib. It the same post there were some guys who found out the there was a problem with gcloud v194 and switching to another version fixed the issue.

Here there is a similar issue which was resolved by upgrading the gcloud tool.

Anyhow, this behaviour is not normal and things should not work like this. You can try to find a workaround, but my recommendation is to report the issue and let an App Engine engineer taking a look over it. There are good chances to be something internal.

Andrei Tigau
  • 2,010
  • 1
  • 6
  • 17