I'm using the Google AppEngine Managed VM/Java since March 2015. Everything is work well. But after September, October 2015, I noticed I 've seen the error log of "com.google.apphosting.vmruntime.VmApiProxyDelegate" in Managed VM instance log frequently.
2 groups of error log I noticed.
The first one is related to datastore operation on Managed VM instance. It happens on
- datastore_v3.Get().
- datastore_v3.RunQuery()
- datastore_v3.Put()
- memcache.Get().
Sample of stacktrace I saw in the log is below ... (sample below is for datastore_v3.Put())
com.google.apphosting.vmruntime.VmApiProxyDelegate runSyncCall: HTTP ApiProxy I/O error for datastore_v3.Put: Read timed out
com.google.apphosting.api.ApiProxy$RPCFailedException: The remote RPC to the application server failed for the call datastore_v3.Get(). at com.google.apphosting.vmruntime.VmApiProxyDelegate.runSyncCall(VmApiProxyDelegate.java:182) at com.google.apphosting.vmruntime.VmApiProxyDelegate.makeApiCall(VmApiProxyDelegate.java:141) at com.google.apphosting.vmruntime.VmApiProxyDelegate.access$000(VmApiProxyDelegate.java:47) at com.google.apphosting.vmruntime.VmApiProxyDelegate$MakeSyncCall.call(VmApiProxyDelegate.java:375) at com.google.apphosting.vmruntime.VmApiProxyDelegate$MakeSyncCall.call(VmApiProxyDelegate.java:351) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745)
So, when this error occurred, at application level (my code), I got RPCFailedException but right now, I've not handled it with retry mechanism ( I only use retry mechanism for ConcurrentModificationException with App Engine Datastore API)
The second group of error log is about TaskQueue API on the Managed VM.
The error message I got is ...
com.google.apphosting.vmruntime.VmApiProxyDelegate runSyncCall: Error body: RPC Error: /StubbyService.Send to (unknown) : APP_ERROR(2)
When tracing , the detailed stacktrace is ...
com.wat_suttiwat.batchengine.job.PushNotificationTaskExecutor executeTask: The remote RPC to the application server failed for the call taskqueue.QueryAndOwnTasks(). com.google.apphosting.api.ApiProxy$RPCFailedException: The remote RPC to the application server failed for the call taskqueue.QueryAndOwnTasks(). at com.google.apphosting.vmruntime.VmApiProxyDelegate.runSyncCall(VmApiProxyDelegate.java:161) at com.google.apphosting.vmruntime.VmApiProxyDelegate.makeApiCall(VmApiProxyDelegate.java:141) at com.google.apphosting.vmruntime.VmApiProxyDelegate.access$000(VmApiProxyDelegate.java:47) at com.google.apphosting.vmruntime.VmApiProxyDelegate$MakeSyncCall.call(VmApiProxyDelegate.java:375) at com.google.apphosting.vmruntime.VmApiProxyDelegate$MakeSyncCall.call(VmApiProxyDelegate.java:351) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745)
"RPCFailedException" exception raised at the application level (like the first case). And I noticed the Google AppEngine front-end instance (not the managed-vm instance), it work as usual no error about these.
So my question is
Should I add retry-mechanism for RPCFailedException ? Is it useful to add retry? I don't see any documentation from the Google AppEngine document on this.
Does anyone has the same issues with me? If yes, please help to vote the issue at this issue-tracker (#12393): https://code.google.com/p/googleappengine/issues/detail?can=2&start=0&num=100&q=&colspec=ID%20Type%20Component%20Status%20Stars%20Summary%20Language%20Priority%20Owner%20Log&groupby=&sort=&id=12393
If you have any workaround, please share.
Thanks so much
