Running Java and Python standard together locally

Question

My project has been using Python on App Engine standard for a while, and we've been able to locally run the services using dev_appserver.py.

Recently, we've started developing services on Java 8 as well. The new service is being built with the app-gradle-plugin and can be run on a dev server using the gradle task appengineRun. However, these services need to interact with each other, so I'm looking for a way to run them side-by-side, especially on the same dev_appserver process.

What I've tried:
Running dev_appserver.py and pointing it at the configuration files for both the Python services and the Java one. This almost seems to work: dev_appserver doesn't report any errors, and it starts the Python and Java apps at ports 8080 and 8081. Then it starts what appears to be a second Jetty server at a random port:

$ dev_appserver.py public-api/src/main/appengine/app.yaml api/api.yaml
INFO     2018-02-16 21:23:05,712 devappserver2.py:105] Skipping SDK update check.
INFO     2018-02-16 21:23:05,758 api_server.py:308] Starting API server at: http://localhost:58551
INFO     2018-02-16 21:23:05,764 dispatcher.py:255] Starting module "public-api" running at: http://localhost:8080
WARNING  2018-02-16 21:23:05,764 dispatcher.py:316] Your python27 micro version is below 2.7.12, our current production version.
INFO     2018-02-16 21:23:05,775 dispatcher.py:255] Starting module "api" running at: http://localhost:8081
INFO     2018-02-16 21:23:05,781 admin_server.py:146] Starting admin server at: http://localhost:8000
WARNING  2018-02-16 21:23:05,781 devappserver2.py:176] No default module found. Ignoring.
Feb 16, 2018 9:23:07 PM com.google.appengine.tools.development.AbstractContainerService configure
WARNING: Null value for containerConfigProperties.get(devappserver.portMappingProvider)
2018-02-16 21:23:07.343:INFO::main: Logging initialized @521ms
2018-02-16 21:23:07.553:INFO:oejs.Server:main: jetty-9.3.18.v20170406
2018-02-16 21:23:07.667:INFO:oeja.AnnotationConfiguration:main: Scanning elapsed time=0ms
2018-02-16 21:23:07.865:INFO:oejsh.ContextHandler:main: Started c.g.a.t.d.j.DevAppEngineWebAppContext@5fbe4146{/,file:///Users/gmiller/workspace/skynet/public-api/src/main/appengine/,AVAILABLE}{/Users/gmiller/workspace/skynet/public-api/src/main/appengine}
2018-02-16 21:23:07.878:INFO:oejs.AbstractConnector:main: Started NetworkTrafficSelectChannelConnector@45b4c3a9{HTTP/1.1,[http/1.1]}{localhost:58560}
2018-02-16 21:23:07.888:INFO:oejs.Server:main: Started @1065ms
Feb 16, 2018 9:23:07 PM com.google.appengine.tools.development.AbstractModule startup
INFO: Module instance public-api is running at http://localhost:58560/
Feb 16, 2018 9:23:07 PM com.google.appengine.tools.development.AbstractModule startup
INFO: The admin console is running at http://localhost:58560/_ah/admin
Feb 16, 2018 9:23:07 PM com.google.appengine.tools.development.devappserver2.DevAppServer2Impl doStart
INFO: Dev App Server is now running

Now that the services have been started, I want to be able to contact either. The Python service acts as I expect, but using either port for the Java service just returns 404s from Jetty:

Feb 16, 2018 9:30:01 PM com.google.appengine.tools.development.jetty9.LocalResourceFileServlet doGet
WARNING: No file found for: /public-api/docs/view
INFO     2018-02-16 21:30:01,667 module.py:833] public-api: "GET /public-api/docs/view HTTP/1.1" 404 83

Is running Python and Java services locally and concurrently supported/possible? Am I missing any sort of configuration?

Edit: I've tried running just the Java app using dev_appserver.py like so: dev_appserver.py public-api/build/exploded-public-api/ and then I'm able to reach static files in the service, but none of the paths configured using Spring Boot. It seems like perhaps the SpringBootServletInitializer is not being called. Do I need to add a web.xml or some other configuration to get the server loaded properly?

Answer 1

I believe you should be able to run both locally

Try going to the admin server at localhost:8000 and clicking on the links for each service in there?

https://cloud.google.com/appengine/docs/standard/python/how-requests-are-routed#routing_in_the_development_server

Also, are you using a dispatch.yaml that could be intercepting requests? https://cloud.google.com/appengine/docs/standard/python/config/dispatchref

Without a dispatch.yaml you normally have to specify the service in the url (although i don't think that applies to the local dev-server):

https://SERVICE_ID-dot-MY_PROJECT_ID.appspot.com

https://cloud.google.com/appengine/docs/standard/python/how-requests-are-routed#routing_via_url

Answer 2

The problem is that a different local server executable (with different invocation args) should be used for Java vs Python modules, see the Python Using the Local Development Server and Local Development Server Options vs Using the Java Local Development Server.

You attempted to use the Python devserver for the Java app, which likely explains why it only partially works (it probably is able to understand some of the configs, but not all).

I'm not familiar enough with the Java side to say if it's indeed possible, but I'd try first to run the Java services alone through the java devserver and get that working properly.

Then I'd try to run them side-by-side with the python server handling the python services, focusing on resolving immediate port conflicts and such.

One other possible investigation path (maybe to be attempted first?) would be to dive into the internals of the wrappers actually invoked from the <gcloud_sdk>/bin directory and see if what they end up executing under the hood can be somehow reconciled across both languages in order for a single core process to serve both environments simultaneously.

If a single core cannot be achieved I'm afraid you won't be able to properly execute all your services simultanesouly - separate cores means different services may not see a single, consistent GAE app "sandbox" (for example each core would have its own version/copy of the datastore/memcache data), defeating the purpose.

Ideally Google should provide in the SDK(s) a way of mixing any combination standard and/or flexible environment services written in any supported language - in other ways properly emulating what the real GAE infra supports. Without it there is no decent way of developing/testing apps with such mixes locally.