10

I have an app-engine app that is deployed and running on the standard Python3 runtime. I am also able to run it locally using normal commands like flask run. But I am unable to run it the way I could run apps in the 2.7 runtime with dev_appserver.py. I am using the latest gcloud version, but running dev_appserver.py results in:

ERROR: Python 3 and later is not compatible with the Google Cloud SDK. Please use Python version 2.7.x.

I assumed we just aren't supposed to do it this way any more until it saw: https://cloud.google.com/appengine/docs/standard/python3/tools/local-devserver-command

Which starts with:

Python 3.7 Local Development Server Options

Usage:
dev_appserver.py [options] yaml_path [files...]

Is dev_appserver.py compatible with Python3 or not (maybe I somehow have an old version in my path)? If not, is there a new way to run apps locally that will adhere to the app.yaml(like treating static paths correctly) and give other niceties like the local data store that dev_appserver.py provided?

Community
  • 1
  • 1
Mark
  • 90,562
  • 7
  • 108
  • 148

3 Answers3

12

The dev_appserver.py support for Python 3 is still limited. From Running the local development server:

Note:

  • Running dev_appserver requires the presence of Python 2.7.12+ on your local machine.
  • The updated dev_appserver does not support development of Python 3 apps on Windows.

Which might be why it is not the recommended solution for local development (or at least not yet). From Running locally:

We do not recommend that you depend on tools like dev_appserver, the local development server provided with the Google Cloud SDK. However, if you are migrating an existing App Engine application from Python 2 to Python 3, we have updated dev_appserver to facilitate this process. For all other local development scenarios, we recommend standard Python testing approaches.

For example, you can usually run a Flask application with Flask's development server using:

python main.py

Django applications can be started using:

python manage.py runserver

To simulate a production App Engine environment, you can run the full Web Server Gateway Interface (WSGI) server locally. To do this, use the same command specified as entrypoint in your app.yaml, for example:

gunicorn -b :$PORT main:app
Dan Cornilescu
  • 39,470
  • 12
  • 57
  • 97
  • This was the piece I was missing. Thanks Dan. – Mark Oct 26 '18 at 18:09
  • 7
    The solution they are suggesting leaves much to be desired, first, what about the url handlers on app.yaml among other things? now we need to duplicate that on our python apps--and cannot test it before production. Second, where is this "updated" dev_appserver? or how would one get it? updating the gcloud yields the same error. Bad app engine. – maininformer Nov 27 '18 at 18:19
  • 3
    @plumSemPy This is my frustration too. In gen 1 GAE, almost all development and test could be done locally. in gen 2, one has to constantly deploy his App in order to test it, slowing down development a lot. I've decided to stick with Gen 1, and then if i need Client Libraries, to make a separate "API" App in Gen 2 GAE for my main Gen 1 App to call. Allows most development to be done locally still. – hamx0r Feb 15 '19 at 00:05
  • 4
    its really annoying and almost like Google don't want users on the App Engine standard platform anymore. all the good features on AppEngine have gone with the move to Paython3 so porting apps from 2.7 Python3 is basically a re-write and if we are re-writing applications we might as well look at other providers.... real shame. – Joe Bourne Mar 01 '19 at 01:19
  • @dan-cornilescu where is the "updated" dev_appserver? Anyway this: "We do not recommend that you depend on tools like dev_appserver, the local development server". Seems ironic – kip2 Mar 04 '19 at 08:19
  • 1
    @kip2 - in the current gcloud SDK (the older GAE SDK may not have it). – Dan Cornilescu Mar 04 '19 at 12:14
  • 3
    Migrating an app to python 3 is a nightmare. It really does seem like google was not prepared to launch this yet but had to due to the impending python 2 deprecation. Local development is harder because of the missing app server and the documentation and features are still miles apart from what you had on python 2, really frustrating. – Kreender May 12 '19 at 12:12
  • 3
    It is 2019-06-04 and the situation is still a problem. The biggest problem is inability to run the local version of google cloud store. I may be wrong about it. I just do not see the way to do it. – Michael Jun 04 '19 at 17:42
  • @Michael: did you try the [Datastore mode Emulator](https://cloud.google.com/datastore/docs/tools/datastore-emulator)? – Dan Cornilescu Jun 05 '19 at 02:18
  • @DanCornilescu, sorry no. It is a good tip. Looks nifty. Have you tried it out yourself? Does it work well on Windows? – Michael Jun 27 '19 at 10:55
  • @Michael I only tried it for this answer https://stackoverflow.com/a/50860147/4495081, actively I'm still using the dev server emulation. And on Linux, I'm not a windows fan ;) – Dan Cornilescu Jun 27 '19 at 11:09
  • @DanCornilescu, Ok, checked it out. Couple of things. One, the Cloud Datastore Emulator is actually slowly becoming the implementation of dev_appserver uses behind the scenes. You can start dev_appserver using CDE via a command line switch. If you do, you'll see console out of a message stating it is being slowly rolled out, and available for some people. Next I ran dev_appserver without the switch - and turns out I am one of those lucky to having it. The biggest problem so far with GCE is the lack of standard UI. Exploring UI options now – Michael Jun 28 '19 at 05:04
  • Its 2023 and I'm unable to migrate from web app2 to django in python 3. Yes, I should sue my company for still using web app 2 – hassansuhaib Mar 20 '23 at 09:19
1

For windows 10 :

The solution was to install a linux subsystem for windows : https://learn.microsoft.com/en-us/windows/wsl/install-win10

I use the Ubuntu app from the windows store, and follow the installation for installing the gcloud component on linux inside the ubuntu terminal: https://https://learn.microsoft.com/en-us/windows/wsl/install-win10cloud.google.com/sdk/docs/#deb

Inside the Ubuntu terminal it is possible to access the windows file : the C drive would be /mnt/c. Getting back to my work folder, it is possible to start the web-app using the dev_appserver command.

Using a navigator from windows we have access to the web-app as normal using localhost:8080.

The development can still be done using an IDE in windows, running the server in Ubuntu.

Stephane Duteriez
  • 934
  • 8
  • 8
0

Adding an update in 2022

  1. From the link Running the local development server mentioned by @dan-cornilescu in his response, they have changed the verbiage and removed the text that says ...we do not recommend that you depend on tools like dev_appserver.... . Instead the text now says

We recommend that you use standard Python tools, such as virtualenv to create isolated environments...

  1. It also looks like they have enhanced dev_appserver.py to work with Python 3. Running the local development server now says

To run dev_appserver with a Python 3 interpreter, you must specify the --runtime_python_path=[PATH_TO_PYTHON3_BINARY] flag.

  1. I don't know why their documentation gives the impression that the advantage of using standard python tools over dev_appserver.py is to be able to have virtual env given the fact that when you run your Python3 App with dev_appserver.py, it will first create a temp folder, create a virtual environment in that folder and install the contents of your requirements.txt file there.

  2. The downside with using dev_appserver.py is that each time you start your App, it will create a new virtual env and reinstall the contents of your requirements.txt. This means that while you're still troubleshooting your App (which might involve multiple restarts), you will end up with lots of folders in your tmp directory and these are not deleted when the app is shut down. There is also the time it takes to install the requirements.

    Update on April 25, 2023 - gcloud CLI version 422.0.0 now supports a flag, --python_virtualenv_path. This is a directory where dev_appserver.py will create your virtual env the first time and subsequent runs of your App will just use that without having to reinstall your requirements

  3. If you start the Cloud Datastore Emulator and set the flag for dev_appserver.py to use it, then you'll be able to view your data in the same UI that you're used to for Python 3 projects (@michael this should answer your question about UIs). Below is a screen shot

    enter image description here

  4. There are also 3rd party UIs from - https://github.com/GabiAxel/google-cloud-gui, https://www.npmjs.com/package/google-cloud-gui

  5. Our App, NoCommandLine also has a UI for datastore

    enter image description here

NoCommandLine
  • 5,044
  • 2
  • 4
  • 15
  • I tried to run it that way and kind of succeeded to start it, but almost immediately it crashed on something releated to datastore emulator. – Michael Apr 18 '23 at 18:16
  • @Michael - can you provide more context? What are you trying to do? What did you do? What is the error? – NoCommandLine Apr 18 '23 at 23:03
  • sorry you are right I should have provided more context. WIll try and do soon. For now I switched to Ubuntu, setting up the whole dev environment. Once it works I'll come back and give windows one more try and report here – Michael Apr 20 '23 at 03:11
  • Ohhh, if you were doing it on Windows, then it won't work. ```dev_appserver.py``` isn't supported for Python 3 on Windows - see [documentation](https://cloud.google.com/appengine/docs/standard/tools/local-devserver-command?tab=python). We have a [patch](https://github.com/NoCommandLine/dev_appserver-python3-windows) that overcomes that limitation – NoCommandLine Apr 20 '23 at 15:21
  • OMG, so I have installed Ubuntu and struggled for 2 days to properly install gcloud and 2 python environments for nothing :) !!!! My environment is that I use venv python 3. I'll revert back to windows and try your solution. – Michael Apr 23 '23 at 04:26
  • **Note:** The latest version of gcoud CLI (427.0.0 which was released on April 18) has a [breaking change](https://cloud.google.com/sdk/docs/release-notes#breaking_changes) i.e. you have to run it with Python 3 (even though you still need Python 2 for the dev_appserver.py). Our Patch hasn't been tested on it. If you use that version, would appreciate feedback on whether it worked and what, if any issues you ran into it – NoCommandLine Apr 23 '23 at 21:16
  • gcloud 427, seems to be working. Took a bit of back and forth with the emulator config, but in the end seems ok. I'll have to recheck carefully, then write it up. One thing I did not check is uploading files to storage. – Michael Apr 25 '23 at 11:10
  • update: works with externally runing datastore emulator, defined via .env. What fails is this parameter --datastore_emulator_cm. When switching to the original python2 based appengine app, this parameter works. That really is not a big issue. I have troubles with file uploads, but not sure if it is the bug in my code or implementation of emulation of GCS in the dev_appserver, where it responds 405 to OPTIONS CORS request. – Michael Apr 25 '23 at 14:38
  • When running dev_appserver , using admin viewer to see the blobstore results in the error ```... File "c:\Program Files (x86)\Google\Cloud SDK\google-cloud-sdk\platform\google_appengine\google\appengine\api\datastore_types.py", line 238, in ResolveAppId ValidateString(app, 'app', datastore_errors.BadArgumentError) File "c:\Program Files (x86)\Google\Cloud SDK\google-cloud-sdk\platform\google_appengine\google\appengine\api\datastore_types.py", line 186, in ValidateString raise exception('%s must not be empty.' % name) BadArgumentError: app must not be empty. ``` – Michael Apr 25 '23 at 16:16
  • Also using google.cloud.storage library to create a bucket fails when `STORAGE_EMULATOR_HOST=http://localhost:8080` points to the running instance of the dev_appengine. So something is not right with what I did – Michael Apr 25 '23 at 16:41
  • @Michael Don't know about ```STORAGE_EMULATOR_HOST```. The last time I checked, Cloud Storage didn't have an emulator but maybe I'm not up to date on that topic – NoCommandLine Apr 25 '23 at 16:48
  • thanks for all your work. I have finished setting up ubuntu. Running dev_appserver seems to work serving API requests and working with datastore emulator, but their UI for viewing datastore entities shows exception. I have posted about it on google cloud forum group (https://www.googlecloudcommunity.com/gc/Serverless/App-Engine-Standard-python3-10-ubuntu-22-04-dev-appserver-fails/m-p/548416#M1521) – Michael Apr 30 '23 at 06:14
  • adding --python_virtualenv_path="%VENV_DIR%" ^ fails the build on windows for me. not sure why – Michael May 04 '23 at 09:46
  • @Michael - Maybe gcloud CLI requires you to specify the full path. Our patch has been updated and any new project/app created will automatically have the flag added and will default to ```venv``` folder in application root directory – NoCommandLine May 12 '23 at 02:05
  • thanks for everything. However I have moved to ubuntu where the things are generally ok. I wish Google treated windows a bit better. – Michael Jun 02 '23 at 07:56