Fundamentally, if you just want your app to scale seamlessly and quickly without much input, use App Engine Flex. If you want more control that you can configure in different ways using other Google products, consider an unmanaged instance group with Compute Engine.
Some history
It's worth noting that the ability to add Docker images to Compute Engine instances was an extremely recent development. And before that, App Engine Flexible was the new kid in town because we used to only have App Engine standard which definitely didn't allow you to use Docker as a base.
Key Differences
Here are the key differences in my experience:
- App Engine is designed as a PaaS product and so you can customize scaling parameters in your app.yaml and App Engine reads those and takes over from you. It's technically true that you can do this using Compute Engine but it involves more configuration - you need to set up an instance group, a backend and a frontend. With App Engine, all of that is taken care of for you
- You can't setup any load balancers or any peripheral services or products on top of App Engine. App Engine Flexible went quite a way further to give the user more control and more (sorry) flexibility. However, it doesn't allow unfettered integration with other services. Whereas with Compute Engine, you can setup an HTTPs load balancer, add your machines to different networks and subnets, set custom tags etc...
On an additional note, a more detailed explanation on the difference between App Engine and Compute Engine can be found here.