7

Does anyone have any advice on making the logging in Google App Engine better? I am currently trying to use Splunk Storm, but they are finicky regarding input and go down often. Has anyone else encountered this and solved it in some capacity?

Currently I have a process that runs in a backend that reads from the LogService and pipes the logs into Splunk Storm via REST api. This often fails, or storm goes down, or the backend IP changes.

My issue is with the logging provided within App Engine, as the logs disappear when new versions are pushed and querying the logs with the provided dashboard is almost unusable. Splunk was a potential solution, but the cloud solution leaves a lot to be desired.

Anything that would provide a better interface into my logs would be appreciated.

BrettJ
  • 6,801
  • 1
  • 23
  • 26
Mantas Vidutis
  • 16,376
  • 20
  • 76
  • 92
  • Can you describe in more detail what is failing or insufficient for your needs? It's unclear if your issue is with App Engine, Splunk, or both. – Adam Thomason Sep 10 '12 at 21:23
  • Re "logs disappear when new versions are pushed", this shouldn't happen; however, you need to explicitly ask for logs app versions other than the one making the query. In Python, for example, the call to logservice.fetch takes a "version_ids" parameter which you can use to request older logs; Java has LogQuery.majorVersionIds(). If you're already doing that and the logs really are disappearing, please file a bug. – Adam Thomason Sep 10 '12 at 21:55
  • Those logs are only around as long as the versions are around. As an application grows in size, we can only keep 2 or 3 versions/backends of the app on GAE. We push multiple times a day. Depending on GAE to hold onto our logs is no longer an option, and while we try to extract them as quickly as possible, that puts a dependency on an external service. I was hoping someone else had come up with a better solution. – Mantas Vidutis Sep 10 '12 at 22:01
  • It's a common misconception that the logs disappear with the version. They really *do* stick around, but you may only be able to see them using appcfg.py request_logs ... – Guido van Rossum Sep 11 '12 at 01:40

2 Answers2

5

You can export logs from GAE to BiqQuery which has quite capable query language. You can use Mache, an open-source project that already does this. You should write your own exporter, to expose (and make queryabe) fields (columns) you are interested in.

Peter Knego
  • 79,991
  • 11
  • 123
  • 154
1

Since you've decided to use Splunk (or another external service) as permanent storage, it sounds like you need a location to buffer logs between the times when they're written to App Engine's log service and when Splunk is available to accept the logs. To avoid losing logs before version churn causes them to fall out of App Engine, this buffer needs to be fast and highly available.

One reasonable choice is the AE datastore. There's no unreliable hop to a 3rd party, it has an availability SLA, and it can be scaled arbitrarily by sharding writes. The downside would be the cost of R/W operations and the storage footprint of in-flight logs, but you'll incur a comparable cost for another backing store.

Whatever choice of service, have one batch process (e.g. backend or cronjob) write to the buffer from the logs reader API. As long as it runs more often than app updates, logs will always exist in durable storage. Then have another batch process wait for Splunk to be available then upload to it from the buffer and delete as you get receipt confirmation from Splunk.

Adam Thomason
  • 993
  • 1
  • 8
  • 10