I ran my app with the --trace_gc flag to attempt to find some performance issues. Well, it looks like I might have found it...
1288678 ms: Mark-sweep 498.8 (549.0) -> 488.8 (548.0) MB, 4085 ms [idle notification: finalize idle round] [GC in old space requested].
Gadzooks! More than 4 seconds to do GC. No wonder I'm having issues.
Now, the real question: how can I find what objects are being GC'd, and more importantly, where they were allocated? I've already used heap snapshots in nodetime, but I'm not seeing enough information to help me discover where these objects are coming from. I do see some hints that lead me to believe, perhaps, that I have a cyclic-reference somewhere (eg, I see hundreds of "user" properties in the heap snapshot after just a couple API calls by a single user).
Are there any good tutorials out there or other good information about how to trace the cause of these large garbage collection times? Or maybe I could print the allocated objects somehow, to try to find a cyclic reference...?