1

I'm using nvprof to profile something (which includes both CPU work and GPU work, i.e. I use nvprof markers etc.), and I get binary files which nvprof produces. I can import these into NVVP (NVidia Visual Profiler; Linux version), and with a little effort also save that out to an XML.

However... the XML does not contain timing data about what my various CPU do when. It mentions their existence, but no more. Also, the end of the XML has this binary blob, probably Base64-encoded or something, inside a PDM tag. It's not clear to me whether there's any help there.

einpoklum
  • 118,144
  • 57
  • 340
  • 684

1 Answers1

5

It is quite an old question, but maybe somebody will find the answer useful.

nvprof output files are actually SQLite3 databases, which you can open either with standalone sqlite3 program or programmatically. The timeline information is inside these tables (all timestamps are in nanoseconds):

  • CUPTI_ACTIVITY_KIND_CONCURRENT_KERNEL - data about kernels
  • CUPTI_ACTIVITY_KIND_MEMCPY - data about memory copies (non-P2P)
  • CUPTI_ACTIVITY_KIND_MEMCPY2 - data about P2P memory copies
  • CUPTI_ACTIVITY_KIND_MEMSET - data about memsets
  • CUPTI_ACTIVITY_KIND_RUNTIME - data about CUDA Runtime API calls
  • CUPTI_ACTIVITY_KIND_DRIVER - data about CUDA Driver API calls
  • CUPTI_ACTIVITY_KIND_MARKER - data about NVTX markers. It has a little different form than the other tables, because it does not have start and end fields. Instead, start and end of a marker are 2 entries (end has name=0)

You can correlate API calls with kernels/memcopies/memsets using correlationId field.

ptrendx
  • 316
  • 2
  • 6
  • Wow, thank you. While I'm not trying to do this just now, in a couple of months I think I'll get around to utilizing this tidbit of information :-) ... how did you figure that out, by the way? – einpoklum Nov 23 '16 at 16:49
  • I work at NVIDIA ;-). But it is actually documented (at least the fact that it is SQLite3 db), although I agree that spotting 1 mention of it in the entire documentation is hard. What do you intend to use it for? – ptrendx Nov 23 '16 at 18:57
  • Automating performance benchmark statistics gathering. If you want more details, find me [through here](https://github.com/eyalroz). – einpoklum Nov 23 '16 at 19:58
  • With the help of this question's answer, I wrote a tool for converting nvprof output files to Google Event Trace format. You can get it at https://github.com/ezyang/nvprof2json – Edward Z. Yang Jun 08 '17 at 20:20