How do I find bottlenecks in my model?

Asked Jun 21 '18 at 15:25

Active Jun 21 '18 at 15:31

Viewed 187 times

I have been testing with a word2vec model. This word2vec model for some reason doesn't use the gpu much. My performance is roughly 1 epoch every 30 seconds with a ~2000 samples dataset.

This doesn't seem normal. There are researchers that have gigabytes of training data, and I doubt they are waiting for months for the training to finish.

My GPU is a gtx 970. The memory usage is around 10% (Note that I have a few programs open too)

The problem might be the batches itself, although I am not sure.

Basically I run a method at the start of the training, and then while training I iterate over the entries in that list.

This is roughly how I do this. Is my approach wrong? (I would guess that it's not suitable for huge datasets)

batch_method(batch_size=x) # I tested with different sizes, all seem to train fine, from 2 to 512.
for epo in self.epochs_num:
    for batch in self.batch_list:
        for input,target in batch:
        ...

edited Jun 21 '18 at 15:31

asked Jun 21 '18 at 15:25

G. Ramistella

1,327
1
9
19

check usage and see if the data is in gpu memory, from what you say you use cpu ... – n1tk Jun 21 '18 at 15:28
How do I check usage? I use pycharm. – G. Ramistella Jun 21 '18 at 15:29
2

use module cProfile – bobrobbob Jun 21 '18 at 15:30
Alright. I have a pretty long log file. I am not really understanding what is going on though. I can't see anything regarding memory usage though. – G. Ramistella Jun 21 '18 at 15:46
If you want to look at memory you can look at some of the answers [to this post](https://stackoverflow.com/questions/110259/which-python-memory-profiler-is-recommended). – Mihai Chelaru Jun 21 '18 at 15:54
It seems that the data is stored within RAM. – G. Ramistella Jun 21 '18 at 16:27

How do I find bottlenecks in my model?

0 Answers0