Using multiple CPU Cores with Tensorflow

Question

I am running a tensorflow training on a Linux machine with 4 cores. When checking the cpu utilization with htop, only one core is fully utilized, whereas the others are utilized only with ~15% (image below shows a screenshot of htop).

How can I make sure TF is using all CPUs to full capacity?
I am aware of this issue "Using multiple CPU cores in TensorFlow" - how to make it work for Tensoflow 2?

.

I am using the following code to generate the samples:

class WindowGenerator():

    def make_dataset(self, data, stride=1):
        data = np.array(data, dtype=np.float32)
        ds = tf.keras.preprocessing.timeseries_dataset_from_array(
            data=data,
            targets=None,
            sequence_length=self.total_window_size,
            sequence_stride=stride,
            shuffle=False,
            batch_size=self.batch_size,)
      
        ds = ds.map(self.split_window)
      
        return ds


    @property
    def train(self):
        return self.make_dataset(self.train_df)
    
    @property
    def val(self):
        return self.make_dataset(self.val_df)
    
    @property
    def test(self):
        return self.make_dataset(self.test_df, stride=24)

I'm using the following code to run the model training. sampleMgmt is of Class WindowGenerator. early_stopping defines the training termination criteria.

history = model.fit(sampleMgmt.train, epochs=self.nrEpochs,
                  validation_data=sampleMgmt.val,
                  callbacks=[early_stopping],
                  verbose=1)

score 0 · Answer 1 · edited Mar 12 '21 at 15:52

0

Have you tried this?

config = tf.ConfigProto(device_count={"CPU": 8})
    with tf.Session(config=config) as sess:

(source)

edited Mar 12 '21 at 15:52

0x6d64

153
2
9

answered Mar 12 '21 at 09:32

Bhargav - Retarded Skills

3,154
1
6
22

1

I'm not sure how to use the sess object to start training. (Actually I thought that was a TF1.x syntax) – vfromate Mar 12 '21 at 09:45
In the link you referenced to, Yaroslav is mentioning that "By default cpu:0 represents all cores available to the process.". Does this mean, that it is perfectly normal, that htop only shows high load in one cpu, and not all of them? – vfromate Mar 12 '21 at 09:49

score 0 · Accepted Answer · answered Mar 15 '21 at 15:58

0

In the end, I did two things outside my code to make TF use all CPUs availible.

I used a SSD instead of a HDD
I used the Tensorflow Enterprise Image, provided in Google Cloud, instead of the Deep Learning Image

answered Mar 15 '21 at 15:58

vfromate

41
7

Using multiple CPU Cores with Tensorflow

2 Answers2