0

I would like to update state, So here is what i wrote:

def create_keras_model():
...
   return model
iterative_process = tff.learning.build_federated_averaging_process(..)

My problem is loss increase contrary to the accuracy which makes a small decrease:

round  1, metrics=OrderedDict([('categorical_accuracy', 0.4675926), ('loss', 8.581259)])
round  2, metrics=OrderedDict([('categorical_accuracy', 0.65625), ('loss', 5.4126678)])
round  3, metrics=OrderedDict([('categorical_accuracy', 0.6018519), ('loss', 6.37924)])
round  4, metrics=OrderedDict([('categorical_accuracy', 0.587963), ('loss', 6.5979366)])
round  5, metrics=OrderedDict([('categorical_accuracy', 0.6400463), ('loss', 5.7463913)])
round  6, metrics=OrderedDict([('categorical_accuracy', 0.6909722), ('loss', 4.872179)])
round  7, metrics=OrderedDict([('categorical_accuracy', 0.6469907), ('loss', 5.6218925)])
round  8, metrics=OrderedDict([('categorical_accuracy', 0.7037037), ('loss', 4.723536)])
round  9, metrics=OrderedDict([('categorical_accuracy', 0.7002315), ('loss', 4.774122)])
round 10, metrics=OrderedDict([('categorical_accuracy', 0.7060185), ('loss', 4.6346316)])
round 11, metrics=OrderedDict([('categorical_accuracy', 0.6724537), ('loss', 5.213738)])
round 12, metrics=OrderedDict([('categorical_accuracy', 0.6608796), ('loss', 5.450448)])

Is there another solution to solve this problem ? Thanks

seni
  • 659
  • 1
  • 8
  • 20

1 Answers1

1

Are these the training metrics or test metrics? Its important to note that training metrics have "weird" behavior in Federated Learning; see MSE error different during training and evaluation in tensorflow federated.

If create_keras_model is loading pretrained weights that already accurately learned the distribution of the data, this might be some fine-tuning step. However if these are training metrics, they may not be telling the whole story (as above). I would also recommend running for more rounds, 10 rounds is generally not enough in Federated Learning.

For more about interpreting machine learning metrics in general, see How to interpret increase in both loss and accuracy?.

Zachary Garrett
  • 2,911
  • 15
  • 23