2

Default Conv2DFlipout layer looks like this

tfp.layers.Convolution2DFlipout(
    filters, kernel_size, strides=(1, 1), padding='valid',
    data_format='channels_last', dilation_rate=(1, 1), activation=None,
    activity_regularizer=None,
    kernel_posterior_fn=tfp_layers_util.default_mean_field_normal_fn(),
    kernel_posterior_tensor_fn=(lambda d: d.sample()),
    kernel_prior_fn=tfp.layers.default_multivariate_normal_fn,
    kernel_divergence_fn=(lambda q, p, ignore: kl_lib.kl_divergence(q, p)), bias_pos
    terior_fn=tfp_layers_util.default_mean_field_normal_fn(is_singular=True),
    bias_posterior_tensor_fn=(lambda d: d.sample()), bias_prior_fn=None,
    bias_divergence_fn=(lambda q, p, ignore: kl_lib.kl_divergence(q, p)), seed=None,
    **kwargs
)

Can someone tell how can I involve kernel and bias priors in training. Also the default multivariate_normal_fn looks like this,

tfp.layers.default_multivariate_normal_fn(dtype, shape, name, trainable, add_variable_fn)

It has trainable as one of the boolean argument, if I set it True, it throws following error.

tfp.layers.Convolution2DFlipout(
    kernel_prior_fn = tfp_layers_util.default_multivariate_normal_fn(trainable=True)
)


TypeError: default_multivariate_normal_fn() missing 4 required positional arguments: 'dtype', 'shape', 'name', and 'add_variable_fn'

Can someone help me solve the issue?

uselessrunner
  • 125
  • 6
  • Why do you want to train a prior? It does not make sense mathematically. Also `default_multivariate_normal_fn` is not trainable in TFP. Yes I also don't know why they put trainable parameter there, since it is deleted. Check [the source](https://github.com/tensorflow/probability/blob/v0.12.2/tensorflow_probability/python/layers/util.py#L202-L224) – Frightera May 18 '21 at 19:27
  • You can refer to this [blog](https://blog.tensorflow.org/2019/03/regression-with-probabilistic-layers-in.html) by tfp team, they have used trainable prior for DenseVariational layers, this method is called Empirical bayes. In my application, with the fix priors, the kl-divergence error is extremely high, out of complexity of the data, I am completely clueless about what my prior belief should be. Hence I wish to implement Empirical Bayes method. – uselessrunner May 19 '21 at 18:34
  • If you train the prior then it is not a completely true Bayesian model. It means you are conditioning the prior on the data which does not make sense as priors should not be conditioned on the data. *I am completely clueless about what my prior belief should be*: That's another problem with Bayesian model, it is hard to pick a good prior. – Frightera May 19 '21 at 18:40

0 Answers0