3

I'm looking into GP regression, but I'm getting some behaviour that I do not understand. Basically, I wanted to show convergence for GP on the osciallatory Genz function (basically a period wave), which led me to this picture Gp convergence, sorry for the missing labels (x axis: num samples, y axis: relative error measure in 2000 points)

This is OK, but I was curious why it took so long before the error started to drop. Plotting the resulting GP fit I got this (busy) plot GP fit is orange, true function is blue. What I don't understand is what happens up until it starts to capture the true function. I assumed it had something to do with the kernel. The plot here uses a RBF kernel with length_scale = 1 (I also tried both higher and lower values, but got the same results).

I kind of expected it to have a more smooth behaviour even if it couldn't capture the true model. So, to my question: why do I see this "spikey" behaviour? And can I do something to change it (kernel-wise or other)?

kernel = RBF(length_scale = 1, length_scale_bounds = (1e-2, 1e2))
gp = GaussianProcessRegressor(kernel=kernel)
gp.fit(X, y)

def genz(x, method = 'default'):
   d = x.shape[1]
   a = 10/d
   w = 1/2
   num_points = x.shape[0]
   funcval = np.empty([1,num_points])        
   for i in range(num_points):
       funcval[0,i] = np.cos(2 * np.pi * w + np.sum(a * x[i,:]))
   return funcval
desertnaut
  • 57,590
  • 26
  • 140
  • 166
anyn12
  • 31
  • 2

1 Answers1

0

It seems like the optimized length scale is very small compared to its domain space. I also felt very weird when I was digging into this library; changing some hyperparameters and the number of optimization didn't work for me as well. It might be helpful to change your kernel function to matern with changing the gamma value but not very much. If you really want to customize as you want, I might recommend you to use gpytorch similar to torch implementation or the GPML matlab toolbox.

Doo-Hyun Andrew Cho
  • 61
  • 5