Selecting/Interpolating 3-D track (time, lat, lon) from dataset with 2-D lat-lon coordinates

Question

I have an xarray Dataset that looks like this

As you can see latitude and longitude coordinates are 2-D arrays with dimensions south_north and west_east. I want to extract from this dataset a timeseries of the data variables over a trajectory defined by time, lat and lon, e.g.

time                         lat    lon
2020-09-15 00:00:00+00:00   32.80   17.02
2020-09-15 01:00:00+00:00   32.89   16.97

With a dataset defined by regular lat and lon coordinates I would do something like this

ds.interp(time=xr.DataArray(time),
          lat=xr.DataArray(lat),
          lon=xr.DataArray(lon),
          method='nearest')

but this method does not work here, as the coordinates are 2-D arrays.

Furthermore, I can not use the method described in xarray select nearest lat/lon with multi-dimension coordinates as I need to select also on the time dimension together with latitude and longitude.

Sure, I could just iterate through the time dimension in the dataset and find the indices of the closest data point in latitude and longitude at every time step, but I'm sure there must be a smarter way in xarray.