Custom google maps marker loading and updating very slowly

Question

I have to implement custom Google maps GMSMarker. I have referred google documentation to create a custom marker.

Issue which I am facing is whenever I want to update the marker it is taking significant amount of time (1 sec - 3sec) to update, which degrades user experience. I am NOT creating multiple markers I just have 2 markers in current implementation.

Here is the code which I am using to create marker :

- (void)showMarker 
{
     gmsMapView = [GMSMapView mapWithFrame:self.mapView.bounds camera:camera];
     CLLocationCoordinate2D position = { latitude, longitude };
     marker = [GMSMarker markerWithPosition:position];
     marker.flat = YES;
     marker.iconView = [self setCustomIcon];
     marker.map = gmsMapView;
     marker.userData = slot;
}
- (UIView *)setCustomIcon
{
     Marker *view =  [[[NSBundle mainBundle] loadNibNamed:@"Marker" owner:self options:nil] objectAtIndex:0];
     /*
       few lines of code to set data
     */    
     return view;
}

NOTE :

1. I know I can create custom info window but it not what I am looking for.

2. GMSMarker has a property tracksViewChanges which stops tracking changes in marker view, when set to NO. But I cannot set it to NO as I need to update view when it is tapped.

3. This issue only comes if I set iconView property of marker, if I just use icon property and set an image to it, It works fine.

EDIT : Update Marker Code

- (void)updateMarker:(GMSMarker *)localMarker
{
   dispatch_async(dispatch_get_main_queue(), ^{
     self.previousSelectedMarker.iconView = [self setCustomIcon];
     localMarker.iconView = [self setCustomIcon];
     self.previousSelectedMarker = localMarker;
     self.slot = localMarker.userData;
      // update data
  });
}

Below image shows how much delay comes when I tap on a marker an update its image :