Why Exoplayer 2.x is not switching to lower bitrates on low network during adaptive playback?

Question

We are using exoplayer v2.x and are playing a HLS file which has 4 bitrate tracks.

When we configure exoplayer for adaptive playback, it is starting with a higher bitrate track but NOT switching back to a lower bitrate track when we throttle the network speed using Charles. The player seems to stick with the already selected higher bitrate track and keep on buffering instead of switching to a lower bitrate one.

We have configured the exoplayer in the following way:

private DefaultBandwidthMeter BANDWIDTH_METER = 
new DefaultBandwidthMeter(mUiUpdateHandler, new BandwidthMeter.EventListener() {

@Override
 public void onBandwidthSample(int elapsedMs, long bytes, long bitrate) {
       Log.v(TAG, "Elapsed Time in MS " + elapsedMs + " Bytes " + bytes + " Bitrate " + bitrate);
       bitrateEstimate = bitrate;
       bytesDownloaded = bytes;

 }
});

TrackSelection.Factory adaptiveTrackSelectionFactory =
             new AdaptiveTrackSelection.Factory(BANDWIDTH_METER);

trackSelector = new DefaultTrackSelector(adaptiveTrackSelectionFactory);


player = ExoPlayerFactory.newSimpleInstance(getActivity(), trackSelector, 
new CustomLoadControl(new CustomLoadControl.EventListener() {

@Override
   public void onBufferedDurationSample(long bufferedDurationUs) {
           long bufferedDurationMs = bufferedDurationUs;
   }
}, mUiUpdateHandler), drmSessionManager, extensionRendererMode);

Can anyone please confirm this is the correct way to configure the player? Also has anyone observed this problem and have a fix for this?

Thanks in advance.