Android Paging Library loadAfter() not called

Question

Let me just give a short overview on what I'm trying to do. So basically, I am new to this Paging Library in Android Jetpack and I'm trying to implement a paged list. The backend already did the part of paginating by request (The backend returns 25 items per page as default) so all I need to do is make a request and specify a page I want to display and then the items I needed will then be returned. I was able to display and make the loadInitial() work but after I scroll to the lowest item it won't fire the loadAfter().

My PagedListAdapter and RecyclerView is very straight forward so I don't think posting it here would help. The onCreateViewHolder returns a ViewHolder by Data Binding. The onBindViewHolder just calls a method inside my viewholder and set the items. I'm not overriding the getItem() method. That's about it for my adapter and the DiffCallback.

However, below are my other code snippets. Note that I have modified the variable and method names for privacy.

My Config for the LivePagedListBuilder:

public MyViewModel() {
    PagedList.Config config =
        new PagedList.Config.Builder()
            .setPageSize(1)
            .setPrefetchDistance(1)
            .build();

    items = new LivePagedListBuilder<>(myDataSourceFactory, config).build();
}

public LiveData<PagedList<Item>> getItems() {
    return items;
}

The way I observe the LiveData in my View:

viewModel.getItems().observe(this, items -> {
  adapter.submitList(items);
});

My Data Source: ItemKeyedDataSource

private int defaultPage = 1;

public void loadInitial(@NonNull LoadInitialParams<Integer> params, @NonNull 
 LoadInitialCallback<Item> callback) {

    remoteRepository.getItems(String.valueOf(defaultPage))
    ...
}

public void loadAfter(@NonNull LoadParams<Integer> params, @NonNull LoadCallback<Item> callback) {

    remoteRepository.getItems(String.valueOf(params.key))
    ...
}

public Integer getKey(@NonNull Item item) {
    return defaultPage++;
}