So I ended up implementing something like what @Jahnold recommended. (I'll post the diagram in the link provided for an idea stackoverflow.com/a/41966497/568898 )
Hannes Dorfmann (the guy who created/manages the famous Mosby MVP library : Github link ) also pointed me in this direction.

Implementation
I have a presenter for the main activity and multiple fragments that can be used in that activity. Each fragment has its own presenter. Then I use a repository (search for repository pattern) which is basically stores the models and the business logic. For my use case, I keep this repository as a singleton. The repository provides data in three forms, from an online api, an sqllite database, or a cache stored in memory (basically an arraylist of items). I also have some currentitem int indexes and stuff in this repository, that get updated based on the current state.
Hence the data, the state and the business logic are stored in this shared repository. The presenters and the views are pretty dumb. I don't have much business logic (application specific logic) in presenters. They simply have the logic associated with how the data has to be displayed (view specific logic) and preprocessing in logic them.
Example
Whenever the fragment and activity need to talk to each other (via presenters) when the user clicks a button in a child fragment, the fragment asks its presenter to handleClick, the presenters updates the repository's currentItemSelected data (or something else) and asks the fragment to fire an event (say onbuttonclick) to an interface listener which the activity implements. When the activity gets the event, it asks it's own presenter to handle it and in turn the activity presenter looks for an update in the repository to get the new currentItemSelected.
Extra Info (advanced version):
You can also follow Clean architecture which is sort of a more advanced version of MVP architecture. MVP just deals with the architecture of the views, where as Clean architecture also deals with business logic and data architecture, MVP is just a small part of clean arch which is used to handle the views. Using this, you can break down the mega repo in my case into even further use-cases (or interactors) that handle a specific business logic use-case, and the repository just provides data. So the logic flow is now view-->presenter-->interactor-->repo and back.