Actionbar(toolbar) of fragment in ViewPager not showing

Question

I am trying to inflate Fragments inside ViewPager. When first it inflate it's calling onCreateOptionMenu() but not inflating the toolbar.

Once the ViewPager hit setoffscreenpagelimit number of fragment it started showing up the toolbar in Fragment.

@Override
public void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);
    setHasOptionsMenu(true);
}

@Override
public void onViewCreated(LayoutInflater inflater, ViewGroup container, Bundle savedInstanceState) {
((AppCompatActivity) getActivity()).setSupportActionBar(toolbar);
((AppCompatActivity) getActivity()).getSupportActionBar().setHomeButtonEnabled(true);
((AppCompatActivity) getActivity()).getSupportActionBar().setDisplayHomeAsUpEnabled(true);
}

@Override
public void onCreateOptionsMenu(Menu menu, MenuInflater inflater) {
    inflater.inflate(R.menu.menu, menu);
    super.onCreateOptionsMenu(menu, inflater);
}

Activity.java

@Override
protected void onCreate(Bundle savedInstanceState) { 
            mViewPager = (ViewPager) findViewById(R.id.viewPager);
            mViewPager.setOffscreenPageLimit(4);
            mPagerAdapter = new PagerAdapter(getFragmentManager());
            mViewPager.setAdapter(mPagerAdapter);
}

PagerAdapter.java :

public class PagerAdapter extends FragmentStatePagerAdapter {

Cursor mCursor;

public PagerAdapter(FragmentManager fm) {
    super(fm);
    //cursor initialization
}

@Override
public Fragment getItem(int position) {

    Fragment fragment = MyFragment.newInstance(position);
    return fragment;
}

@Override
public int getCount() {
    if (mCursor == null)
        return 0;
    else
        return mCursor.getCount();
}

public void swapCursor(Cursor cursor){
    mCursor = cursor;
    notifyDataSetChanged();
}

}

Tried workaround from https://code.google.com/p/android/issues/detail?id=29472 & OptionsMenu of Fragments in Viewpager showing each other's Buttons

Still not able to figure out why it's not inflating for first 4 fragments. Can anyone help with this?

Thanks!