What might be happening is that the fragment is recreated when you go away from it. I mean, the fragment is created the first time when you launch the app or when you swipe near/into that fragment (this depends on the fragment position and on which is the first fragment showed). When you swipe/navigate away from that fragment as much as you can and then come back to that fragment the fragment is recreated (onCreateView method is called again).
Check if that's what is happening with...
Log.d("Fragment X", "onCreateView");
...inside onCreateView method in the fragment.
First of all, be sure that you are not creating a new fragment instance each time the getItem method is called in the viewpager adapter. As far as I know you should create a new fragment instance only in the case that you don't have one, in other case you should just return the fragment already created.
Second: by default, the viewpager sets to 1 the number of pages number of pages that should be retained to either side of the current page in the view hierarchy in an idle state. Pages beyond this limit will be recreated from the adapter when needed.
That's the reason why when you swipe more than 1 pages away from a particular fragment, it gets recreated the next time you navigate near/into it.
To solve your problem there are two different approaches:
1) Set the amount of pages to be retained to a higher number with setOffscreenPageLimit (int limit) method.
This approach really depends on how many tabs you have. If you have a few of them (3/4) is a good approach from my point. Realize that this method sets the number of pages that should be retained TO EITHER SIDE OF THE CURRENT PAGE. So if you have 3 pages, setting this to 2 is enough because you'll never be more than 2 pages away. If you have 4 pages, 3 is enough and so on.
2) Save the state of the fragment. Take a look at this thread -> ViewPager and fragments — what's the right way to store fragment's state?
I'm not an android expert and maybe I'm not completely right but I think this is close to the correct approach.