You've essentially discovered why monad transformers are inwards instead of outwards (nit: I'm pretty sure I'm not using correct terminology here, but you get what I mean - feel free to correct me here). It's much easier (easy == less constraints) to have the monad you know about be in the data position (inside), rather than the container position (outside). Intuitively, you can see why - after all, the monadic nature demands the container to be the same. You can do whatever you want with the data, but you must not change the type of the container during binding.
Of course, it's easier to realize all this by actually trying to implement the absurd. Which is what this is about. The final step you'll be stuck on while implementing >>=/bind for MaybeOT is this-
m a >>= ??? . runMaybeOT . f
Where, m is a Monad, m a :: m a, f :: a -> MaybeOT m a, and runMaybeOT :: MaybeOT m a -> Maybe (m a) (and also ??? :: ???, hehe)
Now, you must obtain a monad with the container type m to successfully bind on m a. But alas, you cannot get the m a out of Maybe (m a)! What happens when it's Nothing? Your primary tool in implementing monad transformers is knowledge about the specific monad you're implementing the transformer for. And your knowledge is telling you that this is absurd.
In contrast, the final step in implementing >>= for MaybeT goes well smoothly. For completeness, here's MaybeT-
newtype MaybeT m a = MaybeT { runMaybeT :: m (Maybe a) }
Here, you'll be binding on the type m (Maybe a). During binding, you must return the type m b, for whatever b. You're given a function, f, of type a -> MaybeT m b, you can easily get m (Maybe b) from MaybeT m b.
Aha! Now you can see the solution. The function you're given returns the same outer monad as the one you're binding on. That is precisely what you need. In the case of MaybeOT, you were stuck with binding on the monad m, with a function that doesn't return a value with outer monad m!
And that is the crucial realization - the function you're given must be able to give you a value with the same outer monad. This is why MaybeT (and other transformers too) keeps the unknown monad outwards - because when you implement >>= for MaybeT, you know the binding function will be required to construct that outer monad.
It is helpful, at least intuitively, to note that the problem you get stuck on here - is the very same problem you'll face when implementing monadic composition. That is, >>= for nested monads. Monads don't compose! That's why you need monad transformers!
If you break this problem down, you'll notice that if you just had a function that could swap monads, to get Maybe (m a) from m (Maybe a) and vice versa - all would be well. Monads would compose, monad transformers could look like however you please, and in fact, monad transformers would essentially serve no purpose. This detail is noticed in the answer linked above. All you need is-
swap :: (Monad m, Monad n) => m (n a) -> n (m a)
This exists. They're called traversable monads, and indeed if you merely add the Traversable constraint to your Monad implementation for MaybeOT, you strike gold-
instance (Traversable m, Monad m) => Monad (MaybeOT m) where
return = pure
MaybeOT Nothing >>= _ = MaybeOT Nothing
MaybeOT (Just ma) >>= f = MaybeOT $ sequence $ ma >>= sequence . runMaybeOT . f
I assume it's law-conforming, though I'd have to check.
In another note, it is totally possible to make MaybeOT a law conforming applicative. After all, the problem you face while implementing Monad is simply non existent here. Applicatives do compose.
instance Applicative f => Applicative (MaybeT f) where
pure = MaybeT . pure . Just
MaybeT f <*> MaybeT a = MaybeT $ liftA2 (<*>) f a
(Or, perhaps more simply)
instance Applicative f => Applicative (MaybeT f) where
pure = MaybeT . pure . Just
MaybeT f <*> MaybeT a = MaybeT $ (<*>) <$> f <*> a
Which should be law conforming, as far as I can tell.
