This is the solution that I found after trial and error
_mockObject
.Expects.One.Method(m =>
m.DeletedVideos(-1, -1, out ignoredValue)) //values are ignored
.With(0, 20, Is.Out) //set the values manually
.Will(new SetIndexedParameterAction(2, 100) , Return.Value(deletedVideos));
When this method is invoked from my controller, NMock3 will pass 0, 20 as the first two parameters (Int32), and it will return the IEnumerable with 3rd Int as 100.
To use SetIndexedParameterAction you need to add the namespace "NMock.Actions".
new SetIndexedParameterAction (2, 100) where 2 is the index of the parameter and 100 is the value to be returned.
Return.Value (deletedVideos) is the value to be returned by the method.