You are not supposed to pass a literal : to a function. The colon's use for indexing only works on variables directly (see here and here). The colon is no real object and has no type. Were you supposed to use it as a function argument, it would need to be a typed object. Of course, it may be confusing that function calls and variable indexing have the same syntax. But if you ask for a legal syntax for using the colon as function argument, there is none.
That being said, it nevertheless works in some cases, as you observed. That's due to some preprocessing step MATLAB takes and only MathWorks has a handle on. It almost seems that MATLAB calls the function via subsref-like prepocessing when you give it three or more arguments (or two arguments, both being colons), but not when you give it less than that. Go figure. MathWorks will avoid giving out any decisive explanation for that. I suspect, MATLAB uses stringified colons internally after applying subsref, because that's what you see when you receive literal colons in your function, and indexing operations seems to work like that throughout. E.g. try >> m(3, ':');
My proposal for your use case is essentially this answer (also referenced in the comments to your question) but hides the indexing in a function named paren, as you suggested. Also, it uses default parenthesis syntax instead of calling subsref, but it's the same anyway. Use stringified colons!
function result = paren(variable, varargin)
result = variable(varargin{:});
Then call something like:
>> paren(magic(5), 3, ':');
In summary, you should not count on using a literal : as function argument in MATLAB, even though it may work for special cases. Use a string colon ':'.
Side note
You can use subsref for calling functions:
>> subsref(@magic, substruct('()', {3}))
In this way you can chain function call and referencing:
>> subsref(subsref(@magic, substruct('()', {3})), substruct('()', {':'}))
But this really is just the same as using a temporary variable. The inner subsref is evaluated first, and its result passed as input argument into the outer subsref call.
Even with subsref's chaining mechanism, you can not force MATLAB to accept two succeeding pairs of parentheses, like magic(3)(:).
>> magic(3)(:)
??? Error: ()-indexing must appear last in an index expression.
>> subs(1) = substruct('()', {3});
>> subs(2) = substruct('()', {':'});
>> subsref(str2func('magic'), subs)
??? Error using ==> subsref
Only a dot field name can follow ()'s.