Differentiate conversion operator template for reference and non-reference types

Question

I have a struct InferValueType which shall be implicitly convertible to any other type including reference types. I accomplish this using the following code.

struct InferValueType {
    // ...

    template <typename Type>
    operator Type() const && {
        // ...
    }

    // We need this special version to allow conversions to reference types
    template <typename Type>
    operator Type&() const & {
        // ...
    }
};

InferValueType is never instantiated by the the user, instead I provide it as the return value of a function.

const InferValueType read();

The combination of InferValueType and read allows the user to write lazy code like this:

int i = read();
MyType t1 = read();
MyType& t2 = read();
const MyType& t3 = read();

I have tested this against GCC 4.8, 4.9, 5, 6 and Clang 3.7 and 3.8. Only GCC 4.8 complains about an ambiguity.

conversion from ‘const InferValueType’ to ‘int’ is ambiguous
conversion from ‘const InferValueType’ to ‘MyType’ is ambiguous

I assume this is because int and MyType can be constructed using const int& and const MyType& respectively.

Question: Is this a peculiarity of GCC before 4.9.2 or is it actually an ambiguity under C++11?

Complete example is here.