Only run JsonConverter on string tokens

Question

I'm trying to make a "reference" system, where JSON files can depend and reference other files. I know JSON.net has a built-in system like this but it doesn't fit my use case. I implemented it like this:

public class ReferenceJsonConverter : JsonConverter<NamedType>
{
    private readonly ContentLoader _contentLoader;
    
    public ReferenceJsonConverter(ContentLoader contentLoader)
    {
        _contentLoader = contentLoader;
    }

    public override bool CanWrite => false;

    public override void WriteJson(JsonWriter writer, NamedType value, JsonSerializer serializer)
    {
        throw new NotImplementedException();
    }

    public override NamedType ReadJson(JsonReader reader, Type objectType, NamedType existingValue, bool hasExistingValue,
        JsonSerializer serializer)
    {
        switch (reader.TokenType)
        {
            case JsonToken.String:
            {
                string value = reader.Value!.ToString().ToLower();
                return _contentLoader.Load<NamedType>(value);
            }
            default:
            {
                NamedType namedType = (NamedType?) serializer.Deserialize(reader, objectType);
                return namedType;
            }
        }
    }
}

So pretty simple, if the token is a string it gets it from memory instead of deserializing it. The problem is, since the token type is the same as the parent type, this will loop infinitely because it will keep trying to use ReferenceJsonConverter to deserialize non-strings.

I've tried the following:

• Re-implement CanConvert to try and "skip" StartObject tokens (janky, didn't work)
• Have a different JsonConverter do the StartObject, then deserialize using ReferenceJsonConverter on string tokens (loops infinitely because I can't "skip" the converter)
• Re-implement the JObject convert method (wayy too much work, will break on updates, and it's all internal anyways)

For clarity, it would look like this:

{ <--- Serialized NamedType, should use default deserializer
    "Name": "Test", <-- Field of NamedType, should use default deserializer
    "ReferenceOther": "ReferencedType" <--- Referenced NamedType, should use ReferenceJsonConverter
}