Swashbuckle SwaggerGen ignores nullable annotations on referenced objects

Question

We are using Swashbuckle SwaggerGen version 5.6.3.0 to generate Swagger API documentation for our Rest API. Everything works fine, except for the fact that the generator ignores nullable annotations on custom object types.

Consider the following code:

public class Vector3f
{
    public float X { get; set; }
    public float Y { get; set; }
    public float Z { get; set; }
}

public class MyClass
{
    [JsonProperty(Required = Required.Always)]
    public string NameAlways { get; set; }

    [JsonProperty(Required = Required.AllowNull)]
    public string NameAllowNull { get; set; }

    [JsonProperty(Required = Required.DisallowNull)]
    public string NameDisallowNull { get; set; }

    [JsonProperty(Required = Required.Default)]
    public string NameDefault { get; set; }


    [JsonProperty(Required = Required.Always)]
    public Vector3f VectorAlways { get; set; }

    [JsonProperty(Required = Required.AllowNull)]
    public Vector3f VectorAllowNull { get; set; }

    [JsonProperty(Required = Required.DisallowNull)]
    public Vector3f VectorDisallowNull { get; set; }

    [JsonProperty(Required = Required.Default)]
    public Vector3f VectorDefault { get; set; }
}

With string, everything works as expected, NameAlways and NameAllowNull are marked as "required", and NameAllowNull and NameDefault are marked as "nullable". However with Vector3f only "required" works, and nothing is marked as nullable!

Here is the relevant part of the generated JSON:

"MyClass": {
  "required": [
    "nameAllowNull",
    "nameAlways",
    "vectorAllowNull",
    "vectorAlways"
  ],
  "type": "object",
  "properties": {
    "nameAlways": {
      "type": "string"
    },
    "nameAllowNull": {
      "type": "string",
      "nullable": true
    },
    "nameDisallowNull": {
      "type": "string"
    },
    "nameDefault": {
      "type": "string",
      "nullable": true
    },
    "vectorAlways": {
      "$ref": "#/components/schemas/Vector3f"
    },
    "vectorAllowNull": {
      "$ref": "#/components/schemas/Vector3f"
    },
    "vectorDisallowNull": {
      "$ref": "#/components/schemas/Vector3f"
    },
    "vectorDefault": {
      "$ref": "#/components/schemas/Vector3f"
    }
  }
}

As you can see, the "nullable" annotation isn't generated. I have tried writing a custom ISchemaFilter, similar to how it's described here, but with limited success. Even if I manually set the nullable flag by setting property.Value.Nullable = true, the generator still ignores it and doesn't insert the nullable annotation when there is a $ref in the schema body.

Workaround

I did, however, manage to add a new property called x-nullable, which correctly listed nullable properties based on the annotation. It worked OK, but it would obviously be better if the schema itself contained the "nullable" annotation (like string does for example) instead of it being in a separate property.

Is there a way to solve this? Does Swashbuckle version 6 change anything?

EDIT:

I tried to use UseAllOfToExtendReferenceSchemas as described here, and while it works well for the generated JSON, it is kind of unusable for the Web Swagger viewer, because now it deletes the type of the object (Vector3f), and only display it's properties.

Differences:

It's also important to note that the type is gone only in the Web viewer, and is still present in the JSON with the UseAllOfToExtendReferenceSchemas setting, but without it, the nullable annotation is gone from both. Could the web viewer be somehow configured to display the type as well?