Problem with Typescript generic conditional types and runtime typing

Question

I am writing a module to display information to the user about a data structure. I am trying to make it as generic as possible, so rather than writing specific code for every possible kind of data structure, I am iterating over the structure to be displayed at runtime, and getting a text label and a formatting function for that field.

To achieve this, I've created the following types:

// base interface for data to be displayed in a panel
interface BaseEntity {
  [key: string]: Formattable;
}
// an example entity whose data we want to display
interface EntityInfo extends BaseEntity {
  [key: string]: Formattable;
  uniqueId: number;
  entityName: string;
  entityPosition: Vector3;
}

// types that we can have formatters for
type Formattable = string | number | Vector3;

// generic type for formatters
type ValueFormatter<T> = (value: T) => string;

// label is the text label to display, formatter is the function to format the value with
type DisplayInfo<T> = { label: string, formatter: ValueFormatter<T extends Formattable ? T : never> };

// a map of fields on a BaseEntity to the label and formatter for each field
type PanelDataDescriptor<T> = Map<string, T extends Formattable ? DisplayInfo<T> : never>;

Here is some sample display code:

 function displayText(id: number, text: string) {
    console.log(`id: ${id} ${text}`);
  }
  function displayEntityInfo(dataDescriptor: PanelDataDescriptor<Formattable>, data: BaseEntity) {
    dataDescriptor.forEach((displayInfo, key: string) => {
      const fieldData: Formattable = data[key];
      displayText(0, displayInfo.label);
      displayText(1, displayInfo.formatter(fieldData));
    });
  }

Now we can set up some structures will describe what we want to display:

const myDescriptor: PanelDataDescriptor<Formattable> = new Map([
  ["uniqueId", { label: "UniqueId", formatter: formatNumber }],
  ["entityName", { label: "Name", formatter: formatString }],
  ["entityPosition", { label: "Scenario Pos", formatter: formatVector }],
]);
const ent: EntityInfo = {
  uniqueId: 0,
  entityName: "Guybrush",
  entityPosition: { x: 0, y: 1, z: 2 }
}
displayEntityInfo(myDescriptor, ent);

My problem is this line in displayEntityInfo():

  displayText(1, displayInfo.formatter(fieldData));

causes this error:

Argument of type 'Formattable' is not assignable to parameter of type 'never'. Type 'string' is not assignable to type 'never'.

Why is the 'value' parameter of formatter() of type 'never'? We have a version of formatter() for each type in Formattable (i.e. string | number | Vector3)

Even though the typescript compiler creates the error, the code actually works: Typescript Playground with all the code