Unless your old database was using a non-standard block size (which is unusual and not recommended), you must be falling victim to this change in PostgreSQL v12:
In new btree indexes, the maximum index entry length is reduced by eight bytes, to improve handling of duplicate entries (Peter Geoghegan)
This means that a REINDEX operation on an index pg_upgrade'd from a previous release could potentially fail.
You will have to modify the data in your table to be a bit shorter.
An alternative, but not commendable solution is to use pg_upgrade
rather than dump and restore. That would leave the index as it is. But then you can never REINDEX
or dump and restore that database.