Issue occured during extension installation for SSMS
Actually, installing an extension in SSMS is not the same as VS IDE. I suggest you could follow these steps:
1) Right-click on the vsix project-->Properties-->Debug-->and then change Start External program into the path of SSMS installation path like this:
C:\Program Files (x86)\Microsoft SQL Server Management Studio 18\Common7\IDE\Ssms.exe

Besides, under that window, you should remove the command line arguments.

2) create a custom folder called VSIXProject
under C:\Program Files (x86)\Microsoft SQL Server Management Studio 18\Common7\IDE\Extensions
Then, Set Deploy VSIX to SSMS:
Right-click on project-->VSIX
-->check option Copy VSIX content to the following location
and then input C:\Program Files (x86)\Microsoft SQL Server Management Studio 18\Common7\IDE\Extensions\VSIXProject

3) Run VS as administrator and you can debug this project which debug in SSMS and when you finish debugging it. The extension will automatically installed into SSMS.
In addition, you can refer to this issue.