I think this question would be more suitable for SuperUser (well, unless you want to do it via a program :) ).
Anyway, here's a way of doing things from console (Cmd). I've tried it 1 or 2 years ago, I just tried it now, so it works regardless of Win (10) version.
Start the process from scratch:
Open a Command Prompt window. Create a new file that the OS doesn't know anything about. I chose the extension .zzz:
e:\Work\Dev\StackOverflow\q052008516>ver
Microsoft Windows [Version 10.0.17134.228]
e:\Work\Dev\StackOverflow\q052008516>dir /b
e:\Work\Dev\StackOverflow\q052008516>:: Create a dummy .zzz file
e:\Work\Dev\StackOverflow\q052008516>echo Some dummy text>file.zzz
e:\Work\Dev\StackOverflow\q052008516>dir /b
file.zzz
Try opening the file (DblClick) from a file browser (it's not relevant, but I use Total Commander), or by typing its name in Cmd. That will yield the dreaded dialog:

Create a new file type and associate our extension with it. [MS.Learn]: assoc utility is used to do the job. First, check if such association doesn't already exist:
e:\Work\Dev\StackOverflow\q052008516>:: No output means no association
e:\Work\Dev\StackOverflow\q052008516>assoc | findstr ".zzz"
e:\Work\Dev\StackOverflow\q052008516>:: Same command for a different extension
e:\Work\Dev\StackOverflow\q052008516>assoc | findstr ".txt"
.dic=txtfile
.exc=txtfile
.log=txtfile
.scp=txtfile
.txt=txtfile
.wtx=txtfile
e:\Work\Dev\StackOverflow\q052008516>:: Create a new FileType (ZZZFile) and associate our extension with it
e:\Work\Dev\StackOverflow\q052008516>assoc .zzz=ZZZFile
.zzz=ZZZFile
e:\Work\Dev\StackOverflow\q052008516>assoc | findstr ".zzz"
.zzz=ZZZFile
No change when trying to open the file.
Associate the file type (ZZZFile, from previous step) with a command. Use the [MS.Learn]: ftype tool for the task. Again, check if the file type is not already associated (this only makes sense if the file type existed before previous step):
e:\Work\Dev\StackOverflow\q052008516>:: As usual, no output means no association
e:\Work\Dev\StackOverflow\q052008516>ftype | findstr ZZZFile
e:\Work\Dev\StackOverflow\q052008516>:: Same thing for txtfile
e:\Work\Dev\StackOverflow\q052008516>ftype | findstr txtfile
txtfile=%SystemRoot%\system32\NOTEPAD.EXE %1
e:\Work\Dev\StackOverflow\q052008516>:: Associate ZZZFile with notepad
e:\Work\Dev\StackOverflow\q052008516>ftype ZZZFile=%SystemRoot%\system32\notepad.exe %1
ZZZFile=C:\WINDOWS\system32\notepad.exe %1
e:\Work\Dev\StackOverflow\q052008516>ftype | findstr ZZZFile
ZZZFile=C:\WINDOWS\system32\notepad.exe %1
Try opening the file again (from Cmd), and voilà:

Summary:
In order to open with Notepad files having .zzz extension, there are only 2 commands that need to be remembered from this whole (and pretty long) answer:
Notes:
My user has (super) administrative privileges, but I guess they shouldn't impact differently depending where the action is performed from (Cmd or UI (if possible)), in other words users that don't have the required privileges, won't be able to do it, no matter what they would try
Apparently, there is a (pretty dark) nebula on this topic, that my knowledge wasn't yet able to "decipher". In my example, I constantly compare the .zzz results to .txt. Yet a big surprise: Notepad++ and not Notepad is used to open txtfile (.txt only), in spite of the above output
Update #0
I did a little more digging on the .txt mystery. Facts:
FType shows Notepad as opening program
It is actually opened by Notepad++ (in Cmd and PS)
In Choose default apps by file type, Notepad++ is shown
So apparently, it's more than meets the eye (over the years, I got used to MS's way of doing things which in some cases seems to be (but maybe it's me who didn't have all the pieces) illogical).
I've also found out many resources like:
I couldn't find anywhere a clear algorithm of how an executable is chosen to run a file with a certain extension. I can think that the 2 keys above are queried, but I'm 100% sure there's more. Not to mention that I've ran into an even stranger problem (for a regular user on my Win 10), for .py files:
FType (and Assoc) reported everything as above
In Choose default apps by file type, Python was shown (just like in my Super Admin user's case)
Attempting to run the file from Cmd, yielded the dialog at the beginning
It worked from PS
Sadly, I selected Python from the dialog, before taking a look at the registry keys (and now it works), so I can't do any more debugging (and also, switching users is annoying).
Might be related: