After the source code has been compiled, most of the actual names of variables will be stripped out to save space, because the actual machine code of the executable doesn't use the long string names of variables, it uses memory addresses.
So debugging symbols are used to provide information to the debugger which normally wouldn't be present in the final executable. Such as the names you provided in the source code to associate with various memory addresses. So if an error occurs, such as:
int* some_invalid_pointer = NULL;
// Segmentation fault should occur here
int some_variable = *some_invalid_pointer;
If you don't have debug symbols enabled, you may get a message such as Error: Segmentation Fault
because the debugger has no idea what the name of each variable actually is. Debugging symbols give the debugger more information so it could print a message closer to Error: Segmentation Fault on Line 3: "int some variable = *some_invalid_pointer;"
which is much easier to debug.
So it takes longer both because of the larger size of the executable needing to be loaded, and because debugging isn't as simple as running a version of the code compiled in debug mode. Debugging usually consists of running an actual separate Debugger executable, which then loads and runs your code. So when debugging you actually need to load and start running the Debugger, then the Debugger loads your program, which is especially slow because it could be the debugger loading it and not the OS like normal.
(To answer your final question, yes technically debugging info can be left in a release executable, but because of the way executables are usually Demand Paged into memory (You can look it up if you're interested), basically what it means is that info within an executable which is never actually referenced, shouldn't ever be loaded into memory and so it shouldn't have an impact on release runtime performance)