There are two parts to "unloading" an add-in.
Unregister the addin so it is not "connected" to your program (it is not hooked up to any events, holding any resource locks, is not needed by or called by the main program at any time, and is not actively running any code).
Unload the assembly containing the addin code.
The first part is entirely in your control - if your addin subscribes to an event, then when you ask it to shut down it must unsubscribe. If your program creates buttons and menu items for the addin, then it must know about them and be able to remove them when the addin shuts down. etc.
It is up to you to decide whether the addin is responsible for this (in which case a rogue addin could take down the entire application if it doesn't clean up properly), or whether the addin framework must provide mechanisms for connecting to the main application so that it is able to clean up all the links between the app and addin automatically (i.e. run the addin in a safer "sandbox" where it is less able to adversely affect the stability of the application or other addins).
Once the addin is shut down it becomes totally dormant, and then it will be safe to unload it. (However, unless you want to be able to update the addin dll while the program is running, you may find you don't even need to unload the addin - simply disabling it may achieve your needs)