In the simplest case you go to the editor window View=>Editor=>Custom actions, and to add an executable you right click (say) the Install node and add the exe, browsing to it. This might help too, still applies:
https://www.simple-talk.com/dotnet/visual-studio/visual-studio-setup---projects-and-custom-actions/
But you haven't said what you've tried exactly, or where you are trying to add the class. If you're trying to add an installer class to the setup project, that's the wrong place. If, for example, you are installing a service with an installer class, then it's a class you add to the service. Then in the setup project you have a custom action to call that class. In your case, have you added an installer class to your C# project (or whatever it is)?