I have two thoughts on this, one is that you can look at how Minecraft Forge created their plugin API.
Another way is to have a limited API between your core code and the actual plugins, but, you need to be careful of the platform. For example, if you write the core application in Java or C#, then I can use Aspect Oriented Programming (AOP) to bypass your security and have my code change the behavior of yours.
If you use functional programming (FP) languages, then you can protect more from this type of approach, if you also are not using languages on these platforms, but they are not perfect.
So, there is a trade-off between power and convenience, so how useful do you want your application to be, and how secure?
One possible solution that may work is if you go with something similar to Minecraft, though I doubt they do this, but, give a stub application to the user. They can extend it with plugins, and the interface functions they can modify are in the stub. When the program starts, the plugins are loaded, and the interface may be modified or extended, but, then the core program is pulled down and put into the stub, and then the actual program runs. The core program can be recompiled and manipulated so method names are changed, so reflection is harder to use, but taking this approach, and doing it well, would be hard.
BTW, I like Alex T's response, I just gave different terms to some of his, such as AOP instead of reflection and immutability is part of FP.
You mention jar, which means you are using something that runs on a JVM, so you may want to read up on AspectJ, as it can significantly alter the behavior of applications. You can have private methods, but I can put code that runs instead of yours, or change the parameters or the return value before or after the method is called.