Semantic versioning is intended to remove the arbitrariness that can be seen when someone decides to select a versioning scheme for their project. To do that, it needs rules, and a public API is a rule that SemVer chose to use. If you are building a personal project, you don't need to follow SemVer, or follow it strictly. You can, for example, choose to loosely interpret is as
- MAJOR: Big new feature or major refactor
- MINOR: New feature which does not impact the rest of the code much
- PATCH: Small bug fix
But the vagueness of this loose interpretation opens you up to arbitrariness again. That might not matter to you, or the people you foresee who will be using your software.
The larger your project is, the more the details of your versioning scheme matters. As someone who has worked in a third level support for a large IT company (which licenses APIs to customers) for quite some time, I have seen the "is it a bug or is it a feature" debate many times. SemVer intends to make such distinctions easier.
A public API can, of course, be a REST API, or the public interface of a software library. The public/private distinction is important, because one should have the freedom to change the private code without it adversely affecting other people. (If someone accesses your private code through, say, reflection, and you make a change which breaks their code, that is their loss.)
But a public API can even be something like command line switches. Think of POSIX compliant CLI tools. These tools are standalone applications. But they are used in shell scripts, so the input they accept, and the output they produce, can matter. The GNU project may choose to reimplement a POSIX standard tool, and include its own features, but in order for a plethora of shell scripts to continue working across different systems, they need to maintain the behaviour of the existing switches for that application. I have seen people having to build wrappers around applications because the output of the version command changes, and they had scripts relying on the output of the version command to be in a certain form. Should the output of the version command be part of the public API, or is what those people using it did a hack? The answer is that it is up to you and what guarantees you make to the people using your software. You might not be able to imagine all use cases. Declaring the intended use of your software creates a contract with your users, which is what a public API is.
SemVer, thus, makes it easier for your uses to know what they are getting when upgrading. Did only the patch level change? Yes, so better install it quick to get the latest patch fix. Did the major version change? Better run a full, potentially expensive, regression test suite to see if my application will still work after the upgrade.