As always, it depends on what you want to protect against.
What will keep a user from forging a full request with an authorized referer/origin, downloading your javascript, and hosting it on their server?
Also note that under some circumstances, referer/origin can be spoofed.For example certain browser extensions (if installed of course) some old versions of Java or Flash plugins, maybe other things as well allow such forgery. As an extreme example, a rouge consumer of your content may ask his visitors to first install a browser extension which "gives access to his content", when in reality it allows him to forge referer/origin.
If you really want to secure your content in the sense that you only want to allow authorized clients to download it, unfortunately the only way to achieve that is authentication of some sort. Now obviously you don't want to authenticate the enduser, but the website that sent him, which is an interesting scenario, and is beyond the scope of this answer.
Besides that, probably the most common and best-practice-like thing you can do is you can provide authorized websites with something similar to an api key, the way Google do it in their services. The key is plaintext in any customer page, but if you have proper monitoring in place, you can revoke abused keys. Note that this is not authentication, anybody can copy the host website key from their page source, but you will have a chance to discover abuse, for example not everybody will install a browser extension or plugin, and you will see mismatching referers on the given api key. In many scenarios, this may be good enough. Note that the difficulty is not the api key part, but effective monitoring.