I think your question has two parts.
First, JSONP is essentially not about dynamic script tags, rather dynamic script tags are a technique used hand in hand with JSONP.
JSONP is a method which allows site to load content from different domains than the ORIGIN, exploiting the browser's tolerance towards SCRIPT tags with src pointing to external domains. (You should know this by going through the links given in other answers).
Dynamic SCRIPT tags on the other hand provides an Asynchronous nature to any script be it JSONP or otherwise.
Point is, whenever a browser hits a SCRIPT tag on a document, it stops most other activities (rendering DOM specially) until that script is download. This affects users experience on how responsive the site is. Effect of this is even worse if the script is not directly contributing to the primary content of the site (such as Google Ads, Tweets, or Facebook Timeline (Asuming you are not Mark Z. :P), etc)
To avoid this problem, you can inject dynamic SCRIPT tags to the page once it has fully loaded on the browser (i.e ready/loaded event). Then the browser will silently load the new script, but user has the full page (almost) rendered for him giving impression of quick loading. In that sense dynamic scripts can be asynchronous to page loading.
However, in practice most of scripts used in this way are JSONP scripts residing on different domains, although it's not a requirement.
Hope this makes sense.
For TRUE async script loading you should look into HTML5 sync attribute: