If you use reference tokens, then the token is just a random string and the receiver of the token uses the reference token to ask IdentityServer about the details that this token refer to.
When you use a JWT token, the token itself contains all the details about who issued it, who is it for and additional claims about the owner of the token.
There are various pros/cons with using either token format.
If I am not mistaken, if you want to use AddJwtBearer in your API to handle tokens, then you must use the JWT format. because it does not support reference tokens.
See this page for more details.