Unfortunately playing an rtsp via your web browser is not that straight forward anymore.
NPAPI Plugins are no longer supported
This means in the end the VLC plugin for browsers is no longer supported. Firefox 52+ and other browser don't support it anymore. See this announcement, basically stating that:
"installed NPAPI plugins are no longer supported in Firefox, except
for Adobe Flash"
This also has been discussed on the videolan Forum, with no current solution.
Use an old Mozilla Firefox ESR browser
Still supports NPAPI plugins. However only until June 2018. Thus not really a future proof solution. Details on Firefox ESR can be found here.
A possible solution
You could use HTML 5 with some other coding technology to transform your RTSP stream into let's say a segmented stream and deliver it via HTML 5 video tag. There is alot of solutions out there; look into posts from stackoverflow such as:
Hope this points you in the right direction.