Nope,
You need a "server" to do push notification. You can use a variety of services to do that, as you mentioned firebase is one of them. You may not need a full on server, but you need some kind of service to do that, you can't simply send notifications between devices straight away:
Some examples are:
AWS Serverless functions
Cloud functions in firebase
If you'd rather use one of these services, then you'll need to configure FCM in your app, you can check how to do that in the docs. And some nice sample codes here :)