There are many ways to create a React web app. And there are many types of them as well.
Client-Side Rendering
Noticeable toolchain: Create React App
Everything is done on the client side. You send a blank HTML file and various JS bundles to the browser. And the browser parses the bundle and renders contents on the HTML and presents it to the users.
It takes less effort to set up so it best suits small projects.
This is properly what you were referring to when you said: "just using React".
Server-Side Rendering
Noticeable Toolchain: Next.js
Most of the works is done on the server-side. When a user requests to view a page, the server generates the need HTML file dynamically with static contents in it. Then, it sends the file to the user together with JS bundles for interactive content. The browser then attaches those JS to the HTML and present it to the users.
It requires far more effort to set up compared to Create React App. It best suits mid to large projects.
Static Site Generator (Prerender)
Noticeable Toolchain: Gatsby
Similar to Next.js, but instead of generating the HTML dynamically. It generates ALL OF THEM at build time and just sends it to users upon being requested.
This property has the best performance overall. But it can become a nightmare when the site is growing bigger and have hundreds of pages. Building a large, static Gatsby site takes ages to complete.
p.s. Next.js is also capable of generating static sites, but why don't you pick the right tool which is designed and optimized for generating a static site in the first place?