@bearsworth It is easy to host React applications on S3 with a tool called AWS Amplify. One thing that would require some clarification is why you would like to host several sites from the same bucket?
With amplify you can start with:
npx create-react-app myappname
then install amplify and its react tools with:
npm i -s aws-amplify aws-amplify-react
next run(follow prompts):
amplify configure
followed by:
amplify init
Afterward go to the AWS Console and search for Amplify. You will see a screen like this one

Where is says get the most out of the Amplify console complete the second step to link your domain name. Click the Add a custom domain with a free SSL certificate link. It should look like this

You can also connect a branch to your favorite code repository and your off to the races!!!
The downside to your request is each site would be in a separate S3 bucket. Amplify is a serverless platform so zero EC2 instances are needed. Going this route has saved me hundreds of dollars per month in infrastructure costs. The Amplify platform takes the stress out of provisioning instances, Kubernetes clusters, DNS configuration, API development, Authentication flows, and storage retrieval. For the full-stack React developer I feel it is a great resource.
Some features to try are:
Amplify add storage
Amplify add API
Amplify add hosting
Amplify add Auth
happy coding