You can restrict the access for a service account to a specific bucket using Cloud IAM.
This is the gsutil command you can use:
gsutil iam ch serviceAccount:my-service-account@project.iam.gserviceaccount.com:objectAdmin gs://my-project/my-bucket
for custom roles use
gsutil iam ch serviceAccount:my-service-account@project.iam.gserviceaccount.com:projects/{your_project_id}/roles/{your_custom_role_id} gs://my-bucket
To remove a service account from all roles on a bucket:
gsutil iam ch -d serviceAccount:my-service-account@project.iam.gserviceaccount.com gs://my-project/my-bucket
Or you can control access to buckets and objects using ACLs.
For example grant the service account WRITE (R: READ,W: WRITE,O: OWNER) access to the bucket:
gsutil acl ch -u my-service-account@project.iam.gserviceaccount.com:W gs://my-project/my-bucket
To remove access of service account from the bucket:
gsutil acl ch -d my-service-account@project.iam.gserviceaccount.com gs://my-project/my-bucket
I would suggest to remove the access of the service account from the buckets. Then grant access to a specific bucket.