It's advaisable using AWS Systems Manager run command versus scp and ssh commands directly to the instance. Update the CodeBuild project role with the necessary permissions. Store files in a private S3 Bucket. To restrict access leverage IAM or resouse based Policies with conditions.
https://docs.aws.amazon.com/systems-manager/latest/userguide/walkthrough-cli.html
Notes:
There is no way to restrict SG ingress rule to allow CodeBuild servise. It just doesn't make sense. Even if you get a pool of IPs used by CodeBuild service it doesn't look like a secure solution.
VPC hosted CodeBuild instance? Maybe... But still not the most secure and as flexible as ssm connection.
VPC Endpoints for CodeBuild? It works another way around.