Google Directory API is not a part of GCP - hence any roles / permissions you assign to your service account will not work.
You have to create a role and assign it to a user in order to be able to work with this API.
Your service account is not a Domain Admin so it doesn't have access. You can however enable domain-wide-delegation and make the service account impersonate domain admin so your requests will be accepted;
This page describes how to allow members and resources to impersonate, or act as, an Identity and Access Management (IAM) service account. It also explains how to see which members are able to impersonate a given IAM service account.
Have a look at this answer which may be usefull to you. One more document that you may find helpful is "Authorising your request".