1

I'm new to AKS and the Azure Identity platform. I have an AKS cluster that is using the Azure AD integration. From an Azure VM that has a user assigned managed identity, I'm trying to run a C# console app to authenticate against Azure AD, get the kubeconfig contents and then work with the kubernetes client to perform some list operations. When the code below is run I get an Unauthorized error when attempting to perform the List operation. I've made sure that in the cluster access roles, the user assigned managed identity has the Owner role.

The code does the following:

  • Creates an instance of DefaultAzureCredential with the user managed identity ID
  • Converts the token from DefaultAzureCredential to an instance of Microsoft.Azure.Management.ResourceManager.Fluent.Authentication.AzureCredentials and authenticates
  • Gets the contents of the kubeconfig for the authenticated user
  • Gets the access token from http://169.254.169.254/metadata/identity/oauth2/token
  • Sets the access token on the kubeconfig and creates a new instance of the Kubernetes client
  • Attempt to list the namespaces in the cluster

I've pulled information from this POST as well from this POST.

I'm not sure if the scopes of TokenRequestContext is correct and if the resource parameter of the oauth token request is correct.

string userAssignedClientId = "0f2a4a25-e37f-4aba-942a-5c58f39eb136";
    var credential = new DefaultAzureCredential(new DefaultAzureCredentialOptions { ManagedIdentityClientId = userAssignedClientId });

    var defaultToken = credential.GetToken(new TokenRequestContext(new[] { "https://management.azure.com/.default" })).Token;
    var defaultTokenCredentials = new Microsoft.Rest.TokenCredentials(defaultToken);
    var azureCredentials = new Microsoft.Azure.Management.ResourceManager.Fluent.Authentication.AzureCredentials(defaultTokenCredentials, defaultTokenCredentials, null, AzureEnvironment.AzureGlobalCloud);
    var azure = Microsoft.Azure.Management.Fluent.Azure.Authenticate(azureCredentials).WithSubscription("XXX");

    var kubeConfigBytes = azure.KubernetesClusters.GetUserKubeConfigContents(
        "XXX",
        "XXX"
    );

    var kubeConfigRaw = KubernetesClientConfiguration.LoadKubeConfig(new MemoryStream(kubeConfigBytes));

    var authProvider = kubeConfigRaw.Users.Single().UserCredentials.AuthProvider;
    if (!authProvider.Name.Equals("azure", StringComparison.OrdinalIgnoreCase))
        throw new Exception("Invalid k8s auth provider!");

    var httpClient = new HttpClient();
    var token = string.Empty;
    using (var requestMessage =
                new HttpRequestMessage(HttpMethod.Get, $"http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource={Uri.EscapeUriString("6dae42f8-4368-4678-94ff-3960e28e3630/.default")}&client_id={userAssignedClientId}"))
    {
        requestMessage.Headers.Add("Metadata", "true");

        var response = await httpClient.SendAsync(requestMessage);
        token = await response.Content.ReadAsStringAsync();

        Console.WriteLine(token);
    }

    var tokenNode = JsonNode.Parse(token);

    authProvider.Config["access-token"] = tokenNode["access_token"].GetValue<string>();
    authProvider.Config["expires-on"] = DateTimeOffset.UtcNow.AddSeconds(double.Parse(tokenNode["expires_in"].GetValue<string>())).ToUnixTimeSeconds().ToString();

    var kubeConfig = KubernetesClientConfiguration.BuildConfigFromConfigObject(kubeConfigRaw);
    var kubernetes = new Kubernetes(kubeConfig);

    var namespaces = kubernetes.CoreV1.ListNamespace();
    foreach (var ns in namespaces.Items)
    {
        Console.WriteLine(ns.Metadata.Name);
        var list = kubernetes.CoreV1.ListNamespacedPod(ns.Metadata.Name);
        foreach (var item in list.Items)
        {
            Console.WriteLine(item.Metadata.Name);
        }
    }

Any help is appreciated!

Ching Kuo
  • 21
  • 4

1 Answers1

0

Try using the resource in the token request without /.default.

So it should be:

resource=6dae42f8-4368-4678-94ff-3960e28e3630
Jeremy Caney
  • 7,102
  • 69
  • 48
  • 77
marenzi
  • 1
  • 1
  • I tried removing /.default and still getting a forbidden. – Ching Kuo Jun 15 '22 at 01:44
  • @ChingKuo Are you getting Unauthorised or Forbidden? If it is "forbidden" then your token should be fine, but you are missing some azure permissions. Can you share the exact message you are getting? – marenzi Jun 15 '22 at 07:29
  • The error is: Operation returned an invalid status code 'Forbidden' at k8s.Kubernetes.SendRequestRaw.....This is thrown on the line "var namespaces = kubernetes.CoreV1.ListNamespace();" Wonder if I'm doing this correctly. The VM that I run this on is assigned a User Assigned Managed identity (that then is used in the app), then I assigned the Owner role to this identity in the Cluster's Access Control > Role assignments section. Or is there another way to configure the access control policy? – Ching Kuo Jun 16 '22 at 13:05
  • So aside to the Owner role you assigned to the managed identity there is also additional permissions needed for cluster. The permissions I am referring to are Azure RBAC and/or Kubernetes RBAC. The token you have generated is authenticating you for the cluster, but you also need this managed identity user authorized for specific actions. This you will do with either Azure RBAC (https://learn.microsoft.com/en-us/azure/aks/manage-azure-rbac) or Kubernetes RBAC (https://learn.microsoft.com/en-us/azure/aks/azure-ad-rbac). The error you get is seems to me like an authorization problem. – marenzi Jun 22 '22 at 20:10
  • I suggest that while you figure out these permissions the best way is to use kubectl CLI to execute commands(queries). When you manage to get it to work with the CLI, then it must work from the code as well. – marenzi Jun 22 '22 at 20:14
  • @ChingKuo did you manage to setup the permissions? – marenzi Aug 10 '22 at 13:35