I used the below code to generate access token and to access Microsoft Graph API using the access token:
For sample, I tried to list the users using Client credential flow:
I created an Azure AD Application and granted API permissions:

using Azure.Identity;
using Azure.Core;
using System;
using System.Net.Http;
using System.Net.Http.Headers;
using System.Text.Json;
using System.Threading.Tasks;
class Program
{
static async Task Main(string[] args)
{
var credential = new ClientSecretCredential(
"TenantID",
"ClientID",
"ClientSecret"
);
var tokenRequestContext = new TokenRequestContext(
new[] { "https://graph.microsoft.com/.default" }
);
var accessToken = (await credential.GetTokenAsync(tokenRequestContext)).Token;
Console.WriteLine(accessToken);
await ListUsersAsync(accessToken);
}
static async Task ListUsersAsync(string accessToken)
{
try
{
using (var httpClient = new HttpClient())
{
httpClient.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", accessToken);
var response = await httpClient.GetAsync("https://graph.microsoft.com/v1.0/users");
if (response.IsSuccessStatusCode)
{
var content = await response.Content.ReadAsStringAsync();
var users = JsonSerializer.Deserialize<GraphUsersResponse>(content);
foreach (var user in users.value)
{
Console.WriteLine($"User: {user.displayName}");
}
}
else
{
Console.WriteLine($"Failed to retrieve users. Status code: {response.StatusCode}");
}
}
}
catch (Exception ex)
{
Console.WriteLine($"An error occurred: {ex.Message}");
}
}
public class GraphUser
{
public string displayName { get; set; }
}
public class GraphUsersResponse
{
public List<GraphUser> value { get; set; }
}
}
Using the generated access token, I am able to list the Azure AD users successfully like below:

I agree with you, access token generated by Microsoft Graph is not meant to be validated as it not generated for API.
Hence when you try to decode the Microsoft Graph access token, Invalid signature error will occur and there is no need of decoding the access token.

I agree with @jdweng, based on your requirement you can choose the authentication flows. Refer this MsDoc.
References:
jwt - Using an Azure AD tenant ID - and a valid token issued for a 'app registration'. The signature verification is is failing - Stack Overflow by me.
List users - Microsoft Graph v1.0