First part will be assumptions but it looks that Azure DevOps takes snapshot for build execution. This is why you see this for specific run but not in general. This is probably done in that way because groups may be shared accros build definitions and thus one piepline can may a change for another one.
If this is what you want to do you can use REST API or CLI.
For CLI you have this command
az pipelines variable-group variable update --group-id
--name
[--detect {false, true}]
[--new-name]
[--org]
[--project]
[--prompt-value {false, true}]
[--secret {false, true}]
[--value]
I don't recall if there is devops extension already installed on agent, so you may be interested also about installing it.
Another option is to use REST API and this endpoint:
PUT https://dev.azure.com/{organization}/{project}/_apis/distributedtask/variablegroups/{groupId}?api-version=5.1-preview.1
Example how to call REST API from a pipeline you can find here: How to send post build message in azure DevOps YAML pipeline?
EDIT
This is how you can change variable in variable group:
variables:
orgName: 'thecodemanual'
variableGroupId: 3
steps:
- pwsh: |
$url = "https://dev.azure.com/$(orgName)/$(System.TeamProject)/_apis/distributedtask/variablegroups/$(variableGroupId)?api-version=5.1-preview.1"
$variableGroup = Invoke-RestMethod -Uri $url -Headers @{Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"}
Write-Host "Pipeline = $($variableGroup | ConvertTo-Json -Depth 100)"
$variableGroup.variables.name.value = "fromPipeline"
$json = $variableGroup | ConvertTo-Json -Depth 100
Invoke-RestMethod -Method PUT -Uri $url -ContentType "application/json" -Headers @{Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"} -Body $json
name: initial
env:
SYSTEM_ACCESSTOKEN: $(system.accesstoken)
And if you get this error:
{"$id":"1","customProperties":{"Descriptor":"Microsoft.TeamFoundation.ServiceIdentity;a960b165-7bec-46ad-9fa3-a4a754989d4e:Build:4fa6b279-3db9-4cb0-aab8-e06c2ad550b2","IdentityDisplayName":"DevOps Manual Build Service (thecodemanual)","Token":"3","RequestedPermissions":2,"NamespaceId":"b7e84409-6553-448a-bbb2-af228e07cbeb"},"innerException":null,"message":"You do not have permissions to perform this operation on the variable group. A variable group Administrator should add you to the Administrator role.","typeName":"Microsoft.VisualStudio.Services.Security.AccessCheckException, Microsoft.VisualStudio.Services.WebApi","typeKey":"AccessCheckException","errorCode":0,"eventId":3000}
Please check this Having no permission for updating Variable Group via Azure DevOps REST API from running Pipeline