This is a follow up from Andy Burn's answer:
From PowerShell I initially tried the following:
$web = Get-SPWeb http://localhost
$web.EnsureUser("domain\test group")
This didn't work, which was puzzling.
Some more experimentation and I found the following did work (Power Users is a built in group):
$web.EnsureUser("builtin\power users")
$web.EnsureUser("power users")
$web.EnsureUser("test group")
I then noticed that I had a different value for the Name (aliased as DisplayName in PowerShell) -- it turns out that I had used different values in Active Directory for 'Group name' and 'Group name (pre-Windows 2000)'.
The group name worked by itself, but with the domain prefix I needed to use the pre-Windows 2000 name.
With this, I was able to get the following to work:
$web.EnsureUser("domain\pre2000 test group")
So, if you are still having issues check for consistency between the two group names in AD.