Yes you can do that using a beanshell in TestNG.
You create a suite xml file and define a beanshell as below :
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Suite" parallel="false">
<test name="Test">
<method-selectors>
<method-selector>
<script language="beanshell">
<![CDATA[whatGroup = System.getProperty("groupToRun");
!Arrays.asList(testngMethod.getGroups()).contains(whatGroup);
]]>
</script>
</method-selector>
</method-selectors>
<classes>
<class name="organized.chaos.GroupsPlayGround" />
</classes>
</test> <!-- Test -->
</suite> <!-- Suite -->
This would basically cause TestNG to look for all tests that don't belong to the group which was passed via the JVM argument groupToRun
I have explained about this in my blog here.
You can also find more information about this on the Official TestNG documentation here.