Solution 1:memberOf (in AD) is stored as a list of distinguishedNames. Your filter needs to be something like:
(&(objectCategory=user)(memberOf=cn=MyCustomGroup,ou=ouOfGroup,dc=subdomain,dc=domain,dc=com))
If you don't yet have the distinguished name, you can search for it with:
(&(objectCategory=group)(cn=myCustomGroup))
Example:
filter = "(&(objectClass=user)(sAMAccountName=#{username})(memberof=CN=group-name,OU=Linux Groups,OU=Linux))"
This example list the all the groups the user is part of.
More details refer this thread
Solution 2: Example using a modern ldapsearch command line tool:
ldapsearch --port 1389 --baseDn 'ou=people,dc=example,dc=com' \
--sizeLimit 3 --searchScope one --bindDn 'cn=directory manager' \
--bindPasswordFile ~/.pwdFile '(uid=user.0)' isMemberOf
dn: uid=user.0,ou=people,dc=example,dc=com
isMemberOf: cn=Dynamic Home Directories,ou=groups,dc=example,dc=com
isMemberOf: cn=bellevue,ou=groups,dc=example,dc=com
isMemberOf: cn=shadow entries,ou=groups,dc=example,dc=com
isMemberOf: cn=persons,ou=groups,dc=example,dc=com
This search response indicated that user.0 is a member of the listed groups.
The above is a general explanation of one way to deal with group membership from an
LDAP perspective.
Also refer this link for more details.