You can do this:
$("li:not(.menu)").whatever();
That's not the fastest way necessarily; it may be faster to do this:
$("li").filter(function() { return !$(this).hasClass("menu"); }).whatever()
edit if you want to operate on <li>
elements that have no class, then just check the .className
property:
$("li").filter(function() { return !$(this).prop("className"); }).whatever()
However, I would suggest that that's not a good coding pattern. It's fragile because it relies on any future changes to your page for purposes completely unrelated to what you're doing now not involving the addition of a class. You'd be better off explicitly checking for specific classes that you're not interested in.
Like, maybe 3 months from now, somebody decides that all the list items that are about penguins be made black and white. You then add the class "penguin" to all those <li>
elements. If you don't remember this change you're making now, all of a sudden that functionality will be broken.