Along with these solutions using picasa is an alternative too... and .nomedia or .foldername suggestions are generally useful but...
Including accepted answer none of the solutions solve album covers problem.
A large number of android users are storing album covers inside the album folder and .dirname or .nomedia solutions are hiding all from both gallery and music player. The required functionality is hiding from gallery and showing in music player (else there is no reason for not deleting cover pics if we will not see them right? )
Here is the solution ;
Rename all album cover picture files to albumart.jpg
(or what picture you desire to hide from gallery)
This way gallery will not include them and music player will still use :)
Alternatively for any folder if you place a folder.jpg file in it that file will be ignored by gallery application.
As you can see we can only exclude max 2 files (pictures) from gallery application, not more :(
After you do this changes no need to clean caches of gallery and media scanner and reboot (still an option) just run an application that triggers scanning of media on every run. I prefer vaulty it refreshes media gallery on every run on photos, but there are lots of apps, maybe you already have one installed , just use it no need to install an application for just this media scan deal.
Knowledge source : http://code.google.com/p/android/issues/detail?id=35879