If you really need only certain directories, you should declare them as submodule of your main git repo.
That way, those directories are in their own git repo, and you can clone them independently of the parent repo.
Otherwise, what you want is called sparsed checkout (as opposed to shallow clone, which clone part of the history of the all repo), and, as mentioned in "Partial clone with Git and Mercurial", wasn't possible with Git.
Git1.7.0 includes it with git read-tree
(but you still need to clone the full repo).
See "Sparse checkout in Git 1.7.0?" for a script.
"clone parts of a github project" mentions other options.