With Git you can clone a repository without checking out a branch into the work directory git clone --no-checkout. But Git will always clone the entire repository. The only limit you can specify is the depth, i.e.
After that you can checkout only those directories you are interested in (see How do I clone a subdirectory only of a Git repository?, please pay attention to @StijndeWitt's comment). But if cloning is what takes long, that won't help you much.
EGit, however, does not (directly) support cloning without checking out the work directory, neither does it support sparse checkouts
I would probably restructure the repository, but how of course dependson what those files are for, when they are used, if/how the number can be reduced, etc.