There is a practical limit of about 5000 custom classes in the current implementation. This is approximate because it depends in part on how complex each custom class is, how many classes and how many classifiers (which contain classes) you have. We have tested correct functioning of the system with more than twice this number, but we think users will probably start experiencing error codes due to timeouts if certain caches are not populated when a request is made around this number. Retrying the request can help to populate the caches.
This is a guideline; we are continuously working on improving Watson Visual Recognition to make it faster and solve bigger problems for our users.
Thanks for your question!