Every small file is an extra HTTP request and that costs a lot of performance.
An HTTP request involves a lot of overhead: that's HTTP headers, which today are quite big and they can't be gzipped.
In addition to that, browsers usually have a limit of connections that they'd make to one server, thus blocking requests for further requests until previous are done.
Until HTTP 2.0 arrives, you should definitely concatenate all your javascript into one or two big files in production, absolutely. (For example, Yahoo guidelines clearly stated that back in 2007!)
Of course, you should not be doing this manually.
Requirejs has optimizations for production, you do not need to do any manual labor.
Readability of your code is almost of utmost priority.