Optimization and micro-optimization thoughts:
Decrease Image FileSize
This is probably the most important factor to really decrease the loading time (less to download -> faster to finish)
- use some image optimization service like SmushIt or some local software like IrfanView
- find out which is the best format for converting your images
- try to lose some quality, you'd be surprised to see what 1% does sometimes with little to unpercievable loss
- strip image MetaData/EXIF if you don't need it
WebServer & Hardware
Access times are another important factor
- Apache is great but others might perform better for serving image files or small files (eg. Nginx/lightHTTPd)
- tweak the webserver's config to suit your needs
- pick a very fast storage (use RAM if possible)
Webpage
- use a NeverEndingPage/load-on-scroll or "Click for more"
- paginate your image sets (maybe by importance)
- make your image containers and links shorter, strip the extension because browsers will get the filetype from the file's header, rename your files as a base(62) counter vs. the regular 1.jpg...20000.jpg (eg. <img src=Q8U>)