Connection refused does NOT mean the server closed the socket (after you connected to it), it means there is no server listening on the socket. So, first, your server crashes, THEN, clients get a connection refused. (Except if the backlog becomes full, if the backlog size you pass to listen() is too small, or your server doesn't accept() connections fast enough, but i've never seen that in real life situations).
Normally, i'd blame it on some firewall, but, since, in your other post, you say it happens only intermittently, i guess what happens is: your mysql server crashes, safe_mysqld restarts it, which takes a few seconds, and during that startup phase, you get your connection refused errors because there's no server there to listen to you. Try setting up a cron job that does "ps -ef | grep mysqld" every minute, check if the process id changes from time to time, and try to match these changes to your connection refused error messages.
If mysqld does NOT restart, i.e. the process ID remains the same, try something like "truss -v listen,accept -p ". This should give you some output every time a client connects - maybe you can match "lots of things going on in truss" to "i get connection refused now". But, don't do this on a heavily loaded production system, or truss will a) drown you in output and b) slow down your server significantly.