The solution is to start your center circle as big as it needs to be, then scale it down as the reference starting point.
Then, on the hover event you scale up to 1, which will preserve the unpixelated center circle.
Reference: jsFiddle
Note other settings such as positioning are done due to compensate for these changes.
Status Update:
Consider instead of using border-radius to make circle, use ASCII Character of circle or outline
circle:
• ○ ☺ ☻ ☼
Reference: jsFiddle (Note positions are not calibrated.)
The above characters are essentially TEXT, hence use ANY CSS2 or CCS3 text or font property!
As certain characters become really big they do pixelate so use "reverse scale" method for these characters as previously answered but note, at least in Firefox, the transitions become expensive when super large fonts are used. Works best with medium to large fonts.
Tip: This ASCII based method may need the width and height properties
for positioning to be realized correctly, so use that if positioning seems broken.