You can restrict access to your published applications in the Google Play developer console, but the restrictions you can use are restrictions like country, type of device (tablet, smartphone), screen size, Android version, etc. Limiting access to specific users or specific devices is however not possible.
You can always publish the apk file on a restriced access download location and then share it amongst your beta users. Another possibility would be to publish the application in the Google Play market and then check in your application whether an user is allowed to run it, but I think that would be more work for you.
Edit: This answer has become outdated since Google introduced beta testing through Google Play, see JuanMa Cuevas' answer.