It's free. Basically Amazon provides the certificate for https://*.s3.amazonaws.com . If you are using your own DNS entry https:// then you need your own certificate.
Keep in mind that you might have to make cross domain javascript requests unless all of your site goes to: https://my.website.s3.amazonaws.com
.
To do cross domain Javascript you need to use JSONP. You can read more about it here: Make cross-domain ajax JSONP request with jQuery
Or you can look at something like easyXDM
[Edit]
If for some reason your bucket has a dot or more in its name, for example: mybycket.is.great you can use the old style URL
https://s3.amazonaws.com/admobius.qa/mybucket.is.great/<bucket object>