What format of file type in youtube hosting?
Youtube offers multiple formats. If you offer both h.264 and WebM, all modern browsers should be able to play it out of the box.
Can I rent the hosting the can store my video files, including unlimited bandwidth and data transfer?
You'll have to pay for the traffic somehow. Some providers limit by bandwidth (and there are technological limits - more than 10GBit/s will be insanely expensive), but these contracts are usually quite expensive. I'd recommend you start with traffic-based accounting and switch as necessary. Have a look at Amazon EC2 prices for a yardstick. In short, expect about 100$/TB.
How does youtube.com censor porn videos?
In general, there is no 100% accurate detection of video content. This stackoverflow question contains some basics. I'd start with a flagging mechanism - let users report inappropriate videos, and manually review these flags.
Can I use php+MySQL for this project?
Yes, that'd certainly be a good start. However, bear in mind that encoding videos is a long-running task and should be run on a separate, dedicated service.