HTTP is not a secure protocol and anyone can intercept the transmitted data in cleartext (e.g. session cookies, passwords, credit card numbers, sexual fetishes). If you can, you should provide consistent HTTPS service throughout.
That said, by the design of the public/private key security, you can only use HTTPS on a server where you have complete and sole control over the IP address, since the client first looks up the IP address, then requests the secure protocol, and only then makes the HTTP query. That means that you cannot deploy HTTPS on virtual hosts (shared hosting).
(Since you already have a partial HTTPS solution, I imagine that's not a problem for you, though.)
The other downside is that the secure handshake and later encryption require computing resources, so that if you have bazillions of connections, you may feel quite a hit on your server performance. That's for you to consider, though.
Short form: If you have a dedicated IP address and enough computing resources, always and exclusively use HTTPS.