Depending on the system use, the public key may be 'well known'. For example, with RSA, the public key is just your modulus plus the public exponent 65537, while the private key is the modulus plus the private exponent (which is the real secret). So someone who knows the private key also knows the public key pretty much by default. The same is true of most elliptic curve based systems.
In theory one could make an RSA-style system where the public exponent is also hard to determine (say a randomly generated value of enough bits to be non-guessable), in which case it would be more symmetric, but that is not the way the system is usually set up. In any case someone who knows the secret parameters underlying the keys (the factors of the modulus in RSA) can easily determine the public key from the private key or the private key from the public key.
In systems like Diffie-Hellman, the public key is actually derived from the private key by a well-known algorithm (there are no secret paramters other than the private key itself), so in such cases the keys are not symmetrical at all, and anyone who knows the private key can trivially determine the public key.