No, They are completely different.
In this one public class Fish extends Pet { }
you are using inheritance to extend the Pet class to the Fish class, meaning that Fish is a subclass of Pet it will inherit the Pet class.
However in this one
public class Fish {
Pet myPet = new Pet (); }
You are creating a whole new object called Fish which does not extend from anything, just that it has a class level object that is a Pet, so you can use the Pet objects methods variables etc through the myPet object however it is not inherited by Fish so Fish would be its own object and not a subclass of Pet.
These are the differences.
As for when you should use which, here is the general rule: if you are enhancing a class then you should use Inheritance, however if you are just going to be using a class for its particular function then you should instantiate it as a variable in the class.