The goal is to reduce the number of equality test, by grouping possible positive matches together.
The best I can think of is to build a hashmap with the first vector, and then substract all the elements of the second vector from the hashmap. This means that you need to come up with a decent hash function for your elements.
Trivially, if you are storing pointers, and that your equality predicate is based on it, you could also base your hash on that pointer integral value.
See What is a good hash function?.
As mentioned in the comments, an alternate possibility is to establish an ordering on certain attributes of the elements, and use that to reduce the possibilities of equality. You may have to sort both vectors first.