The rule of three is about dealing with all the Big Three, but that does not necessarily mean you'll have to define them if you don't want to. Either you provide them or you forbid them. What you shouldn't do is ignore them.
So I have defined only destructor, but not copy constructor and copy operator.
Am I violating Rule of three?
Yes, you are in violation of the rule. The compiler will generate a copy constructor and copy assignment operator, and since you allocate memory in the constructor and release in the destructor, these copies will have wrong semantics: they'll copy the pointers, and you will have two classes aliasing the same memory. The assignment won't even release the old memory, and simply overwrite the pointer.
Is this a problem?
If, like you imply, you don't make copies or assign to instances of those classes, nothing will go wrong. However, it's better to be on the safe side and declare (and don't even bother defining) the copy constructor and copy assignment operator private, so you don't invoke them accidentally.
In C++11 you can use the = delete
syntax instead:
T(T const&) = delete; // no copy constructor
T& operator=(T const&) = delete; // no copy assignment