Don't blindly follow rules. That is, follow rules, but don't do it blindly.
The only case when a virtual destructor is truly necessary is when an object is deleted through its base object pointer. The rule of thumb generalizes and simplifies this condition: if an object could possibly be deleted through its base object pointer, then it will be used polymorphically; a polymorphic object is likely to have virtual functions, and an object having virtual functions is likely to bee used polymorphically; therefore, an object with virtual functions is likely to need a virtual destructor.
This is all fine and dandy, the rule mostly works, but there is more important and more fundamental fact that is rarely mentioned, partly because such rules do indeed work. The fact is, there are value-like objects and there are objects of the other kind, which does not have a good name, but I will call them entity-like objects. Entity-like objects have identity separate from their value, they use reference semantics, they should not be copied without good reasons (such as creating a separate identity), they are likely to be accessed polymorphically, etc. Value-like objects have no identity besides their value, they can be copied freely, they should not be used polymorphically, etc. They are so different it would be worth to have different keywords for their classes! When you design your class, you have to decide to which category it belongs. Then your question resolves itself. Entities get virtual destructors, values don't.