When I create a table, even if it's using integers only, I set the default charset to utf8 (because I copy paste the code and because in case I introduce a string column in the future).
Example:
CREATE TABLE IF NOT EXISTS `articles` (
`id` smallint(6) unsigned NOT NULL,
`disabled` tinyint(1) unsigned NOT NULL DEFAULT 0,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
However, i'm wondering if it's affecting "performances" to have a default charset in a table that do not make use of it.