Not sure if you can do it via a hook, but validation types can be added to REDCap via the database, meaning you have to have access to the back-end.
Here's an example of the DB record that defines a validation type:
select * from redcap_validation_types limit 1;
+-----------------+----------------------------------+------------+------------+-----------+--------------+---------+
| validation_name | validation_label | regex_js | regex_php | data_type | legacy_value | visible |
+-----------------+----------------------------------+------------+------------+-----------+--------------+---------+
| abn | Australian Business Number (ABN) | /^\d{11}$/ | /^\d{11}$/ | | NULL | 1 |
+-----------------+----------------------------------+------------+------------+-----------+--------------+---------+
The meat of this is the two regular expressions. If you can express your custom validation as a regex, then you (provided you have database access) can insert it.
If visible = 0
then your REDCap instance will support it if uploaded via a data dictionary, but it won't be selectable from the online designer.