From this document:
The trigger cannot use statements that explicitly or implicitly begin
or end a transaction such as START TRANSACTION, COMMIT, or ROLLBACK.
So ROLLBACK
won't work inside your trigger, but if the trigger raises an exception/error, it will prevent the insertion to succeed, so what you can do is raise an exception if your condition is met (one way is to call an undefined function).
For example:
DELIMITER $$
CREATE
TRIGGER `db`.`before_insert` BEFORE INSERT
ON `db`.`dummy_table`
FOR EACH ROW BEGIN
IF new.topic = 'test' THEN
CALL func_1();
END IF;
END$$
DELIMITER ;
Assuming func_1 doesn't exist it will prevent your new record of being inserted.