Improve security of contact form

Question

I have a contact form in html and i'd like to know how can i improve security. my html for the form is the following:

<form class="form" id="form" method="post">
    <input type="hidden" id="token" name="token" value="<?php echo hash_hmac('sha256', 'some_string', $_SESSION['token']); ?>">
    <input type="text" class="form-control" placeholder="Nome" id="nome" name="nome">
    <input type="text" class="form-control" placeholder="Email" id="email" name="email">
    <select class="form-control" id="assunto" name="assunto">
        <option value="1">Mensagem</option>
        <option value="2">Testemunho</option>
        <option value="3">Reclama&ccedil;&atilde;o</option>
        <option value="4">Outro</option>
    </select>
    <textarea class="form-control" Placeholder="Mensagem" id="mensagem" name="mensagem" rows="5"></textarea>
    <div class="text-left" id="msgs">
        <p class="p-mb-30">* Todos os campos são de preenchimento <strong>obrigat&oacute;rio</strong>!</p>
    </div>
    <input type="submit" class="button sendMsgs" value="Enviar" name="send">
</form>

the form will be posted with ajax: (note: i dont do javascript validations because any user can desable javascript so correct me if im wrong but its not that necessary.)

$('#form').on('submit', function(e) {
    e.preventDefault();

    $.ajax({
        type: "POST",
        url: 'php/send-email.php',
        cache: false,
        data: $('.form').serialize(),
        success: function(response) {
            var result = $.parseJSON(response);
            $('#msgs').removeClass();
            $('#msgs').addClass('text-left');
            $('#msgs').addClass(result['type']);
            $('#msgs p').html(result['msg']);
            $('.form').find(':input').removeClass('form-error');
            $.each(result['fields'], function(k, v) {
                var id = '#'+v;
                $(id).addClass('form-error');
            })

            if(result['type'] == 'success') {
                $('.form')[0].reset();
            }
        },
        error: function (xhr, ajaxOptions, thrownError) {
            $('#msgs').addClass('error');
            $('#msgs p').html('Ups! Parece que algo correu mal. Por favor, contacte-nos de outra forma.');
        }
    });
});

and on the php page i make validations, etc... :

session_start();

if($_SERVER['REQUEST_METHOD'] === 'POST' && !empty($_POST)) {

    // i know this function is a little exagerated but im just testing
    function sanitize($input) {
        $input = trim($input);
        $input = htmlentities($input);
        $input = stripslashes($input);
        $input = preg_replace('/<[^>]*>[^<]*<[^>]*>/', '', $input);
        $input = preg_replace('#\s*\<.+\>\s*#U', '', $input);

        return($input);
    }

    $msg = sanitize($_POST['mensagem']);
    $nome = sanitize($_POST['nome']);
    $email= sanitize($_POST['email']);
    $assunto = sanitize($_POST['assunto']);
    $token = sanitize($_POST['token']);

    $error = false;
    $fields = array();

    $allowed = ['nome', 'email', 'assunto', 'mensagem', 'token', 'send'];

    foreach($_POST as $key=>$value) {
        if(!in_array($key, $allowed)) {
            $error = true;
        }
    }

    // check if request is made from the server or not
    if ((isset($_SERVER['HTTP_REFERER']) && !empty($_SERVER['HTTP_REFERER']))) {
        if (strtolower(parse_url($_SERVER['HTTP_REFERER'], PHP_URL_HOST)) != strtolower($_SERVER['HTTP_HOST'])) {
            $error = true;
        }
    }

    // check the token against csrf
    $calc = hash_hmac('sha256', 'some_string', $_SESSION['token']);
    if (!hash_equals($calc, $_POST['token'])) {
        $error = true;
        array_push($fields, 'token');
    }

   if(strlen($msg) < 3) {
        $error = true;
        array_push($fields, 'mensagem');
    }

    if(strlen($nome) < 3) {
        $error = true;
        array_push($fields, 'nome');
    }

    if(!is_numeric($assunto) || strlen($assunto) > 1) {
        $error = true;
        array_push($fields, 'assunto');
    }

    if(strlen($email) < 3 || !filter_var($email, FILTER_VALIDATE_EMAIL)) {
        $error = true;
        array_push($fields, 'email');
    }

    if($error) {
        $msg = 'Existem erros no formul&aacuterio. Por favor, certifique-se que o preencheu correctamente.';
        $type = 'error';
    }
    else {
        // Send email here
        // script to send email.
    }

    $result = array(
        'msg' => $msg,
        'type' => $type,
        'fields' => $fields
    );
    // here i send the result back to my browser
    echo json_encode($result);
}
else {
    // do something here
}

My questions are: is this form safe enough? i do not have access to the server so i can only make this as safe as possible. Am i on the right path? what can i do to make it more secure? should i use phpMailer?