Comments re the importance of using mysqli* instead of mysql* API are quite important and should not be ignored, but since you've started with mysql* I will continue in that same route...
I believe the previous answers (using HAVING ...
) were not answering your actual question. Try the code below.
$sql = mysql_query("SELECT * FROM transfer WHERE name = '$name'");
if (mysql_num_rows($sql) >= 5) {
echo "Error - Too many occurrences of $name";
} else {
$insert_query = "INSERT INTO `transfer`(`t_id`, `agent_id`, `agent_name`, `date`, `name`, `phone`, `email`, `tname`, `tphone`, `temail`, `status`) VALUES (NULL,'$agent_id','$agent_name','$date','$name','$phone','$email','$tname','$tphone','$temail','$cmmnt','$status')";
$insert_result= $mysqli -> query($insert_query);
}