Not going to work. The closest approach would be this, protecting your first query by wrapping into a subquery:
SELECT NUM
FROM (SELECT 10000 + ROUND(RAND()*89999) AS NUM) AS SUBQ
WHERE NUM NOT IN (SELECT PHONE_NUMBER FROM PHONE);
But while this does select one number between 10000 and 99999, it will be able to check for its nonexistence only afterwards. If the check fails, the query will return nothing and you'll have to retry it (or complicate it with UNIONs, which still won't give an absolute guarantee).
Another possibility would be to generate another table with all numbers between 10000 and 99999, and run a LEFT JOIN
requesting that the right side is NULL. Then you have to select a random entry from that result; I'm not sure, but in this case you might have to run a ORDER BY RAND() followed by LIMIT 1 to extract one record, after potentially selecting nine hundred thousand. This is going to be costly (but see answer at the end).
The most efficient way if the phone table is almost empty would be to use a stored procedure and keep running the first query (or a generation followed by a check) until it succeeds, which if the phone table is sparse, it should do inside very few tries.
Otherwise you can pre-fill the phone table with all numbers (the table is already significantly filled, so this is not a great additional cost), adding a column that states that the number is "free". You then select a random entry from the table with the condition that the free marker yields true. In which case you'll want to read this answer.