I don't know specifically as I'm at home, but SQL CE is deliberately restricted in what it can do. One reason for this is that it is 'always' running locally to the process referencing it.
What means is that it is Expected that the other process is expected to handle much of the logic that may otherwise be encapsulated in the SQL Server. This often results in firing several queries at the SQL CE instance, where you may be more accustomed to firing off one.
In this case, you could do it with two queries...
1) A query to identify the record that you want to delete
2) Use that Identifier in another query to do the actual delete
You could also try using SET ROWCOUNT 1
to limit the DELETE to just 1 row. But again, I don't know if that works in CE.