If you need to be able to change the way you access the database without having to rewrite to much code, you better keep EF out of the business layer. You could use the Unit of Work and repository patterns to achieve this; access all the functionality of the data tier through interfaces.
If you drop EF, the interfaces stay the same, you just change the implementing classes.
However, there are arguments for not using UoW and repository, the main one being that the DbContext already provides you many of those features.
I started a project with a UoI and repository at the data layer and no EF references at the business layer. As I progressed I felt that I was just making work for myself, and dropped them. Instead I use access the context from the business tier, perform any conversion I need from DTO to business tier POCO.
In my scenario, I'm confident of sticking with EF. If you are not, consider which approach suits you.