I don't believe there is an implementation of List
in the JDK which is faster than O(n) for both lookup and removal.
Data structures which might suit you are the rope, the indexable skiplist, and the enfilade. You might be able to find implementations of those you can use on the web, or in one of the major collections libraries (Guava, Commons Collections, Trove, etc).
However, if what you really want is not fast lookup and removal, but fast random selection and removal, you could use an open-addressed hashtable. You get O(1) lookup by value (not that you care about that), O(1) removal by value, and O(1) selection of a random value. You can select a random value by picking a random slot in the table and using the entry in it; if it's empty, try again.