3

it was known that can't use list.remove in foreach. there is a confuse for me that phase1 runs ok but phase2 was not. can someone explan this?

phase1

List<String> list = new ArrayList<>();
list.add("1");
list.add("2");
for (String item : list) {
    if ("1".equals(item)) {
        list.remove(item);
    }
}
System.out.println(list);

phase2

List<String> list = new ArrayList<>();
list.add("1");
list.add("2");
for (String item : list) {
    if ("2".equals(item)) {
        list.remove(item);
    }
}
System.out.println(list);
huang botao
  • 405
  • 6
  • 13
  • What error message do you get? – geocodezip Jan 25 '21 at 02:04
  • Exception in thread "main" java.util.ConcurrentModificationException at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901) at java.util.ArrayList$Itr.next(ArrayList.java:851) at cn.xxx.main – huang botao Jan 25 '21 at 02:06
  • It would be better to [edit] your question to include the requested information – geocodezip Jan 25 '21 at 02:07
  • no don't close this question, this genius found a java bug! I was just typing up a detailed answer! – rzwitserloot Jan 25 '21 at 02:21
  • This is a bug in line 962 of ArrayList.java from the openjdk. Please await my answer, don't close it unless you fully understand. The information requested is not neccessary, just run the provided samples. – rzwitserloot Jan 25 '21 at 02:21

1 Answers1

4

Crazy. You found a bug in java. Such simple code - it boggles the mind.

Please refer to the source code of ArrayList. The bug is, specifically, in ArrayList.java on line 962.

Ordinarily, arraylists have a so-called 'mod counter'. Anytime you modify the arraylist in any way (be it adding, removing, clearing, etc), the modcounter is incremented by one.

Whenever you invoke .iterator() (which for (String item :list) also does at the very beginning, once), a new iterator object is made (see line 947 in the above link), and this iterator object stores the modcount as it was when the iterator is made.

The idea is that all iterator methods (which aren't many; only hasNext, next and remove) will first check if the modcount of the backing arraylist (the arraylist you got the iterator from by invoking its .iterator() method) is different from the remembered modcount, and if yes, the iterator will instant-abort with a ConcurrentModificationException.

The bug is that the hasNext method fails to do this. I think it's an optimization but it has led to a bug.

Thus, this bizarre interaction occurs:

List<String> list = new ArrayList<String>();

list has modcounter = 0.

list.add("1");
list.add("2");

modcounter is now 2.

for (String item : list) {
    if ("1".equals(item)) {
        list.remove(item);
    }
}

This is syntax sugar. javac compiles it as if it read:

Iterator<String> it$1 = list.iterator();
while (it$1.hasNext()) {
    String item = it$1.next();
    // your actual code inside the for loop here:
    if ("1".equals(item)) {
        list.remove(item);
    }
}

So let's go through it on that basis:

Iterator<String> it$1 = list.iterator();

An iterator object is made; its expectedModCount field is set to 2, as that is the current modcount of list.

while (it$1.hasNext()) {

the position field of the iterator is 0, and the size of the backing list is 2. So, yes, there are more values to return. hasNext() returns true, the while loop will enter.

String item = it$1.next();

modcounter is checked. expectedModCount of the iterator is 2, the mod counter of the list is 2, so that check passes, item is set to "1", and the iterator's position field is incremented, so that is now 1.

if ("1".equals(item)) {
    list.remove(item);
}

The item is indeed "1", so list.remove(item) is called. list updates its modcount to 3, its size to 1, and removes the "1" element from its backing array.

and now the weirdness ensues:

while (it$1.hasNext()) {

well, hasNext() does not check that expectedModCount of the iterator is still equal to the modCount of the list. If it had, this would have failed, but it doesn't do that. The iterator's position field is 1, and the list's size is also 1, so hasNext() returns false, and the while loop exits. That's it: We're out of the loop without ever hitting a ConcurrentModificationException.

In contrast, in the second snippet, you invoke next() twice on it$1 and then the element is removed. At that point, hasNext() is invoked, and then the iterator's position field is 2, the list's size is 1, and the specific check (line 962 of ArrayList) checks if listSize != iteratorPosition. It's not, thus, hasNext returns true (a bit odd). Therefore the while loop enters the body a third time, runs String item = it$1.next(), and the next() method does do a modCount check. expectedModCount is 2, list's modCount is 3, thus, CoModEx is thrown.

To reproduce this, you need to remove the second-to-last element on an arraylist like this. In your example list of 2 elements, that'd be the first element.

How do you actually remove items during iteration?

The right strategy is to use the remove method of iterator. You can't access the iterator in a for(:) loop, so write:

Iterator<String> it = list.iterator();
while (it.hasNext()) {
    String item = it.next();
    if ("2".equals(item)) {
        it.remove(); // this is how to do it!
    }
}

remove() on an iterator is unique: Assuming the modcount check passes, it increments both the iterator's expected modcount as well as the backing list's mod count: It is the only way you can modify a list during iteration without having the iterator fail on you (well, that, and this bug you found).

Next steps

I'll file a bug over at openjdk for this issue.

rzwitserloot
  • 85,357
  • 5
  • 51
  • 72
  • Bug filed via oracle's JavaBugDatabase. I don't have openjdk commit rights, so I hed to go the plebian route: It's bug with ID 9068757, which won't be public until verified and entered into the bug DB. – rzwitserloot Jan 25 '21 at 02:41
  • hmm, how exactly is this is a bug? – Eugene Jan 25 '21 at 03:01
  • @rzwitserloot thansk for your explan, it's very cool! – huang botao Jan 25 '21 at 03:29
  • @Eugene the javadoc states that modifying the list while you are iterating over it will result in a ConcurrentModificationException. In the specific scenario (with the "1"), it fails to do this. This is still a bug: The impl does not do what the javadoc states it must. – rzwitserloot Jan 25 '21 at 04:23
  • That is kinda remarkable. Doesn't look like that code has changed for a long time so hard to imagine how this hasn't been noticed in the last several years. – sprinter Jan 25 '21 at 04:41
  • you mean this : _Fail-fast iterators throw ConcurrentModificationException on a **best-effort basis**_? This is not a bug and imo, it will be rejected. – Eugene Jan 25 '21 at 05:13
  • @Eugene that comment (_Best-effort basis_) refers specifically to what happens when other threads change your list out from under you, and not what happens when the same thread does this. Even if you don't want to read the flanking references to unsynced concurrent modification as indicating that this note (_Best-effort basis_) is strictly about unsynced concurrent modification, the current impl is not best-effort basis. – rzwitserloot Jan 25 '21 at 13:42
  • 1
    Is this a bug? Debatable. The specification says “*it would be wrong to write a program that depended on this exception for its correctness*”. Is this a mind-boggling bug that no-one encountered before? Surely not, see [this old question](https://stackoverflow.com/q/24980651/2711488) and do not forget to check its “Linked” section. And these are surely not all related Q&A. And [this Q&A](https://stackoverflow.com/q/57914151/2711488) suggests that there’s a similar behavior in `HashSet`. – Holger Jan 25 '21 at 16:27
  • 1
    @sprinter of course, this has been noticed before. See [my previous comment](https://stackoverflow.com/questions/65878009/java-list-remove-work-well-for-first-element-but-not-the-others#comment116495569_65878220) – Holger Jan 25 '21 at 16:30
  • 2
    @Eugene since you’ve asked, see [JDK-4902078](https://bugs.openjdk.java.net/browse/JDK-4902078): “*won’t fix*”… – Holger Jan 25 '21 at 16:34
  • @Holger dang, nice find - not sure why I did not find this when searching the bug database. The excuse given (_This change will not be made because a Sun-internal regulatory body rejected it. The formal ruling indicated that the change "has demonstrated the potential to have significant compatibility impact upon existing code." (The "compatibility impact" is that the fix has the potential to replace silent misbehavior with a ConcurrentModificationException.)_ is quality horse exhaust, of course. What a load. Maybe it'll be reconsidered? It's an 18 year old review. – rzwitserloot Jan 25 '21 at 17:09
  • 1
    I spent hours searching for things in that database I knew to exist… I suppose, the “Sun-internal regulatory body” is history. And the view on compatibility requirements may have changed (considering how many collection class implementations have changed fundamentally since then). So the only reason not to change it now, is “because a decision has been made”… – Holger Jan 25 '21 at 17:21