Need assistance 'combining' partial duplicates in an Java ArrayList

Question

        public class Pair{

             public String key;
             public String value;
             public String amount;

             public Pair(String key, String value, String amount){
                  this.key = key;
                  this.value = value;
                  this.amount = amount;
             }

             public Pair(String key, String value){
                  this.key = key;
                  this.value = value;
             }

             public String getKey(){
                  return key;
             }
             public String getValue(){
                  return value;
             }
             public String getAmount(){
                  return amount;
             }
             public void setAmount(String amount){
                  this.amount =  amount;
             }
        }

---

        List<Pair> pairs = new ArrayList<Pair>();
        List<Pair> duplicates = new ArrayList<Pair>();
        List<Pair> duplicatesWithAmounts = new ArrayList<Pair>();

        for (String line:lines){

            String[] lineparts = line.split(",");

            if ( line.startsWith("Date") || lineparts[3].equals("0.00") || lineparts[3].equals("-0.00") || lineparts[3].equals("000") || lineparts[3].equals("0000") || lineparts[3].equals("00000") ) {
                continue;
            }

            String description = lineparts[4];
            String acct = lineparts[2];
            String amt = lineparts[3];

            Pair pair = new Pair(description, acct);
            Pair pair2 = new Pair(description, acct, amt);

                if(!pairs.isEmpty() && pairs.contains(pair)) {
                    duplicates.add(pair);
                    duplicatesWithAmounts.add(pair2);
                }

            pairs.add(pair);

        }


    for (String linefromFile:lines){

        String[] lineparts = linefromFile.split(",");

        String description = lineparts[4];
        String acct = lineparts[2];
        String amt = lineparts[3];

        Pair pair = new Pair(description, acct);
        Pair pair2 = new Pair(description, acct, amt);

        if(!duplicates.contains(pair)){
            continue;
        }
        duplicates.remove(pair);

       //Here is where I'm lost....

    }

note: amounts are strings for now, they will be passed to a new BigDecimal when added together, but haven't gotten there yet

note2: "lines" is an arraylist of CSV strings

What I need to do where the comment is (saying here is where I'm lost...), is combine the duplicates in duplicatesWithAmounts, based on their key and value, and for every object that the key and value matches, sum the amounts (there may be negatives but that shouldn't matter if we use BigDecimal). So if I had 3 lines that had identical keys and values (regardless of different amounts), I would end with 1 line with that key and value, and a sum of the 3 amounts.

Idea of sample data that may be in duplicatesWithAmounts:

"0001, test1, 147.00"

"0001, test1, 129.00"

"0001, test1, -17.00"

"0002, test2, 7.00"

"0002, test2, -7.00"

"0003, test3, 30.00"

"0003, test3, -12.00"

What I want to end with:

"0001, test1, 259.00"

"0002, test2, 0.00"

"0003, test3, 18.00"

Any help is greatly appreciated, I'm a junior developer and have been struggling with this time sensitive project. I'm sure there is a better way to do this from the start, without my Pair class etc, and without my array's of pairs, so that's fine if someone shows me that as well.. But I would also really like to know how to get what I need with what I currently have, for learning purposes. This is all within the "making transactions lines" section.

---

Ok so I tried Stefan's method, and while I thought I had it all figured out. I am getting to correct amount of lines in my output, with no duplicates, however all of the amounts for every line seem grossly large. Way larger than they should be.

package src;

import java.math.BigDecimal;

public class WDETotalsByDescription{

public String start;
public String end;
public String account;
public BigDecimal amount = new BigDecimal(0);

public String getStart() {
    return start;
}

public void setStart(String start) {
    this.start = start;
}

public String getEnd() {
    return end;
}

public void setEnd(String end) {
    this.end = end;
}

public String getAccount() {
    return account;
}

public void setAccount(String account) {
    this.account = account;
}

public BigDecimal getAmount() {
    return amount;
}

public void setAmount(BigDecimal amount) {
    this.amount = amount;
}

public WDETotalsByDescription(){

}

}

---

public static ArrayList<String> makeWestPacDirectEntry(ArrayList<String> lines, ParametersParser pp){

    ArrayList<String> fileLines = new ArrayList<String>();

//making the header record

    fileLines.add("0 " +
            formatField(pp.getBank(), 21, RIGHT_JUSTIFIED, BLANK_FILLED) +
            "       " +
            formatField(pp.getNameOfUser(), 26, LEFT_JUSTIFIED, BLANK_FILLED) +
            formatField(pp.getNumberOfUser(),6,LEFT_JUSTIFIED,BLANK_FILLED)  +
            formatField(pp.getDescription(),12,LEFT_JUSTIFIED,BLANK_FILLED) +
            MakeSapolRMH.getDate() +
            formatField("",40,true,true));

//Making transactions lines

    BigDecimal totals = new BigDecimal(0);

    HashMap<String, WDETotalsByDescription> tempList = new HashMap<String, WDETotalsByDescription>();


//find duplicates

       Map<Pair, Pair> pairMap = new HashMap<Pair, Pair>();

        for (String line : lines) {
            String[] lineparts = line.split(",");

            if (line.startsWith("Date") || lineparts[3].equals("0.00") || lineparts[3].equals("-0.00")
                    || lineparts[3].equals("000") || lineparts[3].equals("0000") || lineparts[3].equals("00000")) {
                continue;
            }

            String description = lineparts[4];
            String acct = lineparts[2];
            String amt = lineparts[3];

            Pair newPair = new Pair(description, acct, amt);

            if (!pairMap.containsKey(newPair)) {
                pairMap.put(newPair, newPair);
            } else {
                Pair existingPair = pairMap.get(newPair);

                BigDecimal mergedAmount = new BigDecimal(existingPair.getAmount()).movePointRight(2).add((new BigDecimal(newPair.getAmount()).movePointRight(2)));
                existingPair.setAmount(mergedAmount.toString());
            }
        }

        Set<Pair> mergedPairs = pairMap.keySet();

////////////////////

    for (String linefromFile:lines){

        String[] lineparts = linefromFile.split(",");

        if ( linefromFile.startsWith("Date") || lineparts[3].equals("0.00") || lineparts[3].equals("-0.00") || lineparts[3].equals("000") || lineparts[3].equals("0000") || lineparts[3].equals("00000") ) {
            continue;
        }

        for(Pair p:mergedPairs) {

        WDETotalsByDescription wde = new WDETotalsByDescription();

        String line = new String("1");

        line += formatField (lineparts[1], 7, RIGHT_JUSTIFIED, BLANK_FILLED);
        line += formatField (lineparts[2], 9, RIGHT_JUSTIFIED, BLANK_FILLED);
        line += " " + formatField(pp.getTransactionCode(),2,LEFT_JUSTIFIED,ZERO_FILLED);

        wde.setStart(line);
        wde.setAmount(new BigDecimal(p.getAmount()));

        line = formatField (lineparts[4], 32, LEFT_JUSTIFIED, BLANK_FILLED);
        line += formatField (pp.getExernalDescription(), 18, LEFT_JUSTIFIED, BLANK_FILLED);
        line += formatField (lineparts[5], 7, RIGHT_JUSTIFIED, BLANK_FILLED);
        line += formatField (lineparts[6], 9, RIGHT_JUSTIFIED, BLANK_FILLED);
        line += formatField (pp.getNameOfRemitter(), 16, LEFT_JUSTIFIED, BLANK_FILLED);
        line += formatField ("", 8, RIGHT_JUSTIFIED, ZERO_FILLED);

        wde.setEnd(line);
        wde.setAccount(lineparts[4]);

        if(!tempList.containsKey(wde.getAccount())){
            tempList.put(wde.getAccount(), wde);
        }
        else{
            WDETotalsByDescription existingWDE = tempList.get(wde.getAccount());
            existingWDE.setAmount(existingWDE.getAmount().add(wde.getAmount()));
            tempList.put(existingWDE.getAccount(), existingWDE);
        }

        }
    }

    for(WDETotalsByDescription wde:tempList.values()){

        String line = new String();

        line = wde.getStart()
            + formatField(wde.getAmount().toString(), 10, RIGHT_JUSTIFIED, ZERO_FILLED)
            + wde.getEnd();

        if (formatField(wde.getAmount().toString(), 10, RIGHT_JUSTIFIED, ZERO_FILLED) != "0000000000") {
            totals = totals.add(wde.getAmount());
            fileLines.add(line);
        }
    }

// make offset record, requested November of 2012.

    String offset = new String();

    offset += "1";
    offset += pp.getAccountNumber(); // ###-### ######## (or else they will have to use spaces (###-###   ######)) 16 chars total
    offset += " ";

    if (pp.isCredit()){
        offset += "50";
    } else {
        offset += "13";
    }

    offset += formatField(totals.toString(), 10, RIGHT_JUSTIFIED, ZERO_FILLED);
    offset += formatField(pp.getNameOfRemitter(), 16, LEFT_JUSTIFIED, BLANK_FILLED);
    offset += "   " +
    formatField(pp.getInternalDescription(), 28, RIGHT_JUSTIFIED, BLANK_FILLED) +
    "   ";
    offset += pp.getAccountNumber(); // ###-### ######## (or else they will have to use spaces (###-###   ######)) 16 chars total
    offset += formatField(pp.getNameOfRemitter(), 16, LEFT_JUSTIFIED, BLANK_FILLED);
    offset += "00000000";

    fileLines.add(offset);

// MAKE TRAILER RECORD

    String trailerRecord = "7999-999            ";

    trailerRecord += formatField("", 10, RIGHT_JUSTIFIED, ZERO_FILLED);
    trailerRecord += formatField(totals.toString(), 10, RIGHT_JUSTIFIED, ZERO_FILLED);
    trailerRecord += formatField(totals.toString(), 10, RIGHT_JUSTIFIED, ZERO_FILLED);
    trailerRecord += formatField("", 24, RIGHT_JUSTIFIED, BLANK_FILLED);
    trailerRecord += formatField(Integer.toString(fileLines.size()-1), 6, RIGHT_JUSTIFIED, ZERO_FILLED);
    trailerRecord += formatField("", 40, RIGHT_JUSTIFIED, BLANK_FILLED);
    fileLines.add(trailerRecord);

    return fileLines;
}
}

Answer 1

Not a complete answer, but if you are using comparison methods in the Collections framework (such as List.contains()) then the objects you are comparing (in your case, Pair) need to over-ride equals() and hashCode(), otherwise contains() might not behave as you expect.

In your equals() method, compare the values of key, value and amount, and return true if all those values are equal.

The hashCode method can use String.hashCode() for key, value and amount combined.

Answer 2

With Java 1.8, I would use Stream.collect(Collectors.toMap(....)) with a merge function summing the amounts (I assume you'll want to convert the amount field to double) :

List<Pair> lst = ...

Map<String, Pair> map = lst.stream().collect(
            Collectors.toMap(p -> p.getKey() + p.getValue(), Function.identity(), (p1, p2) -> new Pair(p1.getKey(), p1.getValue(), p1.getAmount() + p2.getAmount())));

Collection<Pair> result = map.values();

In Java 1.6 as requested, I would go for Guava's Multimap and then a Maps.transformValues to sum the amounts

Answer 3

You're on the right track, but you need to understand a few concepts before you can get an easy solution to this problem.

First, a Collection which cannot contain duplicates is called a Set. This will come handy later, as we want to end up with a Collection that does not contain duplicates. Another important consideration is that the contains method of a List is very inefficient, so you should probably avoid using it. Sets, on the other hand, are able to provide very efficient implementations of the contains method.

But for Sets to work in Java, you must provide an implementation of the equals method, otherwise the Set will use the default implementation from Object, which compares elements by reference.

The most common type of Set to use is the HashSet. If uses a hash table to index elements, which is a staple data structure for efficient lookups (and allows for a fast contains implementation). To use a HashSet, you must also implement the hashCode method of Object (in fact, you need to make sure that equals and hashCode "agree" when an Object is equal or not, ie. if equals returns true for two elements, then their hashCodes must also be equal).

There are other types of Sets, such as the TreeSet, that do not rely on hashes, look it up online if you're interested in these.

Another thing is that, with the solution I suggest below, you need not only to find if the element exists, but also to be able to retrieve it efficiently (so you can add more "amount" to it). For this, you need a Map, not just a Set.

A common technique to achieve that is to create a Map<Pair, Pair>. So, for each Pair instance, you can check and retrieve the current element efficiently from the Map (just like there's a HashSet, there's a HashMap, which we will use here).

Armed with this knowledge, implementing a solution to your problem is quite trivial:

1. Implement the equals method of Pair (two Pairs are equal if both their key and value are the same, from what you say).
2. Implement the hashCode method of Pair which adheres to its contract, as explained above.
3. Find a way to combine two equal Pairs into one and summing their "amount" fields to create a new Pair which is the "sum" of the 2 elements. Probably a static factory method or a copy constructor will do, but that's up to you how you do that.
4. Put the current element you're looking at into the Map. If it's a duplicate, replace it with a new Pair created by adding the element to the existing one (the existing one is returned by the put operation). Notice you can just put the new element, you don't need to remove the old!

If order is important to you (ie. you care about which element comes first, second and so on) use a LinkedHashMap instead of a HashMap (which does not care about ordering).

Here's part of the solution:

Map<Pair, Pair> pairs = new LinkedHashMap<>();

....

// put adds the currentPair to the Map and returns the existing Pair
// if it already exists, or null otherwise.
Pair oldPair = pairs.put(currentPair, currentPair);
if (oldPair != null) { // duplicate
    Pair sumPair = Pair.sumOf(oldPair, currentPair);
    pairs.put(sumPair, sumPair);
}

Hope you can fill the blanks!

Answer 4

You can use a HashMap<Pair, Pair> to solve this problem:

    Map<Pair, Pair> pairMap = new HashMap<Pair, Pair>();

    for (String line : lines) {
        String[] lineparts = line.split(",");

        if (line.startsWith("Date") || lineparts[3].equals("0.00") || lineparts[3].equals("-0.00")
                || lineparts[3].equals("000") || lineparts[3].equals("0000") || lineparts[3].equals("00000")) {
            continue;
        }

        String description = lineparts[4];
        String acct = lineparts[2];
        String amt = lineparts[3];

        Pair newPair = new Pair(description, acct, amt);
        if (!pairMap.containsKey(newPair)) {
            pairMap.put(newPair, newPair);
        } else {
            Pair existingPair = pairMap.get(newPair);
            String mergedAmount = existingPair.getAmount() + newPair.getAmount();
            existingPair.setAmount(mergedAmount);
        }
    }

    Set<Pair> mergedPairs = pairMap.keySet();

For this to work, Pair has to override hashCode and equals, so that two different Pair instances are considered to be equal, iff the key and the value are equal. Here is an example implementation, generated by Eclipse:

@Override
public int hashCode() {
    final int prime = 31;
    int result = 1;
    result = prime * result + ((key == null) ? 0 : key.hashCode());
    result = prime * result + ((value == null) ? 0 : value.hashCode());
    return result;
}

@Override
public boolean equals(Object obj) {
    if (this == obj) {
        return true;
    }
    if (obj == null) {
        return false;
    }
    if (getClass() != obj.getClass()) {
        return false;
    }
    Pair other = (Pair) obj;
    if (key == null) {
        if (other.key != null) {
            return false;
        }
    } else if (!key.equals(other.key)) {
        return false;
    }
    if (value == null) {
        if (other.value != null) {
            return false;
        }
    } else if (!value.equals(other.value)) {
        return false;
    }
    return true;
}