JavaScript Remove duplicates from text/array

Question

I want to remove lines in text based on duplicate keywords. The text goes something like this.

text1,abc,text2
text3,cde,text4
text5,abc,text6
text7,gfh,text8
text9,cde,text10

I want to make it into.

text1,abc,text2
text3,cde,text4
text7,gfh,text8

The idea is to take the text, split it based on lines and put it through 2 loops. Then comparing the two arrays it would remove duplicates from it. How can I do this?

Possible duplicate of http://stackoverflow.com/questions/9229645/remove-duplicates-from-javascript-array — soktinpk, Nov 09 '14 at 19:57
The idea sounds good, so what went wrong? Where did you get stuck? — David Thomas, Nov 09 '14 at 20:01
I wrote some code but didn't work. It splits the text, starts comparing but doesn't do a good job. — edinvnode, Nov 09 '14 at 20:13
And this is not a duplicate from that question.It's different. — edinvnode, Nov 09 '14 at 20:30
You haven't specified on what criteria a line would be considered a duplicate - I had to guess. Also, you haven't shown what you've done so far and why the answer @DavidThomas linked wasn't adequate. — Tom W, Nov 09 '14 at 21:44
But I never linked anything, I just asked where the OP got stuck, implying that he, or she, should show their efforts. Where do those 'lines' come from, are they retrieved from elements in the document, are they in an array, is it *one* string, or multiple strings? — David Thomas, Nov 09 '14 at 22:17

score 0 · Answer 1 · answered Nov 09 '14 at 20:51

0

Here's a jsFiddle that will remove duplicates in the middle column of your data array: http://jsfiddle.net/bonneville/xv90ypgf/

var sortedAr = ar.sort(sortOnMiddleText);
var noDupsAr = removeDups(sortedAr);

Firstly is sorts on the middle column, and then it removes the duplicates. The code is pure javascript but uses jQuery to display the results. (I would prefer to use javaScript forEach instead of the for loop but it is not supported on all browsers so you would need a polyfill.)

answered Nov 09 '14 at 20:51

Bonneville

3,453
4
19
20

Good but can be better. I need the text sorted in lines and lines with duplicates removed, not all lines joined. – edinvnode Nov 09 '14 at 21:19
1

@macroscripts - the text IS sorted with duplicates removed. The result IS an array with duplicates removed (I only joined them for display purposes). – Bonneville Nov 10 '14 at 00:33

score 0 · Answer 2 · answered May 22 '17 at 16:00

The best way to resolve it is to use a key_value array (key-value pair), and note that the key is unique, so you will get automatically a new table with unique values. Hope this will help you.

function RemoveDuplicate(table_data) {
      var listFinale = [], key_value = {};
      for (var i = 0; i < table_data.length; i++) {
        key_value[table_data[i]] = i;
      }
      for (var key in key_value) {
        listFinale.push(key);
      }
      return  listFinale.join(",");
    }

JavaScript Remove duplicates from text/array

2 Answers2

Linked