Is there any query/code that can find common values in records?

Question

Somebody can guide me (maybe Simple and fast query if there is or some fast code) to convert my CSV data file (with commas separation):

1,A,C,Z,F,G
2,G,Q,R,C,
3,Z,G,Q,
4,C,F,
5,O,P,
6,O,X,Y,J,
7,A,P,X,

I have this table with ~1,000,000 records like these 7 records that you see (In real Database A,B,C,... are words in string), Records 1 and 2 are common in G and C value and 2,3 and 1,3 and ...

I want to sync records if they have at least two common value like Records 1 & 2,3,4 (but record 5,6,7 haven't at least 2 shared values with others) and generate a list like this:

1 A C Z F G Q R
2 G Q R C A Z F
3 Z G Q A C F R
4 C F A Z G Q R
5 O P
6 O X Y J
7 A P X

at the end we must have 4 same records if we sort data and one others without sync:

1 A C F G Q R Z
2 A C F G Q R Z
3 A C F G Q R Z
4 A C F G Q R Z
5 O P
6 J O X Y
7 A P X

Maybe I do not use good term for my meaning, please see:

1 A C Z F G 
2 G Q R C

record 1 has C and G common with Record 2 now 1 has not R and Q thus we must have 1 A C Z F G + Q and R and Record 2 has not A,Z and F thus we must have: 2 G Q R C + A,Z and F thus at the end we have:

1 A C Z F G Q R 
2 G Q R C A Z F

I need all records Respectively in the queue from top to bottom. wrote a delphi code but it is so slow. Someone suggest me this groovy code:

def f=[:]
new File('Data.csv').readLines().each{
def items=it.split(',')
def name
items.eachWithIndex { String entry, int i -> 
    if(i==0){
        name=entry
    }
    else if(entry){
        if(!f[entry])
            f[entry]=[]
        f[entry]<<name
    }
 } 

}
f.findAll {it.value.size()>1}

It is very fast (because of using a map file I think), but It only finds the common values.

Answer 1

If you would go for a SQL solution, then that csv data could be put in a normalized table with the data unfolded per ID & WORD.

Once you have that, it becomes a matter of self-joining that table. And concatinate the words back together in alphabetic order.

SqlFiddle test here

Not sure how fast this method would be on a table with 1000k records though.
But it's an interesting puzzle.

Sample data:

DROP TABLE IF EXISTS test_words;
CREATE TABLE IF NOT EXISTS test_words (
  id int unsigned NOT NULL PRIMARY KEY,
  words varchar(60) NOT NULL
);
INSERT INTO test_words (id, words) VALUES
(1,'A C Z F G'),
(2,'G Q R C'),
(3,'Z G Q'),
(4,'C F'),
(5,'P O'),
(6,'O X Y J'),
(7,'A P X');

Tally table with numbers:

DROP TABLE IF EXISTS tmp_numbers;
CREATE TEMPORARY TABLE IF NOT EXISTS tmp_numbers (
  n int unsigned NOT NULL PRIMARY KEY
);
INSERT INTO tmp_numbers (n) VALUES (1),(2),(3),(4),(5),(6),(7),(8),(9),(10);

Unfolding the words:

DROP TABLE IF EXISTS test_words_unfolded;
CREATE TABLE test_words_unfolded (
  word varchar(10) NOT NULL,
  id int unsigned NOT NULL,
  PRIMARY KEY (word, id)
);
INSERT INTO test_words_unfolded (word, id)
SELECT DISTINCT SUBSTRING_INDEX(SUBSTRING_INDEX(t.words,' ', nr.n),' ',-1) as word, t.id
FROM test_words AS t
JOIN tmp_numbers AS nr
  ON CHAR_LENGTH(t.words) - CHAR_LENGTH(REPLACE(t.words,' ','')) >= nr.n - 1
 AND SUBSTRING_INDEX(SUBSTRING_INDEX(t.words,' ', nr.n),' ',-1) != '';

Result table:

DROP TABLE IF EXISTS test_result;
CREATE TABLE IF NOT EXISTS test_result (
  id int unsigned NOT NULL PRIMARY KEY,
  words varchar(60) NOT NULL
);

INSERT INTO test_result (id, words) 
SELECT q.id, GROUP_CONCAT(DISTINCT t3.word ORDER BY t3.word ASC SEPARATOR ' ') as words
FROM 
(
    SELECT t1.id, t2.id as id2
    FROM test_words_unfolded t1
    JOIN test_words_unfolded t2 ON t1.word = t2.word
    GROUP BY t1.id, t2.id
    HAVING COUNT(*) > 1 OR t1.id = t2.id
) q
LEFT JOIN test_words_unfolded t3 ON t3.id = q.id2
GROUP BY q.id
ORDER BY q.id;

SELECT *
FROM test_result
ORDER BY id;

Result:

id  words
--  -----
1   A C F G Q R Z
2   A C F G Q R Z
3   A C F G Q R Z
4   A C F G Z
5   O P
6   J O X Y
7   A P X

Extra

To mark the words that have been added, the query to fill the result table becomes a bit more complicated.

SELECT 
 q2.id, 
 GROUP_CONCAT(DISTINCT CASE WHEN q2.ori = 1 THEN q2.word ELSE CONCAT('[',q2.word,']') END ORDER BY q2.word ASC SEPARATOR ' ') as words
FROM
(
  SELECT
   q1.id, t3.word,
   MAX(CASE WHEN q1.id = t3.id THEN 1 ELSE 0 END) as ori
  FROM 
  (
      SELECT
       t1.id, t2.id as id2
      FROM test_words_unfolded t1
      JOIN test_words_unfolded t2 ON t1.word = t2.word
      GROUP BY t1.id, t2.id
      HAVING COUNT(*) > 1 OR t1.id = t2.id
  ) q1
  LEFT JOIN test_words_unfolded t3 ON t3.id = q1.id2
  GROUP BY q1.id, t3.word
) q2
GROUP BY q2.id
ORDER BY q2.id;

Result:

id  words
--  -----
1   A C F G [Q] [R] Z
2   [A] C [F] G Q R [Z]
3   [A] [C] [F] G Q [R] Z
4   [A] C F [G] [Z]
5   O P
6   J O X Y
7   A P X

Additional experiment here