Remove duplicate data in a text file with php

Question

I am in need to open a text file (file.txt) which contains data in the following format

ai
bt
bt
gh
ai
gh
lo
ki
ki
lo

ultimately I want to remove all the duplicate lines so only one of each data remains. So the result would look like this

ai
bt
gh
lo
ki

any help with this would be awesome

`$unique = array_unique(explode("\n", file_get_contents('file.txt')));` — nkamm, Nov 14 '12 at 18:01
@nkamm please do not use comments to post solutions. Please delete your above comment. — mickmackusa, Jan 08 '18 at 12:40

wassup · Accepted Answer · 2012-11-14T18:16:55.617

26

This should do the trick:

$lines = file('file.txt');
$lines = array_unique($lines);

file() reads the file and puts every line in an array.

array_unique() removes duplicate elements from the array.

Also, to put everything back into the file:

file_put_contents('file.txt', implode($lines));

edited Nov 14 '12 at 18:16

answered Nov 14 '12 at 18:07

wassup

2,333
2
21
30

that did the trick, but it left a blank line between each one. Is there a way to remove the blank lines? – Cesar Bielich Nov 14 '12 at 18:14
Oh, sorry. I forgot that `file()` doesn't remove newline characters. Answer updated. :) – wassup Nov 14 '12 at 18:17
I have a 100MB text file and long rows for which this code is not working. Any alternatives? – Arush Kamboj Apr 09 '15 at 07:38
@ArushKamboj Maybe checkout http://stackoverflow.com/questions/4822165/php-read-large-text-file-log – Cesar Bielich Apr 17 '15 at 16:30

score 2 · Answer 2 · answered Nov 14 '12 at 18:00

2

Take the php function file() to read the file. You get an array of lines from your file. After that, take array_unique to kick out the duplicates.

In the end, you will have something like

$lines = array_unique(file("your_file.txt"));

answered Nov 14 '12 at 18:00

David Müller

5,291
2
29
33

score 0 · Answer 3 · edited Mar 21 '15 at 03:51

0

This might work:

$txt = implode('\n',array_unique(explode('\n', $txt)));

edited Mar 21 '15 at 03:51

Paul Lo

6,032
6
31
36

answered Mar 20 '15 at 16:38

taoufiqaitali

460
5
10

Please add some explanation to this code-only answer. – mickmackusa Jan 08 '18 at 12:46
@mickmackusa: $txt contains the data and `explode()` puts every line in an array (using `\n` delimiter) so we can de-duplicate with `array_unique()`. Everything is then merged back together using `implode()` and the same delimiter. – SuN Dec 02 '19 at 15:37
Try this instead: [edit](https://stackoverflow.com/posts/29171605/edit) – mickmackusa Dec 02 '19 at 20:05

score 0 · Answer 4 · answered Jan 08 '18 at 09:07

0

$lines = file_get_contents('file.txt');
$lines = explode('\n', $lines);
$lines = array_unique($lines);
$lines = implode('\n', $lines);
file_put_contents('file.txt', $lines);

answered Jan 08 '18 at 09:07

Sam Martini

1

1

Your answer is in the Review Queue for being low quality and may be deleted. Please add some explanation with your code-only answer so that future readers can fully benefit. Never post code-only answers on SO. – mickmackusa Jan 08 '18 at 12:43

Remove duplicate data in a text file with php

4 Answers4

Linked

Related