Remove Duplicate Lines in Text File

Question

I have a text file which I am trying to remove duplicate lines.

Text file example:

new featuredProduct('', '21640'), 
new featuredProduct('', '24664'), 
new featuredProduct('', '22142'), 
new featuredProduct('', '22142'), 
new featuredProduct('', '22142'), 
new featuredProduct('', '22142'), 
new featuredProduct('', '22142'),

The PHP Code I've tried:

$lines = file('textfile.txt');
$lines = array_unique($lines);
file_put_contents('textfile.txt', implode($lines));

The PHP file is called duplicates.php and the textfile is in the same directory. I would like to be left with only:

new featuredProduct('', '21640'), 
new featuredProduct('', '24664'), 
new featuredProduct('', '22142'),

The file function is trying to read the file into the $lines array then array_unique() to remove the duplicate entries. Then put the filtered results back in the same file.

Your code works. maybe replace last line by `file_put_contents('textfile.txt', $lines);` — Med, May 22 '15 at 13:16
perhaps this is what you want? `file_put_contents('textfile.txt', implode(PHP_EOL, $lines));` — Jonathan, May 22 '15 at 13:16
How can you use array_unique on a string... try to echo $lines it is a string — Sunil Pachlangia, May 22 '15 at 13:17

score 6 · Accepted Answer · answered May 22 '15 at 13:16

The problem is the new line characters at the end of each line. Because you don't have a new line character at the end of the last line it won't be the same as the others.

So just remove them when you read the file and then add then when you save the file again:

$lines = file('test.txt', FILE_IGNORE_NEW_LINES | FILE_SKIP_EMPTY_LINES);
$lines = array_unique($lines);
file_put_contents('test.txt', implode(PHP_EOL, $lines));

If yo do: var_dump($lines); right after the file() call you will see it:

array(7) {
  [0]=>
  string(36) "new featuredProduct('', '21640'), 
"
  [1]=>
  string(36) "new featuredProduct('', '24664'), 
"
  [2]=>
  string(36) "new featuredProduct('', '22142'), 
"
  [3]=>
  string(36) "new featuredProduct('', '22142'), 
"
  [4]=>
  string(36) "new featuredProduct('', '22142'), 
"
  [5]=>
  string(36) "new featuredProduct('', '22142'), 
"
  [6]=>
  string(34) "new featuredProduct('', '22142'), "
       //^^ See here                            ^ And here
}

@Hexana You're welcome. Have a nice day :) (You know I could say now, that I use the flags from file() for a reason: http://stackoverflow.com/a/30350862 :) — Rizier123, May 22 '15 at 13:32

score 3 · Answer 2 · answered May 22 '15 at 13:25

I know this question is about PHP and I don't know either you use Linux / Unix or Windows, but there is one really nice bash solution to get rid of duplicates that will work way faster for big files I think. You can even execute it from PHP with a system call:

awk '!a[$0]++' input.txt

score 0 · Answer 3 · answered May 22 '15 at 13:25

0

Try this

$string = file_put_contents('textfile.txt');
$splitstr = explode('),', $string );
$str = implode('),',array_unique($splitstr));
var_dump($str);

answered May 22 '15 at 13:25

Mahadeva Prasad

709
8
19

Remove Duplicate Lines in Text File

3 Answers3

Linked