53

I have an email dump of around 400mb. I want to split this into .txt files, consisting of one mail in each file. Every e-mail starts with the standard HTML header specifying the doctype.

This means I will have to split my files based on the above said header. How do I go about it in linux?

Greenhorn
  • 1,811
  • 5
  • 21
  • 39
  • Is that really an email dump? You mean you have no mail headers at all? And what do you call the "standard HTML header specifying the doctype"? – fge Dec 17 '11 at 10:52
  • " This is followed by the entire e-mail! – Greenhorn Dec 17 '11 at 10:53

5 Answers5

91

If you have a mail.txt

$ cat mail.txt
<html>
    mail A
</html>

<html>
    mail B
</html>

<html>
    mail C
</html>

run csplit to split by <html>

$ csplit mail.txt '/^<html>$/' '{*}'

 - mail.txt    => input file
 - /^<html>$/  => pattern match every `<html>` line
 - {*}         => repeat the previous pattern as many times as possible

check output

$ ls
mail.txt  xx00  xx01  xx02  xx03

If you want do it in awk

$ awk '/<html>/{filename=NR".txt"}; {print >filename}' mail.txt
$ ls
1.txt  5.txt  9.txt  mail.txt
kev
  • 155,172
  • 47
  • 273
  • 272
  • Am afraid! I did the same and did a $ls mail.txt xx00 and obviously mail.txt was the same as xx00 Any fixes? – Greenhorn Dec 17 '11 at 12:09
  • @Ramprakash My `csplit`'s ver is `8.5`. Maybe yours don't have the `{*}` which repeat pattern. please check manpage. I just add `awk` solution. You can try it. – kev Dec 17 '11 at 12:19
  • 1
    @Greenhorn My version of `csplit` also didn’t support `{*}`, but this worked: `csplit -n 6 -f 'mail-' -k mail.txt '/^$/' '{5000}'` – Daniel Gasienica Mar 30 '16 at 07:31
  • To prevent an `awk` error if the first line doesn't match the pattern (for `gawk` at least), do: `awk 'BEGIN {filename="0.txt"} /...'` – mwfearnley Aug 28 '19 at 11:21
3

csplit is the best solution to this problem. Just thought I'd post a bash-solution to show that there is no need to go perl on this task:

#!/usr/bin/bash

MAIL='mail'        # path to huge mail-file

#get linenumbers for all headers
line_no=$(grep -n html $MAIL | cut -d: -f1)

read -a LINES<<< $line_no

file=0
for i in $(seq 0 2 ${#LINES[@]}); do
    start=${LINES[i]}
    end=$((${LINES[i+1]}-1))
    echo $start, $end
    sed -n "${start},${end}p" $MAIL > ${MAIL}${file}.txt
    file=$((file+1))
done
Fredrik Pihl
  • 44,604
  • 7
  • 83
  • 130
  • In the seq command, I don't know why a step-width of 2 was chosen. I changed it to 1 in order to work for me. – boutta May 17 '22 at 13:45
2

The csplit program solves your problem elegantly:

csplit '/<!DOCTYPE.*/' $FILE
thiton
  • 35,651
  • 4
  • 70
  • 100
1

It is doable with some perl "magic"... Many people would call this ugly but here goes.

The trick is to replace $/ with what you want and read your input, as such:

#!/usr/bin/perl -W
use strict;
my $i = 1;

$/ = <<EOF;
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"><html><head> <xmeta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
EOF

open INPUT, "/path/to/inputfile" or die;

while (my $mail = <INPUT>) {
    $mail = substr($mail, 0, index($mail, $/));
    open OUTPUT, ">/path/to/emailfile." . $i . ".txt" or die;
    $i++;
    print OUTPUT $mail;
    close OUTPUT;
}

edit: fixed, I always forget that $/ is included in the input. Also, the first file will always be empty, but then it can be easily handled.

fge
  • 119,121
  • 33
  • 254
  • 329
1

I agree with fge. With perl it would be a lot simpler. You can try something like this - 

#!/usr/bin/perl

undef $/;
$_ = <>;
$n = 0;

for $match (split(/(?=HEADER_FORMAT)/)) {
      open(O, '>mail' . ++$n);
      print O $match;
      close(O);
}

Replace HEADER_FORMAT with your header type.

jaypal singh
  • 74,723
  • 23
  • 102
  • 147
  • Yep, a positive lookahead would work nicely, especially since here the header does not contain any metacharacter. You could even use `qr//` to build the split regex. – fge Dec 17 '11 at 11:16