How to split a datafile into multiple parts along with comments in each splitted files?

Question

I have a datafile like this:

# coating file for detector A/R
# column 1 is the angle of incidence (degrees)
# column 2 is the wavelength (microns)
# column 3 is the transmission probability
# column 4 is the reflection probability
      14.2000     0.531000    0.0618000     0.938200
      14.2000     0.532000    0.0790500     0.920950
      14.2000     0.533000    0.0998900     0.900110
# it has lots of other lines
# datafile can be obtained from pastebin

The link to input datafile is: http://pastebin.com/NaNbEm3E

I like to create 20 files from this input such that each files have the comments line.

That is :

#out1.txt
#comments
   first part of one-twentieth data

# out2.txt
# given comments
   second part of one-twentieth data

# and so on upto out20.txt

How can we do so in python?

My intitial attempt is like this:

#!/usr/bin/env python
# -*- coding: utf-8 -*-
# Author    : Bhishan Poudel
# Date      : May 23, 2016


# Imports
from __future__ import print_function
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

# read in comments from the file
infile = 'filecopy_multiple.txt'
outfile = 'comments.txt'
comments = []
with open(infile, 'r') as fi, open (outfile, 'a') as fo:
    for line in fi.readlines():
        if line.startswith('#'):
            comments.append(line)
            print(line)
            fo.write(line)


#==============================================================================
# read in a file
#
infile = infile
colnames = ['angle', 'wave','trans','refl']
print('{} {} {} {}'.format('\nreading file : ', infile, '','' ))
df = pd.read_csv(infile,sep='\s+', header = None,skiprows = 0,
                 comment='#',names=colnames,usecols=(0,1,2,3))
print('{} {} {} {}'.format('length of df : ', len(df),'',''))


# write 20 files
df = df
nfiles = 20
nrows = int(len(df)/nfiles)
groups = df.groupby(   np.arange(len(df.index)) / nrows   )
for (frameno, frame) in groups:
    frame.to_csv("output_%s.csv" % frameno,index=None, header=None,sep='\t')

Till now I have twenty splitted files. I just want to copy the comments lines to each of the files. But the question is: how to do so?

There should be some easier method than creating another 20 output files with comments only and appending twenty_splitted_files to them.

Some useful links are following:
How to split a dataframe column into multiple columns
How to split a DataFrame column in python
Split a large pandas dataframe

It's not quite clear why do you need pandas/data frames in this case... Do you want to keep existing file format or do you want to save splited files as normal CSV or HDF5 files? — MaxU - stand with Ukraine, Jun 17 '16 at 19:39
@MaxU i want to save splitted files as normal CSV files so that each twenty output files have the same header comments as that of input file. — BhishanPoudel, Jun 17 '16 at 19:57
Does your original CSV file fit into RAM or do you have to read it line-by-line? — MaxU - stand with Ukraine, Jun 17 '16 at 20:16
@MaxU my original CSV fits into RAM, its not very large file. — BhishanPoudel, Jun 17 '16 at 20:24

piRSquared · Answer 1 · 2016-06-17T20:33:55.153

This ought to do it

# Store comments in this to use for all files
comments = []

# Create a new sub list for each of the 20 files
data = []
for _ in range(20):
    data.append([])

# Track line number
index = 0

# open input file
with open('input.txt', 'r') as fi:
    # fetch all lines at once so I can count them.
    lines = fi.readlines()

    # Loop to gather initial comments
    line = lines[index]
    while line.split()[0] == '#':
        comments.append(line)
        index += 1
        line = lines[index]

    # Calculate how many lines of data
    numdata = len(lines) - len(comments)

    for i in range(index, len(lines)):
        # Calculate which of the 20 files I'm working with
        filenum = (i - index) * 20 / numdata
        # Append line to appropriately tracked sub list
        data[filenum].append(lines[i])

for i in range(1, len(data) + 1):
    # Open output file
    with open('output{}.txt'.format(i), 'w') as fo:
        # Write comments
        for c in comments:
            fo.write(c)
        # Write data
        for line in data[i-1]:
            fo.write(line)

@piRSquared_ Thanks a lot. – BhishanPoudel Jun 17 '16 at 21:29 — BhishanPoudel, Jun 17 '16 at 21:29

MaxU - stand with Ukraine · Accepted Answer · 2016-06-17T23:13:14.653

1

UPDATE: optimized code

fn = r'D:\download\input.txt'

with open(fn, 'r') as f:
    data = f.readlines()

comments_lines = 0
for line in data:
    if line.strip().startswith('#'):
        comments_lines += 1
    else:
        break

nfiles = 20
chunk_size = (len(data)-comments_lines)//nfiles

for i in range(nfiles):
    with open('d:/temp/output_{:02d}.txt'.format(i), 'w') as f:
        f.write(''.join(data[:comments_lines] + data[comments_lines+i*chunk_size:comments_lines+(i+1)*chunk_size]))
        if i == nfiles - 1 and len(data) > comments_lines+(i+1)*chunk_size:
            f.write(''.join(data[comments_lines+(i+1)*chunk_size:]))

Original answer:

comments = []
data = []

with open('input.txt', 'r') as f:
    data = f.readlines()

i = 0
for line in data:
        if line.strip().startswith('#'):
            comments.append(line)
            i += 1
        else:
            break

data[:] = data[i:]

i=0
for x in range(0, len(data), len(data)//20):
    with open('output_{:02d}.txt'.format(i), 'w') as f:
        f.write(''.join(comments + data[x:x+20]))
        i += 1

edited Jun 17 '16 at 23:13

answered Jun 17 '16 at 20:55

MaxU - stand with Ukraine

205,989
36
386
419

Traceback (most recent call last): File "split_file_with_comments.py", line 25, in data = [line] + f.readlines() ValueError: Mixing iteration and read methods would lose data – BhishanPoudel Jun 17 '16 at 21:00
@MaxU_ I am using macos 10.9 and this code shows same error for python2 and python3, I just removed D:download\ and d:/temp/ names. python3 again shows VALUE_ERROR – BhishanPoudel Jun 17 '16 at 21:12
@BhishanPoudel, i've updated my answer - please check – MaxU - stand with Ukraine Jun 17 '16 at 21:13
@MaxU_ Now it works for both python2 and python3 for me. Thanks a lot! – BhishanPoudel Jun 17 '16 at 21:16
@BhishanPoudel, i've optimized the code a little bit, so it'll loop much less now - it should work much faster – MaxU - stand with Ukraine Jun 17 '16 at 21:38
@MaxU_ Small problem: there are 21 files (00 to 20), even though i changed 20 to 19 in your code. – BhishanPoudel Jun 17 '16 at 22:31
@BhishanPoudel, it should be fixed now - please check – MaxU - stand with Ukraine Jun 17 '16 at 23:13
@MaxU_sir_ Now it is perfect, total data lines = 166, first 19 files have 19*8=152 lines, and last file = 14 lines, total 152+14=166, its perfect! – BhishanPoudel Jun 17 '16 at 23:33

How to split a datafile into multiple parts along with comments in each splitted files?

2 Answers2