Fetch random rows from a large file and save in another file using python

Asked May 17 '16 at 17:18

Active May 17 '16 at 17:27

Viewed 64 times

I have two large files (8gb text file and 16 gb csv file), I want to extract random 10000 rows from each file and save in new files. i.e. random sample from text file in one file and random sample from csv in another file. I have tried Random Sample

but the problem is it is unable to save the data into a new file properly it is giving weird output (only 3 column results out of 16 columns)

f= raw_input("Enter the fp of the csv file: ")
n = sum(1 for line in open(f)) - 1 #number of records in file (excludes header)
n1=raw_input ("Enter Number of rows required in sample file")



with open(f) as mf:
    head = list(islice(mf, n1))

with open('1.csv' , 'w') as out:
    out.write(str(head))

any help

is highly appreciated

edited May 23 '17 at 12:16

Community

asked May 17 '16 at 17:18

Illuminati

Do you have an ID column? Is it sequential? – Paulo Scardine May 17 '16 at 17:21
1

Show the code you've developed for the problem – Mauro Baraldi May 17 '16 at 17:22
There is no id column and they are not sequential. – Illuminati May 17 '16 at 17:24
Updated question with sample code. – Illuminati May 17 '16 at 17:28
You can avoid having to make to passes over the input file by using [reservoir sampling](https://en.wikipedia.org/wiki/Reservoir_sampling). – Sven Marnach May 17 '16 at 17:42

Fetch random rows from a large file and save in another file using python

0 Answers0