How to handle large inputs in c

Question

I was solving a practice problem on a site which states that

The purpose of this problem is to verify whether the method you are using to read input data is sufficiently fast to handle problems branded with the enormous Input/Output warning. You are expected to be able to process at least 2.5MB of input data per second at runtime.

Also how do I optimize input/output routines other than printf and scanf?

To read that much data, just do it in chunks of some size, using for example `fread`. See http://en.cppreference.com/w/c/io/fread — Dan, Oct 26 '12 at 20:04
sorry but i am a newbie and unable to get it can you please guide me or explain me a little bit — nishant agrawal, Oct 26 '12 at 20:08
2.5MB per second is not very impressive if you're using standard desktop hardware... If that's a speed limit of your data processing (not the input), then even unoptimized I/O with buffered `fread` will handle that. Input would surely not be the bottleneck in this case. — paddy, Oct 26 '12 at 20:09
can you please help me how to use unoptimized I/O with buffered fread — nishant agrawal, Oct 26 '12 at 20:11
No. Please try to code by yourself and measure the performance. Ask us only when you did have some working code. — Basile Starynkevitch, Oct 26 '12 at 20:11
can you tell me how to find the run time of my program so that i can compare the performances?? — nishant agrawal, Oct 26 '12 at 20:16
On Linux, use `time` to measure the run time. Inside the program, use `gettimeofday(2)` or `clock_gettime(2)` or `clock(3)` or `time(2)` — Basile Starynkevitch, Oct 26 '12 at 20:18
You are unlucky. On Linux, things are probably simpler. I cannot help you (I never used Windows). — Basile Starynkevitch, Oct 26 '12 at 20:20
ya i think that i am unlucky but what should i do now to find the runtime on windows — nishant agrawal, Oct 26 '12 at 20:21
http://stackoverflow.com/questions/673523/how-to-measure-execution-time-of-command-in-windows-command-line — Basile Starynkevitch, Oct 26 '12 at 20:22
`clock` is a standard C routine, in the `` header, that is likely adequate for the purpose of measuring a practice program like this. — Eric Postpischil, Oct 26 '12 at 20:27
This was closed hastily. Simply directing the questioner to the direct I/O routines in (rather than the formatted or character I/O routines) would likely have been useful (and polite). — Eric Postpischil, Oct 26 '12 at 20:28

Basile Starynkevitch · Answer 1 · 2012-10-26T20:14:00.487

2

It is operating system specific (because the C standard only knows about <stdio.h>). With Linux consider using low-level syscalls for efficiency, like open(2), mmap(2), read(2), pread(2), write(2). You might also want to use readahead(2). Don't forget to make I/O in rather large blocks (e.g. 128Kbytes), page aligned if possible. Read the Advanced Linux Programming book.

If restricted to standard C99 functions, use fread(3) on rather big chunks. Consider also increasing the internal buffer with setvbuf(3)

And 2.5Mbyte/sec is not very impressive. Probably, the bottleneck is the hardware, but you should be able to get perhaps 20 or 50Mbytes/sec on a standard desktop hardware. Using SSD would help a big lot.

edited Oct 26 '12 at 20:14

answered Oct 26 '12 at 20:07

Basile Starynkevitch

223,805
18
296
547

Yes, unbuffered I/O with paged-sized (or slightly less), page-aligned chunks that are also a multiple of the disk sector size is a recipe for speed. You have to make it worthwhile by processing that data efficiently. – paddy Oct 26 '12 at 20:11
i am new to c programming and don't know much so unable to get all technical terms – nishant agrawal Oct 26 '12 at 20:15
Try to code something. If on Linux, use `gcc -Wall -g` to compile it. Debug it with a debugger (e.g. `gdb`). Run it, and test it on large (megabyte sized) files. – Basile Starynkevitch Oct 26 '12 at 20:17
i am working on a windows system can you tell me how to find the runtime of my program so that i can compare the performances – nishant agrawal Oct 26 '12 at 20:18

How to handle large inputs in c

1 Answers1