Worrying about the speed of the stream extraction operators (<< and >>) in C++ is something to do when you have lots of data to process (over 1E06 items). For smaller sets of data, the execution time is negligible to other factors with the computer and your program.
Before you worry about the speed of formatted I/O, get your program working correctly. Review your algorithms for efficiency. Review your implementation of the algorithms for efficiency. Review the data for efficiency.
The slowness of the stream extraction operators is first translating from textual representation to internal representation, then the implementation. Heck, if you are typing in the data, forget about any optimizations. To speed up your file reading, organize the data for easy extraction and translation.
If you are still panicking about efficiency, use binary file representation. The data in the file should be formatted so that it can be loaded directly into memory without any translations. Also, the data should be loaded in large chunks.
From the Hitchhiker's Guide to the Galaxy, DON'T PANIC.