My code is below and just used too much of much and file size right now is 700 mb in txt format
StringBuilder dogs = new StringBuilder();
StreamReader str = new StreamReader(file);
while ((line = str.ReadLine()) != null)
{
dogs.AppendLine(line);
}
can any suggest me to store file in any data types but I have to write file after reading and perform operation on it and storing in csv format line by line data
For your scenario with emails i will strongly recommend you to use any SQL database.
You should read and parse first file into database table line-by-line and use SQL queries to search emails from second file. Or you even can parse both files into separate tables and use SQL query to get similar records.
If you do not want to bother with SQL queries and MS Access i will recommend you to use SQLite and sqlite-net ORM library.
I will suggest to read file line by line, process data in each line and the write it to another file stream that way it is not require to have complete data into memory.
If it is require to have past lines data available to process data in current line or if you require to go through all lines to extract some info then I will suggest to save each line into a database and then process data / update rows in database and at last retrieve again to prepare csv file.
On a 64 bit system with enough RAM, this should be fine:
List<string> dogs = new List<string>();
StreamReader str = new StreamReader(file);
while ((line = str.ReadLine()) != null)
{
dogs.Add(line);
}
Here is a brute force version. The bad thing is you are iterating all file2 lines for every line in file1. But you would be doing that in memory too. Best solution is to import files into an RDBMS where you can use indexes.
Is this a one off exercise? What about using a file diff tool such as WinDiff or Beyond Compare?
Or how about this: .bat file to compare two text files and output the difference
using System.IO;
class Program
{
static void Main(string[] args)
{
string line1;
string line2;
using (var fileout = new StreamWriter(@"C:\test\matched.txt"))
{
using (var file1 = new StreamReader(@"C:\test\file1.txt"))
{
while ((line1 = file1.ReadLine()) != null)
{
using (var file2 = new StreamReader(@"C:\test\file2.txt"))
{
while ((line2 = file2.ReadLine()) != null)
{
if (line1 == line2)
{
fileout.WriteLine(line1);
}
}
}
}
}
}
}
}
When you read in the emails from the comparison file, instead of storing the contents of each email, you could compute a hash value for each email and store that instead.
Now when you read emails from the other file, you again compute a hash value for each email and search your list of hashes from the previous pass. If the hash is located, you know that the email was present in the first file.
Since hashes tend to be a lot smaller that the original text (SHA-1 hashes are, for example, 140 bytes each) the collection of hashes should easily fit in RAM.
The following example assumes that emails are stored one per line of text.
using System.Collections.Generic;
using System.IO;
using System.Security.Cryptography;
using System.Text;
var exclude = new List<byte[]>();
var sha1 = new SHA1CryptoServiceProvider();
// read exclusion emails
using (var sr = new StreamReader("exclude-file")) {
string email;
// assume one email per line of text
while ((email = sr.ReadLine()) != null) {
exclude.Add(sha1.ComputeHash(new MemoryStream(Encoding.UTF8.GetBytes(email))));
}
}
// read emails
using (var sr = new StreamReader("email-file")) {
string email;
// again, assume one email per line of text
while ((email = sr.ReadLine()) != null) {
if (exclude.Contains(sha1.ComputeHash(new MemoryStream(Encoding.UTF8.GetBytes(email))))) {
// exclusion file contains email
} else {
// exclusion file does not contain email
}
}
}
