Read a file line by line and write it transformed line by line into another file

Question

I have to read the content of an existing file "operativedata.txt" line by line and write this data into an excelsheet "datawarehouse.csv" line by line. The content of the file "operativedata.txt" looks like following in the picture (operativedata.txt)):

The data in operativedata.txt have to be written transformed in a different way to datawarehouse.csv. It have to look like this in the csv file:

"date;time;randomvalue\n" \n means after these three lines do a return

This type of format have to be written (all 10 seconds) in the datawarehouse.csv file. It should look like this at the end: datawarehouse.csv

Code for generating datawarehouse.csv:

using System;
using System.IO;
using System.Threading;
namespace etltxt2csv
{
 class Program
            {
              string[]content;//Array for reading each line of the file operativedata.txt
              public void loop()
                    {

                        content = File.ReadAllLines(@"C:\ETL-Process\operativedata.txt");//File to be Read

                        while (true)
                        {
                            for (int i = 0; i < content.Length; i++)
                            {
                                Thread.Sleep(10000);
                                File.AppendAllText(@"C:\ETL-Process\datawarehouse.csv", content[i] + ";" + "\n");
                                Console.WriteLine(content[i]);
                               }
                        }
                    }


                    static void Main(string[] args)
                    {
                        Program a= new Program();
                        a.loop();
                    }
                }
            }

operativedata.txt was created with the following code:

using System;
using System.Threading;
using System.IO;

namespace createData
{


    class Program
    {
        string file, date, time;
        int randomValue;

        public void createFile()
        {
            file = @"C:\ETL-Process\operativedata.txt";
            date = DateTime.Now.ToString("yyMMdd");
            time = DateTime.Now.ToString("HHmmss");
            Random random = new Random();

            while (true)
            {
                randomValue = random.Next(200);
                Thread.Sleep(10000);
                File.AppendAllText(file, "\r\n" +date + "\r\n" + time + "\r\n" + randomValue);
            }
        }



        static void Main(string[] args)
        {
            Program a = new Program();
            a.createFile();
        }
    }
}

What do I have to change in the code of etltxt2csv so that the data can be written transformed in this way:

"date;time;randomvalue\n" (always the first three lines in then the next three lines with a return \n)

into the file datawarehouse.csv ?

When I'm executing the code of etltxt2 I listed here, the Output of my excel file is not transformed like above in the picture (datawarehouse.csv)

your question has too much information, and it will be downvoted and ignored.. please provide a shorter, more to the point question.. no one needs to know your assignment's specifics.. — Eyal Perry, Jun 01 '16 at 20:33
What's with that `Thread.Sleep(10000);`? You want to wait 10 seconds for every line you process? I wouldn't worry about chewing up too many system resources with this program, but if you're paranoid, just sleep maybe 1 millisecond instead. — ebyrob, Jun 01 '16 at 20:39
@EyalPerry Is it still to long? Because it was edited from Blorgebeard :) — Beginner, Jun 01 '16 at 20:41
Its too long/unclear... Also pretty much every part of it is already answered many times on SO (like [read file line by line](http://stackoverflow.com/questions/8037070/whats-the-fastest-way-to-read-a-text-file-line-by-line) ). Editors can't remove code or add questions - so while Blorgeberd removed some fluff it does not make question concrete and well researched. — Alexei Levenkov, Jun 01 '16 at 20:43
@ebyrob The exercise was an idea of my Professor. From my view yes its more better if I say 2000 ms = 2 Sek so that I dont have to wait always 10 seconds. — Beginner, Jun 01 '16 at 20:44
@Karim, sorry, it is long and I do not understand the question.. you provide too much information, and it is unclear what you are asking.. — Eyal Perry, Jun 01 '16 at 20:44
Thanks for your feedback @Alexei Levenkov . Im new here Im sorry. I will try to be more clearly and edit it. — Beginner, Jun 01 '16 at 20:46
@Karim Are you sure your professor meant 10 seconds *per line*? It seems more likely they would be referring to processing the entire file in 10 seconds... For debugging, you can always debug-step through each line — ebyrob, Jun 01 '16 at 20:49
@Eyal Perry Thats ok. Thanks for your feedback. Im trying to be more cleary and edit it then. Its just because its a complicated exercise. — Beginner, Jun 01 '16 at 20:49

Jim Mischel · Answer 1 · 2016-06-01T21:11:35.963

You need to read three lines from the .txt file, and combine them into a single line for output:

using (var input = File.OpenText(txt_file_name))
{
    using (var output = File.CreateText(csv_file_name))
    {
        var date = input.ReadLine();
        var time = input.ReadLine();
        var random = input.ReadLine();
        output.WriteLine("{0};{1};{2}", date, time, random);
    }
}

var is a keyword that tells the compiler to infer the type. For example, when you write:

var input = File.OpenText(filename);

The compiler knows that File.OpenText returns a StreamReader, so it automatically makes input type StreamReader. Here's the same code as above, with var replaced by the actual types:

using (StreamReader input = File.OpenText(txt_file_name))
{
    using (StreamWriter output = File.CreateText(csv_file_name))
    {
        string date = input.ReadLine();
        string time = input.ReadLine();
        string random = input.ReadLine();
        output.WriteLine("{0};{1};{2}", date, time, random);
    }
}

thank you for your fast reply. Im going to try this out. "var" is something new for me. Is this a data type? Sorry im a beginner. — Beginner, Jun 01 '16 at 21:06
@Beginner: If this solved your problem, please mark it as the accepted answer. — Jim Mischel, Jun 07 '16 at 20:50

score 0 · Accepted Answer · answered Jun 01 '16 at 20:52

0

Agreed with Jim Mischel, 3 lines need to be read per line in the output file.

Here's what the loop could look like if you're keeping the same API calls:

(Note: It's better memory use and performance to use Jim Mischel's API calls though...)

for (int i = 0; i + 2 < content.Length; i += 3)
{
   // Console.WriteLine(content[i] + ";" + content[i+1] + ";" + content[i+2]);
   File.AppendAllText(
      @"C:\ETL-Process\datawarehouse.csv",
      content[i + 0] + ";"
      + content[i + 1] + ";"
      + content[i + 2] + "\n"
      );
}

answered Jun 01 '16 at 20:52

ebyrob

667
7
24

I edited my post a bit. Hope its a bit better and clearer. Thanks a lot for your help too. Im going to try this out. My professor wanted us to use in the "createdata" the Thread.Sleep with 2000milliseconds. In etltxt2csv he wanted us to let the process sleep for 10 secons (10000 milli) – Beginner Jun 01 '16 at 21:09
@Karim See my updated answer. – Jim Mischel Jun 01 '16 at 21:11
@Jim Mischel thanks a lot !! Do I got it right that the Keyword "var" sets the type for the value in the variable ? Like you said var input = File.OpenText(filename); returns a StreamReader type which means that the the variable "Input" is of the type StreamReader ? Thanks again.:) – Beginner Jun 01 '16 at 21:27
@Karim That's essentially correct. You can look up the use of `var` in the MSDN documentation. – Jim Mischel Jun 01 '16 at 23:14
@Jim Mischel thank you Mr. Mischel and thanks to ebyrob – Beginner Jun 02 '16 at 14:25

Read a file line by line and write it transformed line by line into another file

2 Answers2