1

The following code works and is slightly shorter than the standard python version (basically each filter statement saves 1 line and each map saves 1 line so this code is 4 lines shorter than the straight python...at the cost of 4 lambdas.

    def xform_logfile(infile, ofile):
        with open(infile, "rt") as inf:
            with open(ofile, "wt") as of:
                m = map(lambda x: x.replace(":", " ").replace("\t", " ").strip().split(), inf)
                m = filter(lambda x: len(x) == 34, m)
                m = map(lambda
                        p: f"{p[0]} {p[1]}:{p[2]},{p[7]},{p[9]},{p[12]},{p[14]},{p[16]},{p[18]},{p[20]},{p[22]},{p[24]},{p[26]},{p[28]},{p[30]},{p[33]}\n",
                    m)
                m = map(lambda line: of.write(line), m)
                # Write new log
                of.write("DateTime, TempLeft, TempRight,Air Input, Driver , YA1, YA2, YB1, YB2, ZA1, ZA2, ZB1,ZB2,TempComp\n")
                list(m)

I'm somewhat new to functional programming in python and am wondering how to handle side effects related to file IO, and really just handling real world examples that aren't pure sequence transformations.

The code above transforms data from a free form log file into a csv file that can be processed by other parts of our system. (I'm just showing the code to show that there are a few functional steps before writing out the data, I've removed exception handling, comments and other filters).

It feels awkward that header has it's own line and then I need to put the data pipeline data into a list to get the side effect to occur. I realize that is what makes the generators pull data through the pipeline, but it just looks wrong in that it is making a list that I am going to ignore. My alternative was to write the last map out in a loop like this:

    header = "DateTime,TempLeft,TempRight,Air Input,Driver,YA1,YA2,YB1,YB2,ZA1,ZA2,ZB1,ZB2,TempComp\n"
    of.write(header)    
    for line in m:
        of.write(line)

instead of using map/lambda, but that also feels wrong.

Is there a way to nicely inject the header into the functional pipeline?

Is this the pythonic syntax to force the generators to be processed or is there something builtin like m.iterate() (I made that up) that will force the generators?

Not shown in this example, but similarly, what if a certain line in my file required two lines to be emitted sort of like a backwards filter. Would the lambda return a list that might be flattened or a string with multiple lines like this?

    map(lambda(x):"New Test" if x=="Test" else [x,"Old Test"])
    or
    map(lambda(x):"New Test" if x=="Test" else f"{x}\nOld Test")
Hucker
  • 671
  • 1
  • 8
  • 25
  • possible duplicate of https://stackoverflow.com/questions/14447560/side-effects-in-python-map-python-do-block – Aviad Rozenhek Nov 22 '19 at 19:06
  • 1
    also, I would use list comprehensions or comprehension expressions instead of maps and filters – Aviad Rozenhek Nov 22 '19 at 19:08
  • 3
    also, I would not put any side effects into lambdas and maps. instead, think about a pure inner functions that just receives the lines of the input file, and returns a generator function that returns output. then, xform_logfile could open the input/output files, call the inner function, and use a loop to iterate over the output of the inner function to write lines to the output file – Aviad Rozenhek Nov 22 '19 at 19:08
  • Just a side note: In FP effects are dealt with lazy types (strictly evaluated environment assumed) often implemented as a function, either a nullary one (thunk) or a continuation. These types are than composed with a set of combinators that implement the Functor, Applicative and Monad typeclass. –  Nov 22 '19 at 20:40

0 Answers0