7

I'm trying to write an etl pipeline using luigi. As far as I understand from the documentation a task in luigi can generate a target that can be either some type of file storage or a database. To decrese the processing time I would like to have as an output an in-memory list. Is this possible? Do I have to create a custom target?

djWann
  • 2,017
  • 4
  • 31
  • 36
  • Related to question https://stackoverflow.com/questions/42515463/passing-python-objects-between-tasks-in-luigi, which some useful looking answers – Scott H Mar 24 '21 at 21:59

1 Answers1

9

I found out I can use a MockFile as a target. A good example:

http://gouthamanbalaraman.com/blog/building-luigi-task-pipeline.html

djWann
  • 2,017
  • 4
  • 31
  • 36
  • 1
    the author noted that: "I use the MockFile here just as a way to see the results on the console. This is usually a good testing tool, though I am not sure if it is production ready approach." So have you tried MockFile for (semi-)production pipelines? Is it right way for in-memory pipelining? – Eugene Krevenets Jun 17 '19 at 11:22