1

I have this warn on my spark-shell :

"WARN lineage.LineageWriter: Lineage directory /var/log/spark2/lineage doesn't exist or is not writable. Lineage for this application will be disabled."

I don't understand what it means.

What are the consequences? Is it important? Do I have to do something to deal with that?

Thank you

Nastasia
  • 557
  • 3
  • 22
  • https://stackoverflow.com/questions/45751113/what-is-lineage-in-spark explains what Lineage is. And the rest of the error message is pretty self-explanatory, no? – misorude Nov 14 '18 at 15:06
  • I didn't think it was exactly the same thing. Does this mean that without lineage RDD cannot be recomputed? So, when an executor fails it has to begin from the beginning, isn't it? – Nastasia Nov 14 '18 at 15:56
  • https://community.cloudera.com/t5/Advanced-Analytics-Apache-Spark/CDS-2-3-release-2-Lineage-File-Missing-Error/m-p/68081 – thebluephantom Nov 14 '18 at 21:34
  • Thnak you for your answers. i understand better what is lineage. What I still don't understand is the importance of this feature. – Nastasia Nov 15 '18 at 09:11

0 Answers0