Struggling with colon ':' in file names

Question

I have the following code that is used to load a lots of "csv.gz" and dump them in other folder with the source filename as a column.

object DailyMerger extends App {
  def allFiles(path:File):List[File]= {
    val parts=path.listFiles.toList.partition(_.isDirectory)
    parts._2 ::: parts._1.flatMap(allFiles)
  }

  val sqlContext = SparkSession.builder().appName("DailyMerger").master("local").getOrCreate()
  val files = allFiles(new File("/Logs/"))
      .map(_.getAbsolutePath())
      .filter(_.endsWith(".csv.gz"))

  val df = sqlContext
      .read
      .format("com.databricks.spark.csv")
      .option("header", "true")
      .option("inferSchema", "true").load(files:_*)
      .withColumn("SENSOR", input_file_name())
      .write
      .option("header", "true")
      .option("compression", "gzip")
      .csv("/tmp/out")
 }

It works like a charm with my test data. But in my "real" data, I have a lots of file containing ':' in their name.

It result in the following exception when Hadoop tries to generate the associate crc file:

java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: .ac:a3:1e:c6:5c:7c.csv.gz.crc
    at org.apache.hadoop.fs.Path.initialize(Path.java:206)
    at org.apache.hadoop.fs.Path.<init>(Path.java:172)
    at org.apache.hadoop.fs.Path.<init>(Path.java:94)
    at org.apache.hadoop.fs.ChecksumFileSystem.getChecksumFile(ChecksumFileSystem.java:90)
    at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:145)
    at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:346)
    at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:766)
    at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.initialize(LineRecordReader.java:85)
    at org.apache.spark.sql.execution.datasources.HadoopFileLinesReader.<init>(HadoopFileLinesReader.scala:46)
    at org.apache.spark.sql.execution.datasources.text.TextFileFormat$$anonfun$buildReader$2.apply(TextFileFormat.scala:105)
    at org.apache.spark.sql.execution.datasources.text.TextFileFormat$$anonfun$buildReader$2.apply(TextFileFormat.scala:104)
    at org.apache.spark.sql.execution.datasources.FileFormat$$anon$1.apply(FileFormat.scala:136)
    at org.apache.spark.sql.execution.datasources.FileFormat$$anon$1.apply(FileFormat.scala:120)
    at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.org$apache$spark$sql$execution$datasources$FileScanRDD$$anon$$readCurrentFile(FileScanRDD.scala:124)
    at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:174)
    at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:105)
    at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source)
    at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
    at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:395)
    at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:234)
    at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:228)
    at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:827)
    at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:827)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
    at org.apache.spark.scheduler.Task.run(Task.scala:108)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:338)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.URISyntaxException: Relative path in absolute URI: .ac:a3:1e:c6:5c:7c.csv.gz.crc
    at java.net.URI.checkPath(URI.java:1823)
    at java.net.URI.<init>(URI.java:745)
    at org.apache.hadoop.fs.Path.initialize(Path.java:203)

Renaming input files is not an option, what are my remaining ones?

[This post](https://stackoverflow.com/questions/25334604/hadoop-java-net-urisyntaxexception-relative-path-in-absolute-uri-rsrchbase-co) suggests that without changing the file names there's not much you can do... HDFS simply fails on such file names; But maybe some other answerer has a cure. — Tzach Zohar, Feb 21 '18 at 17:07

tk421 · Accepted Answer · 2018-02-21T19:25:44.017

As @tzach-zohar said not much. There is a very long history about trying to fix this and it's a non-trivial issue.

The relevant JIRAs are:

Since all the JIRAs are still open, I would say renaming files or using something other than HDFS are the only options.

score 1 · Answer 2 · answered Jul 23 '21 at 11:59

I had the same issue with Jupyter and PySpark in GCP. Found a workaround by saving files paths to list:

path = "gs://path/*.csv"  # can include filenames with with ':'
paths_file = "log_path_list.txt"

!gsutil ls -r $path > $paths_file
path_list = open(paths_file, 'r').read().split("\n")[:-1]

t = spark.read.csv(path_list)

Struggling with colon ':' in file names

2 Answers2

Linked