0

there are some parquet file paths are:

/a/b/c='str1'/d='str'

/a/b/c='str2'/d='str'

/a/b/c='str3'/d='str'

I want to read the parquet files like this:

df = spark.read.parquet('/a/b/c='*'/d='str')

but it doesn't work by using "*" wildcard character.How can I do that? thank you for helping

pault
  • 41,343
  • 15
  • 107
  • 149
Frank
  • 977
  • 3
  • 14
  • 35

1 Answers1

1

You need to escape single quotes:

df = spark.read.parquet('/a/b/c=\'*\'/d=\'str\'')

... or just use double quotes:

df = spark.read.parquet("/a/b/c='*'/d='str'")
Sergey Khudyakov
  • 1,122
  • 1
  • 8
  • 15
  • @ZhangXin hm, it should work. Does the HDFS path contain those quotes? Also, please provide the _exact_ code you have - `'/a/b/c='*'/d='str'` is not a valid Python string – Sergey Khudyakov May 21 '18 at 16:25