0

I'm working in a cluster with spark(version 2.3.2) using python 3.6.9. And I need to load some audio files .wav in an RDD and perform some operation like compute the spectrogram. So first I load the wav files using spark_context.binaryFiles()

binary_wave_rdd = spark_context.binaryFiles(audio_dir+'*.wav')

And then I need to map the binary file in the RDD to something else that I can use. I'm trying to do this using LibROSA python package but it gives me some strange error when I use collect() on the RDD. This is the code:

binary_wave_rdd = self.spark_context.binaryFiles(audio_dir+'*.wav')
rdd = binary_wave_rdd.map(lambda x : (x[0], librosa.load(io.BytesIO(x[1]))))
coll = self.rdd.collect()

Does anyone know another way to transform the binary data in the RDD? I essentially need it to create the spectrogram of each file

Error

Traceback (most recent call last):
  File "/home/user24/LSCproject/Main.py", line 45, in <module>
    wav = WAV(spark_session, spark_context)
  File "/home/user24/LSCproject/wav_manipulation/wav.py", line 29, in __init__
    self.binary_to_librosa_rdd()
  File "/home/user24/LSCproject/wav_manipulation/wav.py", line 43, in binary_to_librosa_rdd
    l = self.rdd.collect()
  File "/home/hadoop/spark/python/lib/pyspark.zip/pyspark/rdd.py", line 814, in collect
  File "/home/hadoop/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__
  File "/home/hadoop/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 63, in deco
  File "/home/hadoop/spark/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py", line 328, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe.
: org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input Pattern hdfs://master:9000/user/user24/Database/audio_and_txt_files/* 1.wav matches 0 files
        at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:323)
        at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:265)
        at org.apache.spark.input.StreamFileInputFormat.setMinPartitions(PortableDataStream.scala:51)
        at org.apache.spark.rdd.BinaryFileRDD.getPartitions(BinaryFileRDD.scala:51)
        at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
        at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
        at scala.Option.getOrElse(Option.scala:121)
        at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
        at org.apache.spark.api.python.PythonRDD.getPartitions(PythonRDD.scala:57)
        at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
        at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
        at scala.Option.getOrElse(Option.scala:121)
        at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
        at org.apache.spark.SparkContext.runJob(SparkContext.scala:2099)
        at org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:945)
        at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
        at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
        at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)
        at org.apache.spark.rdd.RDD.collect(RDD.scala:944)
        at org.apache.spark.api.python.PythonRDD$.collectAndServe(PythonRDD.scala:165)
        at org.apache.spark.api.python.PythonRDD.collectAndServe(PythonRDD.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
        at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
        at py4j.Gateway.invoke(Gateway.java:282)
        at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
        at py4j.commands.CallCommand.execute(CallCommand.java:79)
        at py4j.GatewayConnection.run(GatewayConnection.java:238)
        at java.lang.Thread.run(Thread.java:748)
Karots96
  • 57
  • 6
  • Please post the errors, i used your code snippet, and it returned with this [('file:/...../sample.wav', (array([-1.5836485e-02, -2.9712297e-02, -2.4191301e-02, ..., -1.8160863e-06, -6.0704710e-06, 0.0000000e+00], dtype=float32), 22050))], please make sure LibROSA is available on the cluster nodes – Ranga Vure May 09 '20 at 17:07
  • you can try these options using other libs to get the spectrogram https://stackoverflow.com/questions/44787437/how-to-convert-a-wav-file-to-a-spectrogram-in-python3 – Ranga Vure May 09 '20 at 17:12
  • @RangaVure also for me in local works, but when I run the code in the cluster gives me an error. I add it on the post – Karots96 May 09 '20 at 18:38
  • Please fix this, it is not finding the input files, Input Pattern hdfs://master:9000/user/user24/Database/audio_and_txt_files/* 1.wav matches 0 files – Ranga Vure May 10 '20 at 08:55

0 Answers0