0

Im just new to using pyspark but I find the warnings very annoying, I have googled for days but I could find the suitable answer. Hope someone here can help me?

Im using below code:

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName('project1').getOrCreate()

df_pyspark=spark.read.csv('test_file.txt')

print(df_pyspark.show())

I receive this:

Microsoft Windows [Version 10.0.22000.258]
(c) Microsoft Corporation. All rights reserved.

C:\Users\eigenaar\Desktop\Python>"C:/Program Files/Python310/python.exe" c:/Users/eigenaar/Desktop/Python/test.py
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/C:/opt/spark/spark-3.2.0-bin-hadoop3.2/jars/spark-unsafe_2.12-3.2.0.jar) to constructor java.nio.DirectByteBuffer(long,int)
WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
21/10/25 21:56:56 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
+--------+----+
|     _c0| _c1|
+--------+----+
|    name| age|
|Mohammed|  24|
|    Omar|  13|
|  Othman|  16|
+--------+----+

None

C:\Users\eigenaar\Desktop\Python>SUCCESS: The process with PID 11160 (child process of PID 18252) has been terminated.
SUCCESS: The process with PID 18252 (child process of PID 4772) has been terminated.
SUCCESS: The process with PID 4772 (child process of PID 17916) has been terminated.

My config log4j.properties in C:\opt\spark\spark-3.2.0-bin-hadoop3.2\conf looks like this:

enter image description here

Al-Andalus
  • 115
  • 1
  • 8
  • Perhaps try changing the default log level in `org/apache/spark/log4j-defaults.properties` file to "ERROR" instead of "WARN". – PApostol Oct 25 '21 at 20:07
  • @PApostol I changed WARN to ERROR in `log4j.properties`? but I still get the same warnings... – Al-Andalus Oct 25 '21 at 20:11
  • Have a look here: https://stackoverflow.com/questions/52155078/how-to-fix-hadoop-warning-an-illegal-reflective-access-operation-has-occurred-e – PApostol Oct 25 '21 at 20:13
  • @PApostol could you check my `log4j.properties` file I added it in the question? Is that what you meant? Otherwise I guess I just have to live with those warnings..... – Al-Andalus Oct 25 '21 at 20:18
  • Try changing the other lines with "INFO" to "ERROR" but judging from the question in my previous comment, it's probably not enough. – PApostol Oct 25 '21 at 20:20
  • Yea done that but not enough... :( do you have those warnings to? – Al-Andalus Oct 25 '21 at 20:30
  • No, but I use spark 2.4 and linux – PApostol Oct 25 '21 at 20:32
  • Is there an easy way for me to revert back to spark 2? I can only find spark 3 on their website – Al-Andalus Oct 25 '21 at 20:32
  • You seem to be using spark 3.2.0, you can try downloading an older version and installing that one. – PApostol Oct 25 '21 at 20:33

0 Answers0