0

I have below python (3.10.9) and spark version (3.4.0) installed on mac.

(base) xyz@terminal ~ % python --version
Python 3.10.9
(base) xyz@terminal ~ % python3 --version
Python 3.10.9
(base) xyz@terminal ~ % where python     
/Users/xyz/anaconda3/bin/python
(base) xyz@terminal ~ % where python3
/Users/xyz/anaconda3/bin/python3
/usr/bin/python3

(base) xyz@terminal ~ % which spark-shell
/opt/homebrew/bin/spark-shell

when I launch PyCharm IDE, create a simple test .py script having below code, I can see red warning on top right side of IDE - 'unresolved reference PySpark' (install package pyspark), 'unresolved reference col' (install package col). Should I click on install, does this cause any conflicts with already installed spark? Since I already have pyspark installed , why PyCharm is not able to use them?
I have already added Python 3.9 (/usr/bin/python3) as Python Interpreter

from pyspark.sql import SparkSession
from pyspark.sql.functions import col

spark = SparkSession.builder \
    .appName("test Consumer") \
    .getOrCreate()
Matthew
  • 55
  • 7
  • you'd have to configure pycharm to accept the package – samkart Aug 30 '23 at 09:38
  • I have already configured Python 3.9 (/usr/bin/python3) as Python Interpreter on PyCharm. Is there anything else to configure? – Matthew Aug 30 '23 at 09:46
  • add the paths to pyspark bins – samkart Aug 30 '23 at 10:04
  • 1
    Mostly likely you have created a fresh pycharm project with its own virtual environment. Click on `Terminal` tab (at the bottom row of pycharm) and install whatever package it is recommending to install. Since you are beginner, don't worry about environment issues now. – user238607 Aug 30 '23 at 10:06
  • /opt/homebrew/bin/ path contains pyspark and spark-submit, hence added export PATH="$PATH:/opt/homebrew/bin/" to .zshrc file, did source and restarted pycharm. Still running into same message. – Matthew Aug 30 '23 at 10:26
  • 1
    Try Manual method mentioned in this answer if you insist on using system pyspark. https://stackoverflow.com/a/34714207/3238085 – user238607 Aug 30 '23 at 10:38

0 Answers0