2

I am trying to read a specific key from Redis using pyspark. As per documentation, I haven't found any specific command to read a particular key. Using the below code I can read all data from Redis:

testid = spark.read.format("org.apache.spark.sql.redis")\
.option("table",'testing123')\
.option("key.column","id")\
.load()

Kindly suggest

Alex Ott
  • 80,552
  • 8
  • 87
  • 132
Abhishek Vij
  • 53
  • 6

1 Answers1

1

You can try keys.pattern. From the docs:

To read Redis Hashes you have to provide a keys pattern with .option("keys.pattern", keysPattern) option. The DataFrame schema should be explicitly specified or can be inferred from a random row.

[...] Spark-Redis tries to extract the key based on the key pattern:

  • if the pattern ends with * and it's the only wildcard, the trailing substring will be extracted
  • otherwise there is no extraction - the key is kept as is.
testid = spark.read.format("org.apache.spark.sql.redis") \
.option("keys.pattern", "keyPattern:*") \
.option("key.column","id") \
.option("infer.schema", "true") \
.load()
blackbishop
  • 30,945
  • 11
  • 55
  • 76
  • Thank you for your reply, i tried this option but getting below error. Seems it is not taking table and key.pattern together testid = spark.read.format("org.apache.spark.sql.redis")\ .option("table",'testing123')\ .option("keys.pattern", "testing123:44038:1000") \ .option("key.column","id")\ .load() error :-java.lang.IllegalArgumentException: Both options 'table' and 'table' are set. You should only use either one. – Abhishek Vij Feb 09 '21 at 11:10
  • `table` is the implicit `keys.pattern`... so you can use either one or another – Alex Ott Feb 09 '21 at 11:15
  • it seems working with below command `testid = spark.read.format("org.apache.spark.sql.redis")\ .option("keys.pattern", "testing123:44038:1000") \ .option("infer.schema", "true") \ .load() ` But format seems wrong, id column should come first but other column is coming. +-----------------------+---------------------+ |1367366400_002222222222|_id| +-----------------------+---------------------+ |1 | testing123:44038:1000| +-----------------------+---------------------+ – Abhishek Vij Feb 09 '21 at 11:19
  • @blackbishop , can we pass keyPattern:* values dynamically, using some variable? lets say if x = cell:12345 then it should be like `.option("keys.pattern", "x") \` – Abhishek Vij Feb 09 '21 at 13:07