Stack Overflow
Stack Overflow

Questions tagged [cluster-mode]

22 questions
49
votes
5 answers

Can I add arguments to python code when I submit spark job?

I'm trying to use spark-submit to execute my python code in spark cluster. Generally we run spark-submit with python code like below. # Run a Python application on a cluster ./bin/spark-submit \ --master spark://207.184.161.138:7077 \ …
Jinho Yoo
  • 1,352
  • 5
  • 17
  • 28
14
votes
4 answers

express server port configuration issue with pm2 cluster mode

Problem: We start pm2 in cluster mode, and pm2 starts as many processes as there are cpu cores, pm2 also tries to start as many node servers as there are cpu cores but the problem here is that it fails to start as many servers because they all try…
Gaurav Sobti
  • 569
  • 1
  • 4
  • 14
5
votes
0 answers

pyarrow through spark-submit in cluster mode fails

I have a simple Pyspark code import pyarrow fs = pyarrow.hdfs.connect() If I run this using spark-submit in "client"mode, it works fine, but in "cluster" mode, throws the error Traceback (most recent call last): File "t3.py", line 17, in…
VShankar
  • 151
  • 3
4
votes
0 answers

In Pm2 cluster mode the global variable has different value for each thread

I have a NodeJs application which is running in cluster mode with 4 instance. A map is defined in a file "pipeline.js" like below : var map = new Map(); module.exports = map; I entered value in map like below : map[value] = value_1; In the code I…
Pranav Anand
  • 467
  • 6
  • 12
3
votes
0 answers

Possible to add extra jars to master/worker nodes AFTER spark submit at runtime?

I'm writing a service that runs on a long-running Spark application from a spark submit. The service won't know what jars to put on the classpaths by the time of the initial spark submit, so I can't include it using --jars. This service will then…
Shailesh Patel
  • 71
  • 1
  • 5
2
votes
1 answer

How do I connect to an AWS ElastiCache for Redis Cluster using Jedis?

Previously, we were using Redis with cluster mode disabled via AWS ElastiCache. Our Java code using Jedis was pointing to the primary single node endpoint, which was used for reads and writes. We've now enabled cluster mode. We have now changed the…
codemonkey1010
  • 69
  • 2
  • 8
2
votes
0 answers

PM2 cluster mode nodeJS

PM2: v.4.4.0 NodeJS: v.10.16.0. config.json: { "name": "app-name", "script": "server", "exec_mode": "fork", "instances": 0, "wait_ready": true "listen_timeout": 10000, "kill_timeout": 5000,
Hamber
  • 21
  • 2
1
vote
1 answer

Using JedisCluster with RedisTemplate in Spring

Originally in our code we were using RedisTemplate in conjuction with JedisConnectionFactory and JedisPoolConfig as we were using a Redis on a single node: @Bean(name = "redisTemplate") public RedisTemplate…
codemonkey1010
  • 69
  • 2
  • 8
0
votes
0 answers

With PM2 cluster mode, how to keep state consistency between nodejs instances

I made my nodejs app based on the controller/service/model architecture. All my services are classes with static methods called by controllers or others services. The services class can also have static properties that are used to "cache" some data…
Thomas Perrin
  • 675
  • 1
  • 8
  • 24
0
votes
0 answers

ERROR while running PDI transformation on Carte Cluster

I am working with PDI and Carte Cluster. My goal is to successfully run a transformation/ job on a Carte cluster. I already have deployed each Carte server running as a docker container. From PDI Client, I connected to Pentaho Repository, opened up…
MaiTruongSon
  • 97
  • 1
  • 2
  • 11
0
votes
0 answers

ElastiCache(redis) cluster mode enabled with django

I configured ElastiCache redis with cluster mode enabled. I want to connect ElastiCache with local Django. So I configured bastion host. I connected ElastiCache(non-cluster mode) with local Django. I tried cache.set(), cache.get(). It's OK. I…
김찬우
  • 1
0
votes
1 answer

Which version of Jedis client supports AWS Redis in Cluster mode?

We are on older version of the Jedis client, was wondering which version started support for AWS Elasticache Redis in cluster mode
dgunc
  • 25
  • 2
0
votes
1 answer

Read CSV file in Spark kept in local using Java in cluster mode

I am trying to read a CSV file kept in my local filesystem in UNIX, while running it in cluster mode it's not able to find the CSV file. In local mode, it can read both HDFS and file:/// files. However, in cluster mode, it can only read HDFS…
mukesh dewangan
  • 59
  • 4
0
votes
0 answers

is it possible to zero-downtime with only one pm2 fork mode instance using the pm2 reload command?

Hi I am making api server with nodejs and pm2. I have a question. is it possible to zero-downtime with only one pm2 fork mode instance using the pm2 reload command? Or is it essential to have two instances of pm2 cluster mode to implement zero…
syoo
  • 23
  • 4
0
votes
1 answer

Can two executors / drivers from different Spark applications run on same node in cluster mode?

I read an article in Medium which claims that the number of executors + 1 (for driver), should be a multiple of 3, to efficiently utilize the core on a machine (16 cores, in this case, i.e, 5 per executor and 1 will be reserved for OS and node…
Saiteja Parsi
  • 412
  • 1
  • 5
  • 15
1
2