Sunday, May 31, 2015

Launch spark streaming samples


Then you try to launch streaming sample, test RDD has no output data and, at first sight, no java error stacks, but include warning:

WARN BlockManager: Block input ... replicated to only 0 peer(s) instead of 1 peers

Possible reason is that spark can not execute submitted task, because can not create additional execution instances. It must be at least 2 for streaming. Setting more instances in submit command with same result.

spark-submit --class JavaCustomReceiver -master local[*]

Check user parameter
export SPARK_EXECUTOR_INSTANCES=N


No comments:

Post a Comment