Install Spark on the cluster - Revision history

Sinoa: /* Test run Spark */

2022-10-31T12:17:01Z

Test run Spark

← Older revision		Revision as of 12:17, 31 October 2022
Line 47:		Line 47:
	spark-submit spark-test.py		spark-submit spark-test.py

	~~Change~~ the line		In the program, change the line
	.master("local") \		.master("local") \
	to		to
Line 75:		Line 75:
	* When you run on top of YARN, Spark expects to input files from HDFS, not from the regular file system.		* When you run on top of YARN, Spark expects to input files from HDFS, not from the regular file system.
	* Use the HDFS and YARN web UIs to check what is going on.		* Use the HDFS and YARN web UIs to check what is going on.
	* The default replication factors and other settings are not clever. Don't worry about that now: they are enough to get you started.		* The default replication factors and other settings in YARN and Spark are not clever. Don't worry about that now: they are enough to get you started.

	== Web UIs ==		== Web UIs ==
	While spark is running, you can attempt to access Spark's web UI at http://158.39.201.197:4040 (assuming that 158.39.201.197 is the IPv4 address of ''spark-driver''). But when you run on top of YARN it just attempts to redirect to YARN's web UI at http://158.39.201.197:8088 , which you have accessed already.		While spark is running, you can attempt to access Spark's web UI at http://158.39.201.197:4040 (assuming that 158.39.201.197 is the IPv4 address of ''spark-driver''). But when you run on top of YARN it just attempts to redirect to YARN's web UI at http://158.39.201.197:8088 , which you have accessed already.

Sinoa: /* Test run Spark */

2022-10-31T12:15:04Z

Test run Spark

← Older revision		Revision as of 12:15, 31 October 2022
Line 28:		Line 28:

	== Test run Spark ==		== Test run Spark ==
	On ''spark-driver'', make sure HDFS/YARN are running ('''$HADOOP_HOME/sbin/start-all.sh''', because Spark also defines a ''start-all.sh'' script), and start Spark with:		On ''spark-driver'', make sure HDFS/YARN are running (to be certain, you can use''$HADOOP_HOME/sbin/start-all.sh'', because Spark also defines a ''start-all.sh'' script...), and start Spark with:
	SPARK_PUBLIC_DNS=$MASTER_NODE pyspark		SPARK_PUBLIC_DNS=$MASTER_NODE pyspark
	SPARK_PUBLIC_DNS is the public IP address that Spark's web UI listens to at port 4040. To set it permanently:		SPARK_PUBLIC_DNS is the public IP address that Spark's web UI listens to at port 4040. To set it permanently:

Sinoa: /* Install Spark */

2022-10-31T12:12:10Z

Install Spark

← Older revision		Revision as of 12:12, 31 October 2022
Line 8:		Line 8:
	ln -fns spark-3.3.1-bin-hadoop3 spark		ln -fns spark-3.3.1-bin-hadoop3 spark

	Set $SPARK_HOME and add Spark binaries to path:		Set $SPARK_HOME and add the Spark binaries and scripts to your path:
	export SPARK_HOME=/home/ubuntu/volume/spark		export SPARK_HOME=/home/ubuntu/volume/spark
	export PATH=${PATH}:${SPARK_HOME}/bin:${SPARK_HOME}/sbin		export PATH=${PATH}:${SPARK_HOME}/bin:${SPARK_HOME}/sbin

Sinoa: /* Install Spark */

2022-10-31T12:11:29Z

Install Spark

← Older revision		Revision as of 12:11, 31 October 2022
Line 3:		Line 3:
	Go to [https://spark.apache.org/downloads.html Apache Spark Downloads]. Download and unpack a recent Spark binary. For example, on each instance:		Go to [https://spark.apache.org/downloads.html Apache Spark Downloads]. Download and unpack a recent Spark binary. For example, on each instance:
	cd ~/volume		cd ~/volume
	wget https://dlcdn.apache.org/spark/spark-3.3.0/spark-3.3.0-bin-hadoop3.tgz		wget https://dlcdn.apache.org/spark/spark-3.3.1/spark-3.3.1-bin-hadoop3.tgz
	tar xzvf spark-3.3.0-bin-hadoop3.tgz		tar xzvf spark-3.3.1-bin-hadoop3.tgz
	rm spark-3.3.0-bin-hadoop3.tgz		rm spark-3.3.1-bin-hadoop3.tgz
	ln -fns spark-3.3.0-bin-hadoop3 spark		ln -fns spark-3.3.1-bin-hadoop3 spark

	Set $SPARK_HOME and add Spark binaries to path:		Set $SPARK_HOME and add Spark binaries to path:
Line 15:		Line 15:
	echo "export PATH=\${PATH}:\${SPARK_HOME}/bin:\${SPARK_HOME}/sbin" >> ~/.bashrc		echo "export PATH=\${PATH}:\${SPARK_HOME}/bin:\${SPARK_HOME}/sbin" >> ~/.bashrc

			== Configure Spark ==
	On your local machine, create the file ''spark-defaults.conf'':		On your local machine, create the file ''spark-defaults.conf'':
	spark.master yarn		spark.master yarn

Sinoa: /* Test run Spark */

2022-10-28T17:55:57Z

Test run Spark

← Older revision		Revision as of 17:55, 28 October 2022
Line 38:		Line 38:
	.master("local") \		.master("local") \
	.appName("Word Count") \		.appName("Word Count") \
	~~.config("spark.some.config.option", "some-value") \~~
	.getOrCreate()		.getOrCreate()
	print('*** The result row is',		print('*** The result row is',
Line 46:		Line 45:
	Test run with:		Test run with:
	spark-submit spark-test.py		spark-submit spark-test.py

			Change the line
			.master("local") \
			to
			.master("yarn") \
			to test run on the Hadoop cluster.

	<!--		<!--

Sinoa: /* Test run Spark */

2022-10-25T15:57:13Z

Test run Spark

@@ Line 27: / Line 27: @@
 == Test run Spark ==
-On ''spark-driver'', <!-- make sure HDFS/YARN are running ('''start-all.sh'''),--> and start Spark with:
+On ''spark-driver'', make sure HDFS/YARN are running ('''$HADOOP_HOME/sbin/start-all.sh''', because Spark also defines a ''start-all.sh'' script), and start Spark with:
   SPARK_PUBLIC_DNS=$MASTER_NODE pyspark
 SPARK_PUBLIC_DNS is the public IP address that Spark's web UI listens to at port 4040. To set it permanently:
@@ Line 70: / Line 70: @@
 * Use the HDFS and YARN web UIs to check what is going on.
 * The default replication factors and other settings are not clever. Don't worry about that now: they are enough to get you started.
 == Web UIs ==
 While spark is running, you can attempt to access Spark's web UI at http://158.39.201.197:4040 (assuming that 158.39.201.197 is the IPv4 address of ''spark-driver''). But when you run on top of YARN it just attempts to redirect to YARN's web UI at http://158.39.201.197:8088 , which you have accessed already.

Sinoa: /* Test run Spark */

2022-10-25T15:48:40Z

Test run Spark

← Older revision		Revision as of 15:48, 25 October 2022
Line 27:		Line 27:

	== Test run Spark ==		== Test run Spark ==
	On ''spark-driver'', make sure HDFS/YARN are running ('''start-all.sh'''), and start Spark with:		On ''spark-driver'', <!-- make sure HDFS/YARN are running ('''start-all.sh'''),--> and start Spark with:
	SPARK_PUBLIC_DNS=$MASTER_NODE pyspark		SPARK_PUBLIC_DNS=$MASTER_NODE pyspark
	SPARK_PUBLIC_DNS is the public IP address that Spark's web UI listens to at port 4040. To set it permanently:		SPARK_PUBLIC_DNS is the public IP address that Spark's web UI listens to at port 4040. To set it permanently:

Sinoa: /* Test run Spark */

2022-10-25T15:46:06Z

Test run Spark

← Older revision		Revision as of 15:46, 25 October 2022
Line 27:		Line 27:

	== Test run Spark ==		== Test run Spark ==
	On ''spark-driver'', start Spark with:		On ''spark-driver'', make sure HDFS/YARN are running ('''start-all.sh'''), and start Spark with:
	SPARK_PUBLIC_DNS=$MASTER_NODE pyspark		SPARK_PUBLIC_DNS=$MASTER_NODE pyspark
	SPARK_PUBLIC_DNS is the public IP address that Spark's web UI listens to at port 4040. To set it permanently:		SPARK_PUBLIC_DNS is the public IP address that Spark's web UI listens to at port 4040. To set it permanently:

Sinoa: /* Test run Spark */

2022-10-17T17:43:27Z

Test run Spark

Sinoa: /* Test run Spark */

2022-10-17T17:42:52Z

Test run Spark

← Older revision		Revision as of 17:42, 17 October 2022
Line 53:		Line 53:
	mkdir -p ~/volume/python		mkdir -p ~/volume/python
	cd ~/volume/python		cd ~/volume/python
	~~sudo apt install emacs~~
	~~# edit exercise1.py, first to read form big file, then from hdfs~~
	sudo apt install python3-pip python3-dev python3-venv		sudo apt install python3-pip python3-dev python3-venv
	python3 -m venv venv		python3 -m venv venv
	. venv/bin/activate		. venv/bin/activate
	python3 -m pip install --upgrade pip		python3 -m pip install --upgrade pip
	pip install findspark		pip install tweepy # kafka-python afinn
			# pip install findspark
	# because SPARK_HOME is set, findspark is not needed		# because SPARK_HOME is set, findspark is not needed
			sudo apt install emacs
			# edit exercise1.py, first to read form big file, then from hdfs
	-->		-->

	'''Task:''' Run ''exercise1.py'' from Exercise 1, for example with '''spark-submit exercise1.py''. A few tips:		'''Task:''' Run ''exercise1.py'' from Exercise 1, for example with ''spark-submit exercise1.py''. A few tips:
	* Use the large dataset in ''tweets_id_text_100000.jl'' (i.e., '''small_dataset = False''').		* Use the large dataset in ''tweets_id_text_100000.jl'' (i.e., '''small_dataset = False''').
	* Because SPARK_HOME is set, you do not need ''findspark''.		* Because SPARK_HOME is set, you do not need ''findspark''.
	* ~~WHen~~ you run on top of YARN, Spark expects to ~~find~~ files in HDFS, not in the regular file system.		* When you run on top of YARN, Spark expects to input files from HDFS, not from the regular file system.
	* Use the HDFS and YARN web UIs to ~~look a little at~~ what is going on.		* Use the HDFS and YARN web UIs to check what is going on.
	* The default replication factors and other settings are not clever. Don't worry about that now: they are enough to get you started.		* The default replication factors and other settings are not clever. Don't worry about that now: they are enough to get you started.

	'''Task:''' Run the full Twitter pipeline from Exercise 3.		'''Task:''' Run the full Twitter pipeline from Exercise 3. Tips:
			* You need to create virtual environment to download packages, specifically ''tweepy'', ''kafka-python'', and ''afinn''.
			* You need to create ''keys/bearer_token'' with restricted access.
			* With the default settings, it can take time before the final join is written to console. Be patient...
			* Start-up sequence from ''spark-driver'':
			# assuming zookeeper and kafka run already
			# create kafka topics
			kafka-topics.sh --create --topic tweets --bootstrap-server localhost:9092
			kafka-topics.sh --create --topic media --bootstrap-server localhost:9092
			kafka-topics.sh --create --topic photos --bootstrap-server localhost:9092
			# optional monitoring
			kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic tweets --from-beginning &
			kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic media --from-beginning &
			kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic photos --from-beginning &
			# main programs
			python twitterpipe_tweet_harvester.py # run this repeatedly or modify to reconnect after sleeping
			python twitterpipe_media_harvester.py & # this one can run in background
			spark-submit --packages org.apache.spark:spark-sql-kafka-0-10_2.12:3.3.0 twitterpipe_tweet_pipeline.py &
			Because you no longer use ''findspark'', you need to use the '''--packages''' option with '''spark-submit'''. Make sure you use exactly the same package as you did with ''findspark.add_packages(...)'''.

	== Web UIs ==		== Web UIs ==
	While spark is running, you can attempt to access Spark's web UI at http://158.39.201.197:4040 (assuming that 158.39.201.197 is the IPv4 address of ''spark-driver''). But when you run on top of YARN it just attempts to redirect to YARN's web UI at http://158.39.201.197:8088 , which you have accessed already.		While spark is running, you can attempt to access Spark's web UI at http://158.39.201.197:4040 (assuming that 158.39.201.197 is the IPv4 address of ''spark-driver''). But when you run on top of YARN it just attempts to redirect to YARN's web UI at http://158.39.201.197:8088 , which you have accessed already.

← Older revision		Revision as of 17:43, 17 October 2022
Line 64:		Line 64:
	-->		-->

	'''Task:''' Run ''exercise1.py'' from Exercise 1, for example with ''spark-submit exercise1.py''. A few tips:		'''Task 1:''' Run ''exercise1.py'' from Exercise 1, for example with ''spark-submit exercise1.py''. A few tips:
	* Use the large dataset in ''tweets_id_text_100000.jl'' (i.e., '''small_dataset = False''').		* Use the large dataset in ''tweets_id_text_100000.jl'' (i.e., '''small_dataset = False''').
	* Because SPARK_HOME is set, you do not need ''findspark''.		* Because SPARK_HOME is set, you do not need ''findspark''.
Line 71:		Line 71:
	* The default replication factors and other settings are not clever. Don't worry about that now: they are enough to get you started.		* The default replication factors and other settings are not clever. Don't worry about that now: they are enough to get you started.

	'''Task:''' Run the full Twitter pipeline from Exercise 3. Tips:		'''Task 2:''' Run the full Twitter pipeline from Exercise 3. Tips:
	* You need to create virtual environment to download packages, specifically ''tweepy'', ''kafka-python'', and ''afinn''.		* You need to create virtual environment to download packages, specifically ''tweepy'', ''kafka-python'', and ''afinn''.
	* You need to create ''keys/bearer_token'' with restricted access.		* You need to create ''keys/bearer_token'' with restricted access.
Line 89:		Line 89:
	python twitterpipe_media_harvester.py & # this one can run in background		python twitterpipe_media_harvester.py & # this one can run in background
	spark-submit --packages org.apache.spark:spark-sql-kafka-0-10_2.12:3.3.0 twitterpipe_tweet_pipeline.py &		spark-submit --packages org.apache.spark:spark-sql-kafka-0-10_2.12:3.3.0 twitterpipe_tweet_pipeline.py &
	Because you no longer use ''findspark'', you need to use the '''--packages''' option with '''spark-submit'''. Make sure you use exactly the same package as you did with ''findspark.add_packages(...)'''.		Because you no longer use ''findspark'', you need to use the '''--packages''' option with '''spark-submit'''. Make sure you use exactly the same package as you did with ''findspark.add_packages(...)''.

	== Web UIs ==		== Web UIs ==
	While spark is running, you can attempt to access Spark's web UI at http://158.39.201.197:4040 (assuming that 158.39.201.197 is the IPv4 address of ''spark-driver''). But when you run on top of YARN it just attempts to redirect to YARN's web UI at http://158.39.201.197:8088 , which you have accessed already.		While spark is running, you can attempt to access Spark's web UI at http://158.39.201.197:4040 (assuming that 158.39.201.197 is the IPv4 address of ''spark-driver''). But when you run on top of YARN it just attempts to redirect to YARN's web UI at http://158.39.201.197:8088 , which you have accessed already.