Create Spark cluster - Revision history

Sinoa: /* Security group for spark-driver */

2022-10-31T11:52:29Z

Security group for spark-driver

← Older revision		Revision as of 11:52, 31 October 2022
Line 195:		Line 195:
	* direction: Ingress		* direction: Ingress
	* ports: 4040, 8042, 8080, 8088, 9092, 9870 (you need to create one rule for each)		* ports: 4040, 8042, 8080, 8088, 9092, 9870 (you need to create one rule for each)
	* CIDR: 129.177.146.20/20 (this covers most UiB addresses, add more rules to access from outside UiB/VPN)		* CIDR: 129.177.146.20/20 (this covers most UiB addresses - you can add more rules to access from outside UiB/VPN)

	Still in the Overview window, go to ''Compute -> Instances'', use the ''Actions'' drop-down menus ('''Edit Security Groups''') to add the new security group ''info319-spark-driver'' to the ''spark-driver'' instance.		Still in the Overview window, go to ''Compute -> Instances'', use the ''Actions'' drop-down menus ('''Edit Security Groups''') to add the new security group ''info319-spark-driver'' to the ''spark-driver'' instance.

Sinoa: /* Security group for spark-driver */

2022-10-31T11:51:40Z

Security group for spark-driver

← Older revision		Revision as of 11:51, 31 October 2022
Line 191:		Line 191:
	It is useful to open a few more ports on ''spark-driver'' to be able to access the web UIs for HDFS, YARN, Spark, Zookeeper, and Kafka later.		It is useful to open a few more ports on ''spark-driver'' to be able to access the web UIs for HDFS, YARN, Spark, Zookeeper, and Kafka later.

	In the NREC Overview window, go to ''Network -> Security groups''. Create a new security group ''info319-spark-driver''. '''Add rules''' that open the following TCP ports for ~~Ingress~~:		In the NREC Overview window, go to ''Network -> Security groups''. Create a new security group ''info319-spark-driver''. '''Add rules''' that open the following TCP ports for incoming connections:
	* rule: Custom TCP Rule		* rule: Custom TCP Rule
	* direction: Ingress		* direction: Ingress

Sinoa: /* Set up SSH on the cluster */

2022-10-31T11:47:04Z

Set up SSH on the cluster

← Older revision		Revision as of 11:47, 31 October 2022
Line 132:		Line 132:

	== Set up SSH on the cluster ==		== Set up SSH on the cluster ==
	The virtual machines in the cluster also need to have SSH connections ''between each other''. Internally in the cluster, we will use IPv4 addresses, because some of the software we will run do not yet support IPv6.		The virtual machines in the cluster also need to have SSH connections ''between each other''. Internally in the cluster, we will use IPv4 addresses, because some of the software we will run does not yet support IPv6.

	On your local computer, create an ''exercise-4'' folder, for example a subfolder of ''info319-exercises'', and '''cd''' into it.		On your local computer, create an ''exercise-4'' folder, for example a subfolder of ''info319-exercises'', and '''cd''' into it.
Line 141:		Line 141:
	10.1.1.202 spark-worker-2		10.1.1.202 spark-worker-2
	10.1.2.49 spark-worker-3		10.1.2.49 spark-worker-3
	(''spark-driver'' has a differently-looking IPv4 address because it is global, whereas the workers have local IPv4 addresses that are only valid inside the cluster.)		(''spark-driver'' has a differently-looking IPv4 address because it is global (''DualStack''), whereas the workers have local IPv4 addresses that are only valid inside the cluster.)

	Copy the ''hosts'' file into each instance:		Copy the ''hosts'' file into each instance:

Sinoa: /* Set up SSH on the cluster */

2022-10-31T11:46:05Z

Set up SSH on the cluster

← Older revision		Revision as of 11:46, 31 October 2022
Line 136:		Line 136:
	On your local computer, create an ''exercise-4'' folder, for example a subfolder of ''info319-exercises'', and '''cd''' into it.		On your local computer, create an ''exercise-4'' folder, for example a subfolder of ''info319-exercises'', and '''cd''' into it.

	On your local computer, create a file ''hosts'' with the IPv4 addresses and names of all the instances. The ''spark-driver'' must be first, for example:		On your local computer, create a file ''hosts'' with the IPv4 addresses and names of all the instances. '''The ''spark-driver'' must be first,''' for example:
	158.39.77.125 spark-driver		158.39.77.125 spark-driver
	10.1.0.143 spark-worker-1		10.1.0.143 spark-worker-1

Sinoa: /* Create and mount virtual disks */

2022-10-31T11:44:48Z

Create and mount virtual disks

← Older revision		Revision as of 11:44, 31 October 2022
Line 100:		Line 100:
	Use the ''Actions'' drop-down menus to '''Manage Attachments''' of each new volume to its respective instance. For ''Device Name'' you can try ''/dev/sdb'', or leave it open.		Use the ''Actions'' drop-down menus to '''Manage Attachments''' of each new volume to its respective instance. For ''Device Name'' you can try ''/dev/sdb'', or leave it open.

	Check which path each volume has become ''Attached To''. Here, we will assume that the new virtual disks are all attached at ''/dev/sdb''.		Check which path each volume has become ''Attached To''. Here, we will assume that the new virtual disks are all attached at ''/dev/sdb'' (this is what usually happens).

	Log into each each instance (''spark-driver'', ''spark-worker-1'', ...) in a separate terminal window. ''In each window,'' run the following commands ~~in each~~ to mount and format the disks (we follow [https://techguides.yt/guides/how-to-partition-format-and-auto-mount-disk-on-ubuntu-20-04/ this guide], which works well on 22.04 too):		Log into each each instance (''spark-driver'', ''spark-worker-1'', ...) in a separate terminal window. ''In each window,'' run the following commands to partition, mount and format the disks (we follow [https://techguides.yt/guides/how-to-partition-format-and-auto-mount-disk-on-ubuntu-20-04/ this guide], which works well on 22.04 too).

			Partition the disk:
	sudo gdisk /dev/sdb		sudo gdisk /dev/sdb
	You have to enter this chain of responses:		You have to enter this chain of responses:
Line 112:		Line 114:
	w		w
	Y		Y
	Back at the Linux prompt:
			Back at the Linux prompt on each instance, format and mount the new partition:
	sudo mkfs.ext4 /dev/sdb1 # the '1' is because you chose partition number '1' a few lines up		sudo mkfs.ext4 /dev/sdb1 # the '1' is because you chose partition number '1' a few lines up
	mkdir ~/volume		mkdir ~/volume

Sinoa: /* Create the Spark worker machines (instances) */

2022-10-31T11:37:07Z

Create the Spark worker machines (instances)

← Older revision		Revision as of 11:37, 31 October 2022
Line 77:		Line 77:
	ssh -i ~/.ssh/info319-spark-cluster -J YOUR_USERNAME@login.uib.no ubuntu@2001:700:2:8301::2222		ssh -i ~/.ssh/info319-spark-cluster -J YOUR_USERNAME@login.uib.no ubuntu@2001:700:2:8301::2222

	You can now name the instances and simplify login using ''~/.ssh/config'' as before. To avoid duplicate lines, you can use ~~wildcards~~ like this:		You can now name the instances and simplify login using ''~/.ssh/config'' as before. To avoid duplicate lines, you can use a wildcard like this:
	Host spark-*		Host spark-*
	User ubuntu		User ubuntu

Sinoa: /* Create the Spark driver machine (instance) */

2022-10-31T11:32:25Z

Create the Spark driver machine (instance)

← Older revision		Revision as of 11:32, 31 October 2022
Line 23:		Line 23:
	* boot source: GOLD Ubuntu 22.04 LTS		* boot source: GOLD Ubuntu 22.04 LTS
	* flavor: m1.large (to get started)		* flavor: m1.large (to get started)
	* networks: DualStack (most of your instances will be IPv6 only, but the driver will need ~~Ipv4~~ too later)		* networks: DualStack (most of your instances will be IPv6 only, but the driver will need IPv4 too later)
	* security groups: in addition to the ''default'', add the ''info319-spark-cluster'' group		* security groups: in addition to the ''default'', add the ''info319-spark-cluster'' group
	* key pair: import the public SSH key you just created from ''~/.ssh/info319-spark-cluster.pub''		* key pair: import the public SSH key you just created from ''~/.ssh/info319-spark-cluster.pub''

Sinoa: /* Create the Spark driver machine (instance) */

2022-10-31T11:32:05Z

Create the Spark driver machine (instance)

← Older revision		Revision as of 11:32, 31 October 2022
Line 23:		Line 23:
	* boot source: GOLD Ubuntu 22.04 LTS		* boot source: GOLD Ubuntu 22.04 LTS
	* flavor: m1.large (to get started)		* flavor: m1.large (to get started)
	* networks: DualStack		* networks: DualStack (most of your instances will be IPv6 only, but the driver will need Ipv4 too later)
	* security groups: in addition to the ''default'', add the ''info319-spark-cluster'' group		* security groups: in addition to the ''default'', add the ''info319-spark-cluster'' group
	* key pair: import the public SSH key you just created from ''~/.ssh/info319-spark-cluster.pub''		* key pair: import the public SSH key you just created from ''~/.ssh/info319-spark-cluster.pub''

Sinoa: /* Create security group */

2022-10-31T11:30:41Z

Create security group

← Older revision		Revision as of 11:30, 31 October 2022
Line 16:		Line 16:
	* rule: SSH		* rule: SSH
	* CIDR: 2001:700:200:13::204/64		* CIDR: 2001:700:200:13::204/64
	Here, ''2001:700:200:13::204/64'' is a range of IPv6 UiB-addresses that includes ''login.uib.no''. We will use IPv6 in the course because it is future-leaning and many organisations are quickly running out of IPv4 addresses. Because your machine may not run IPv6 yet, this exercise uses ''login.uib.no'' as a JumpHost to log into the Spark cluster. To log in directly from other IPv6-machines, or using other protocols, you can add more rules to the security group.		Here, ''2001:700:200:13::204/64'' is a range of IPv6 UiB-addresses that includes ''login.uib.no''. We will use IPv6 in the course because it is future-leaning and many organisations (UiB including) are quickly running out of IPv4 addresses. Because your machine may not run IPv6 yet, this exercise uses ''login.uib.no'' as a JumpHost to log into the Spark cluster. This means that SSH connects to your cluster machines via ''login.uib.no'', which converts from IPv4 to IPv6 (and back). To log in directly from other IPv6-machines, or using other protocols, you can add more rules to the security group.

	== Create the Spark driver machine (instance) ==		== Create the Spark driver machine (instance) ==

Sinoa: /* Create SSH key pair */

2022-10-31T11:28:21Z

Create SSH key pair

← Older revision		Revision as of 11:28, 31 October 2022
Line 1:		Line 1:
	== Create SSH key pair ==		== Create SSH key pair ==
	You need a new SSH key pair for your Spark cluster. Do not use a key pair you are using elsewhere, because you will need to upload the private key to the the virtual machines ("instances") in the cluster, and ~~the~~ may not always be secure. On your local computer, in a terminal window that runs Ubuntu Linux and bash shell:		You need a new SSH key pair for your Spark cluster. Do not use a key pair you are using elsewhere, because you will need to upload the private key to the the virtual machines ("instances") in the cluster, and they may not always be secure. On your local computer, in a terminal window that runs Ubuntu Linux and bash shell:
	cd ~/.ssh		cd ~/.ssh
	ssh-keygen -b 4096 -f info319-spark-cluster		ssh-keygen -b 4096 -f info319-spark-cluster