Create Spark cluster using Terraform - Revision history

Sinoa: /* Configure SSH */

2022-10-31T13:38:50Z

Configure SSH

← Older revision		Revision as of 13:38, 31 October 2022
Line 243:		Line 243:
	On your local host, change ''info319-cluster.tf'' so it also writes a file like this to ''~/.ssh/config.terraform-hosts'':		On your local host, change ''info319-cluster.tf'' so it also writes a file like this to ''~/.ssh/config.terraform-hosts'':
	Host terraform-driver		Host terraform-driver
	Hostname 2001:700:2:8300::~~28c4~~		Hostname 2001:700:2:8300::27c4

	Host terraform-worker-0		Host terraform-worker-0
	Hostname 2001:700:2:8301::~~14a1~~		Hostname 2001:700:2:8301::13a1

	...		...

	Host terraform-worker-5		Host terraform-worker-5
	Hostname 2001:700:2:8301::~~120d~~		Hostname 2001:700:2:8301::110d

	On your local host, add these lines to ''~/.ssh/config'':		On your local host, add these lines to ''~/.ssh/config'':
	Host terraform-*		Host terraform-* localhost
	User ubuntu		User ubuntu
	IdentityFile ~/.ssh/info319-spark-cluster		IdentityFile ~/.ssh/info319-spark-cluster

Sinoa: /* Create hosts files */

2022-10-31T13:36:52Z

Create hosts files

← Older revision		Revision as of 13:36, 31 October 2022
Line 204:		Line 204:

	=== Create ''hosts'' files ===		=== Create ''hosts'' files ===
	Make OpenStack write out the ~~file~~ ''ipv4-hosts'' looking like this:		Make OpenStack write out the files ''ipv4-hosts'' looking like this:
	158.37.65.58 terraform-driver		158.37.65.59 terraform-driver
	10.1.2.~~234~~ terraform-worker-0		10.1.2.233 terraform-worker-0
	...		...
	10.1.2.63 terraform-worker-5		10.1.2.63 terraform-worker-5
	and ''ipv6-hosts'' looking like this:		and ''ipv6-hosts'' looking like this:
	2001:700:2:8300::~~28c4~~ terraform-driver		2001:700:2:8300::27c4 terraform-driver
	2001:700:2:8301::~~14a1~~ terraform-worker-0		2001:700:2:8301::13a1 terraform-worker-0
	...		...
	2001:700:2:8310::~~120d~~ terraform-worker-5		2001:700:2:8310::110d terraform-worker-5

	Tips:		Tips:
Line 220:		Line 220:
	value = openstack_compute_instance_v2.terraform-driver.access_ip_v4		value = openstack_compute_instance_v2.terraform-driver.access_ip_v4
	}		}
	* you can use the '''[https://www.terraform.io/cli/commands/console terraform console]''' to explore expressions you can use to define outputs and locals, for example:		* you can use the '''[https://www.terraform.io/cli/commands/console terraform console]''' to explore expressions you can use to define outputs and locals, for example (inside the console):
	openstack_compute_instance_v2.terraform-workers[5]		openstack_compute_instance_v2.terraform-workers[5]
	* when an output works as expected, you can redefine it as a local variable and use it to ~~defined~~ more complex outputs		* when an output works as expected, you can redefine it as a local variable and use it to define more complex outputs
	* a few useful functions are:		* a few useful functions (you may not need all of them) are:
	~~concat([local.element], local.list)~~
	length(string)		length(string)
			join(string, list)
			concat([element], list)
	substr(string, offset, length)		substr(string, offset, length)
	zipmap(list1, list2)		zipmap(list1, list2)

Sinoa: /* Create and attach volumes */

2022-10-31T13:32:02Z

Create and attach volumes

← Older revision		Revision as of 13:32, 31 October 2022
Line 201:		Line 201:
	}		}

	Log in with SSH and to '''ls /dev''' to see that the volumes are attached where they should (but they are not yet partitioned, formatted, and mounted).		Log in with SSH and do '''ls /dev''' to see that the volumes are attached where they should (but they are not yet partitioned, formatted, and mounted).

	=== Create ''hosts'' files ===		=== Create ''hosts'' files ===

Sinoa: /* Spark cluster */

2022-10-31T13:31:15Z

Spark cluster

← Older revision		Revision as of 13:31, 31 October 2022
Line 168:		Line 168:
	ssh -i ~/.ssh/info319-spark-cluster -J YOUR_USERNAME@login.uib.no ubuntu@2001:700:2:8300::21f4		ssh -i ~/.ssh/info319-spark-cluster -J YOUR_USERNAME@login.uib.no ubuntu@2001:700:2:8300::21f4

	=== Create the Spark driver ===		=== Create the Spark driver and workers ===
	The ''terraform-test'' resource can be renamed to for example ''terraform-driver'' and modified as you like according to the [https://registry.terraform.io/providers/terraform-provider-openstack/openstack/latest/docs/resources/compute_instance_v2 documentation here].		The ''terraform-test'' resource can be renamed to for example ''terraform-driver'' and modified as you like according to the [https://registry.terraform.io/providers/terraform-provider-openstack/openstack/latest/docs/resources/compute_instance_v2 documentation here].

	~~=== Create the Spark workers ===~~
	You can create multiple ''terraform-worker-'' resources in a similar way, but adding a line		You can create multiple ''terraform-worker-'' resources in a similar way, but adding a line
	count = 6		count = 6
Line 178:		Line 177:
	Check that you can login to the new instances using '''ssh''' and a IPv6 JumpHost.		Check that you can login to the new instances using '''ssh''' and a IPv6 JumpHost.

			=== Local Terraform variables ===
	As ''info319-cluster.tf'' grows larger, you can ensure consistency by defining local variables in the beginning of the file, for example:		As ''info319-cluster.tf'' grows larger, you can ensure consistency by defining local variables in the beginning of the file, for example:
	locals {		locals {
Line 189:		Line 189:
	all_names = concat([local.driver_name], local.worker_names)		all_names = concat([local.driver_name], local.worker_names)
	}		}
	Inside strings, you can refer to ~~the variables using~~ ''${local.var_name}''. Outside of ~~string~~, you can write just ''local.var_name''.		Inside strings, you can refer to a variable as ''${local.var_name}''. Outside of strings, you can write just ''local.var_name''.

	=== Create and attach volumes ===		=== Create and attach volumes ===
Line 201:		Line 201:
	}		}

	Log in with SSH and to '''ls /dev''' to see that the volumes are attached where they should (but not yet partitioned, formatted, and mounted).		Log in with SSH and to '''ls /dev''' to see that the volumes are attached where they should (but they are not yet partitioned, formatted, and mounted).

	=== Create ''hosts'' files ===		=== Create ''hosts'' files ===

Sinoa: /* Test login */

2022-10-31T13:23:00Z

Test login

← Older revision		Revision as of 13:23, 31 October 2022
Line 164:		Line 164:
	openstack server list # or OS_CLOUD=info319-cluster openstack ...		openstack server list # or OS_CLOUD=info319-cluster openstack ...
	to find the IPv4 and IPv6 addresses of ''terraform-test''. Log in with for example:		to find the IPv4 and IPv6 addresses of ''terraform-test''. Log in with for example:
	ssh -i ~/.ssh/info319-spark-cluster ubuntu@158.37.65.222		ssh -i ~/.ssh/info319-spark-cluster ubuntu@158.37.165.222
	and		and
	ssh -i ~/.ssh/info319-spark-cluster -J YOUR_USERNAME@login.uib.no ubuntu@2001:700:2:8300::~~22f4~~		ssh -i ~/.ssh/info319-spark-cluster -J YOUR_USERNAME@login.uib.no ubuntu@2001:700:2:8300::21f4

	=== Create the Spark driver ===		=== Create the Spark driver ===

Sinoa: /* Test login */

2022-10-31T13:22:26Z

Test login

← Older revision		Revision as of 13:22, 31 October 2022
Line 162:		Line 162:
	=== Test login ===		=== Test login ===
	Use		Use
	openstack server list		openstack server list # or OS_CLOUD=info319-cluster openstack ...
	to find the IPv4 and IPv6 addresses of ''terraform-test''. Log in with for example:		to find the IPv4 and IPv6 addresses of ''terraform-test''. Log in with for example:
	ssh -i ~/.ssh/info319-spark-cluster ubuntu@158.37.65.222		ssh -i ~/.ssh/info319-spark-cluster ubuntu@158.37.65.222

Sinoa: /* ~/.config/openstack/clouds.yaml (optional) */

2022-10-31T13:21:15Z

~/.config/openstack/clouds.yaml (optional)

← Older revision		Revision as of 13:21, 31 October 2022
Line 145:		Line 145:
	}		}
	The advantage of this setup is that it easier manage multiple clusters from the same local machine. You do not need to run the ''keystonerc.sh'' script, and you avoid problems with changed environment variables.		The advantage of this setup is that it easier manage multiple clusters from the same local machine. You do not need to run the ''keystonerc.sh'' script, and you avoid problems with changed environment variables.

			''Note:'' There are now two ways to run '''openstack'''. One assumes that all the ''OS_*'' environment variables defined in ''keystonerc.sh'' are set. The other assumes that there is a ''~/.config/openstack/clouds.yaml'' file and that ''OS_CLOUD'' is set, for example:
			OS_CLOUD=info319-cluster openstack server list

	== Spark cluster ==		== Spark cluster ==

Sinoa: /* Spark cluster */

2022-10-31T13:20:49Z

Spark cluster

← Older revision		Revision as of 13:20, 31 October 2022
Line 147:		Line 147:

	== Spark cluster ==		== Spark cluster ==
	Use Terraform to create a Spark cluster, similar to the one in Exercise 4. The ''Resources'' menu in the left pane of [https://registry.terraform.io/providers/terraform-provider-openstack/openstack/latest/docs this page] ~~gives you details~~.		Use Terraform to create a Spark cluster, similar to the one in Exercise 4. The ''Resources'' menu in the left pane of [https://registry.terraform.io/providers/terraform-provider-openstack/openstack/latest/docs this page] lists all the available resources, but it is easier to list the most relevant ones with '''openstack ''resource_type'' list'''.

	=== Create or import a key pair ===		=== Create or import a key pair ===
Line 155:		Line 155:
	key_pair = "info319-spark-cluster"		key_pair = "info319-spark-cluster"

	Rerun '''terraform plan''' and '''terraform ~~apply~~''' continuously to check that things work.		Rerun '''terraform plan''', '''terraform apply''', and sometimes '''terraform destroy''' continuously as you build the cluster to check that things work.

	=== Test login ===		=== Test login ===
Line 164:		Line 164:
	and		and
	ssh -i ~/.ssh/info319-spark-cluster -J YOUR_USERNAME@login.uib.no ubuntu@2001:700:2:8300::22f4		ssh -i ~/.ssh/info319-spark-cluster -J YOUR_USERNAME@login.uib.no ubuntu@2001:700:2:8300::22f4

	''Note:'' There are two ways to run '''openstack'''. One assumes that all the ''OS_*'' environment variables defined in ''keystonerc.sh'' are set. The other assumes that there is a ''~/.config/openstack/clouds.yaml'' file and that ''OS_CLOUD'' is set, for example:
	~~OS_CLOUD=info319-cluster openstack server list~~

	=== Create the Spark driver ===		=== Create the Spark driver ===

Sinoa: /* ~/.config/openstack/clouds.yaml (optional) */

2022-10-31T13:10:05Z

~/.config/openstack/clouds.yaml (optional)

← Older revision		Revision as of 13:10, 31 October 2022
Line 144:		Line 144:
	cloud = "info319-cluster" # defined in a clouds.yaml file		cloud = "info319-cluster" # defined in a clouds.yaml file
	}		}
	The advantage of this setup is that it easier manage multiple clusters from the same local machine, and you avoid problems with changed ~~enrivonment~~ variables.		The advantage of this setup is that it easier manage multiple clusters from the same local machine. You do not need to run the ''keystonerc.sh'' script, and you avoid problems with changed environment variables.

	== Spark cluster ==		== Spark cluster ==

Sinoa: /* Test run Terraform */

2022-10-31T13:02:19Z

Test run Terraform

← Older revision		Revision as of 13:02, 31 October 2022
Line 90:		Line 90:
	Try to run Terraform with:		Try to run Terraform with:
	terraform plan		terraform plan
	'''terraform plan''' lists everything ''terraform'' will do if you ''apply'' it. This list is important to check so you do not permanently delete something critical, like disks/volumes.		'''terraform plan''' lists everything ''terraform'' will do if you ''apply'' it. This list is important to check so you do not permanently delete something critical, like disks/volumes with important data on them...

	To create the new test instance:		To create the new test instance:
	terraform apply # answer 'yes' to approve		terraform apply # answer 'yes' to approve
	Check ''Compute -> Instances'' in the ''NREC Overview'' to see the new instance appear.		Check ''Compute -> Instances'' in the ''NREC Overview'' to see the new instance appear.

			(You can also run '''terraform plan -out plan.tf''' to save the plan to a file and then run it faster with '''terraform apply plan.tf'''.)

	=== user-data.cfg ===		=== user-data.cfg ===