Configure Spark cluster using Ansible - Revision history

Sinoa: /* Install Zookeeper on the cluster */

2022-10-31T13:55:23Z

Install Zookeeper on the cluster

← Older revision		Revision as of 13:55, 31 October 2022
Line 152:		Line 152:
	As for the first point, an easy solution is to run Zookeeper on all hosts if the number is odd, but only on the workers if the total number is even.		As for the first point, an easy solution is to run Zookeeper on all hosts if the number is odd, but only on the workers if the total number is even.

	As for the second point, Exercise 4 suggested shell commands you can use to manage zookeeper ids. But Ansible also has powerful [https://docs.ansible.com/ansible/latest/user_guide/playbooks_loops.html loop constructs] you can ~~use~~.		As for the second point, Exercise 4 has already suggested shell commands you can use to manage zookeeper ids. But Ansible also has powerful [https://docs.ansible.com/ansible/latest/user_guide/playbooks_loops.html loop constructs] you can explore.

	~~This~~ task will start Zookeeper on the selected nodes:		In the end, this task will start Zookeeper on the selected nodes:
	- name: Start Zookeper		- name: Start Zookeper
	ansible.builtin.shell:		ansible.builtin.shell:

Sinoa: /* Install HDFS and YARN */

2022-10-31T13:52:21Z

Install HDFS and YARN

← Older revision		Revision as of 13:52, 31 October 2022
Line 106:		Line 106:
	Use the ''ansible.builtin.unarchive'' module to unpack the archives. Use the ''apt'' module to install ''gzip'' if you need it. Use ''ansible.builtin.file'' to create symbolic links as in Exercise 4.		Use the ''ansible.builtin.unarchive'' module to unpack the archives. Use the ''apt'' module to install ''gzip'' if you need it. Use ''ansible.builtin.file'' to create symbolic links as in Exercise 4.

			=== Configure HDFS and YARN ===
	Use ''ansible.builtin.lineinfile'' to define environment variables by adding them to ''~/.info319'' (instead of ''~/.bashrc'').		Use ''ansible.builtin.lineinfile'' to define environment variables by adding them to ''~/.info319'' (instead of ''~/.bashrc'').

Sinoa: /* Install HDFS and YARN */

2022-10-31T13:51:39Z

Install HDFS and YARN

← Older revision		Revision as of 13:51, 31 October 2022
Line 102:		Line 102:
	Write two corresponding tasks for ''num_workers''.		Write two corresponding tasks for ''num_workers''.

	Use the ''ansible.builtin.get_url'' module to download the Hadoop and ~~later Spark~~ archives directly to each cluster host. But if you re-run your script many times, it takes time and can trigger rate limitations in the download hosts. If so, it is better to let Ansible download each archive once to your local machine and then copy it onto the hosts as needed.		Use the ''ansible.builtin.get_url'' module to download the Hadoop (and other) archives directly to each cluster host. But if you re-run your script many times, this takes time and can trigger rate limitations in the download hosts. If so, it is better to let Ansible download each archive once to your local machine and then copy it onto the hosts as needed.

	Use the ''ansible.builtin.unarchive'' module to unpack the archives. Use the ''apt'' module to install ''gzip'' if you need it. Use ''ansible.builtin.file'' to create symbolic links as in Exercise 4.		Use the ''ansible.builtin.unarchive'' module to unpack the archives. Use the ''apt'' module to install ''gzip'' if you need it. Use ''ansible.builtin.file'' to create symbolic links as in Exercise 4.
Line 123:		Line 123:
	</property>		</property>
	</configuration>		</configuration>
	You can now use the ''ansible.builtin.template'' module instead of Linux' ''envsubst'' command to configure the files. Note that the files use the {{ hadoop_namenode } } variable. It is not defined yet, but should have the same value as {{ master_node } }.		You can now use the ''ansible.builtin.template'' module (instead of Linux' ''envsubst'' command) to configure the files. Note that the files use the {{ hadoop_namenode } } variable. It is not defined yet, but should have the same value as {{ master_node } }.

	Use ''ansible.builtin.shell'' to create Hadoop's ''worker'' and ''master'' files as in Execise 4, and to format the HDFS namenode and create the HDFS root folder on the ''terraform-driver'' host.		Use ''ansible.builtin.shell'' to create Hadoop's ''worker'' and ''master'' files as in Execise 4, and to format the HDFS namenode and create the HDFS root folder on the ''terraform-driver'' host.

Sinoa: /* Mount volumes */

2022-10-31T13:49:33Z

Mount volumes

← Older revision		Revision as of 13:49, 31 October 2022
Line 83:		Line 83:

	=== Mount volumes ===		=== Mount volumes ===
	~~Using~~ the ''community.general.parted'', ''community.general.filesystem'' and ''ansible.posix.mount'' modules ~~from~~ your local machine ~~may require installation~~:		Use the ''community.general.parted'', ''community.general.filesystem'' and ''ansible.posix.mount'' modules to do this. They may require installation on your local machine:
	ansible-galaxy collection install community.general		ansible-galaxy collection install community.general
	ansible-galaxy collection install ansible.posix		ansible-galaxy collection install ansible.posix

Sinoa: /* Configure SSH */

2022-10-31T13:47:34Z

Configure SSH

← Older revision		Revision as of 13:47, 31 October 2022
Line 55:		Line 55:
	path: /etc/hosts		path: /etc/hosts
	block: "{{ lookup('file', 'ipv4-hosts') } }"		block: "{{ lookup('file', 'ipv4-hosts') } }"
	become: yes		become: yes # because you need root privilege (sudo) to update /etc/hosts

	''Note:'' There should not be a space between the two curly braces at the end of the ''key:'' line. But without the space, WikiText misinterprets them as a template marker.		''Note:'' There should not be a space between the two curly braces at the end of the ''key:'' line. But without the space, WikiText misinterprets them as a template marker.
Line 77:		Line 77:
	user: ubuntu		user: ubuntu

	''Tip:'' '''ansible-playbook''' has a '''--start-at-task "Task name"''' option to avoid repeating all earlier blocks and stages. You can also use '''--step''' to have Ansible ask before each step whether to execute, skip~~, or finish~~.		''Tip:'' '''ansible-playbook''' has a '''--start-at-task "Task name"''' option you can use to avoid repeating all earlier plays (blocks) and stages. You can also use '''--step''' to have Ansible ask before each step whether to execute or skip it.

	=== Install Java ===		=== Install Java ===

Sinoa: /* Preparing .bashrc */

2022-10-31T13:45:05Z

Preparing .bashrc

← Older revision		Revision as of 13:45, 31 October 2022
Line 43:		Line 43:

	* Add a task that uses the ''ansible.builtin.file'' module to create ("touch") ''/home/ubuntu/.info319'' on all the hosts.		* Add a task that uses the ''ansible.builtin.file'' module to create ("touch") ''/home/ubuntu/.info319'' on all the hosts.
	* Add a task that uses the ''ansible.builtin.lineinfile'' module to add this line to ''/home/ubuntu/.~~info319~~'' on all the hosts:		* Add a task that uses the ''ansible.builtin.lineinfile'' module to add this line to the end of ''/home/ubuntu/.bashrc'' on all the hosts:
	source .info319		source .info319

Sinoa: /* Create Ansible playbook */

2022-10-31T13:44:17Z

Create Ansible playbook

← Older revision		Revision as of 13:44, 31 October 2022
Line 36:		Line 36:
	ansible-playbook info319-cluster.yaml		ansible-playbook info319-cluster.yaml

	== ~~Create~~ Ansible playbook ==		== Ansible playbook ==
	Extend the playbook file ''info319-cluster.yaml'' to re-create the setup from Exercise 4 on the new Terraform cluster. You have some freedom with respect to the order of things.		Extend the playbook file ''info319-cluster.yaml'' to re-create the setup from Exercise 4 on the new Terraform cluster. You have some freedom with respect to the order of things.

Sinoa: /* Test run Ansible */

2022-10-31T13:43:23Z

Test run Ansible

← Older revision		Revision as of 13:43, 31 October 2022
Line 22:		Line 22:
	ansible all -m ping		ansible all -m ping

	On your local machine, create the file ''info319-cluster.yaml'' with a simple task:		On your local machine, create the file ''info319-cluster.yaml'' with a simple task that backs up ''~/.bashrc'':
	- name: Prepare .bashrc		- name: Prepare .bashrc
	hosts: all		hosts: all

Sinoa: /* Test run Ansible */

2022-10-31T13:42:46Z

Test run Ansible

← Older revision		Revision as of 13:42, 31 October 2022
Line 23:		Line 23:

	On your local machine, create the file ''info319-cluster.yaml'' with a simple task:		On your local machine, create the file ''info319-cluster.yaml'' with a simple task:
	- name: Prepare .bashrc		- name: Prepare .bashrc
	hosts: all		hosts: all
	tasks:		tasks:

	- name: Save original .bashrc		- name: Save original .bashrc
	ansible.builtin.copy:		ansible.builtin.copy:
	src: /home/ubuntu/.bashrc		src: /home/ubuntu/.bashrc
	dest: /home/ubuntu/.bashrc.original		dest: /home/ubuntu/.bashrc.original
	remote_src: yes		remote_src: yes

	Run Ansible:		Run Ansible:

Sinoa: /* Test run Ansible */

2022-10-31T13:42:18Z

Test run Ansible

← Older revision		Revision as of 13:42, 31 October 2022
Line 19:		Line 19:

	=== Test run Ansible ===		=== Test run Ansible ===
	Make sure you can log into your cluster machines without a password. Test your Ansible ~~set up~~ from your local machine:		Make sure you can log into your cluster machines without a password. Test your Ansible setup from your local machine:
	ansible all -m ping		ansible all -m ping