Install HDFS and YARN on the cluster - Revision history

Sinoa: /* Web UIs */

2022-10-31T12:10:20Z

Web UIs

← Older revision		Revision as of 12:10, 31 October 2022
Line 173:		Line 173:
	== Web UIs ==		== Web UIs ==
	If you have opened the ports on ''spark-driver'', you can now access		If you have opened the ports on ''spark-driver'', you can now access
	* HDFS' web UI at http://158.39.77.~~125~~:9870		* HDFS' web UI at http://158.39.77.103:9870
	* YARN's web UI at http://158.39.77.~~125~~:8088		* YARN's web UI at http://158.39.77.103:8088
	(assuming that 158.39.~~201~~.~~197~~ is the IPv4 address of ''spark-driver'').		(assuming that 158.39.77.103 is the IPv4 address of ''spark-driver'').

Sinoa: /* Run HDFS on the cluster */

2022-10-31T12:07:44Z

Run HDFS on the cluster

← Older revision		Revision as of 12:07, 31 October 2022
Line 147:		Line 147:
	hdfs namenode -format # answer 'Y' if you are asked about reformatting		hdfs namenode -format # answer 'Y' if you are asked about reformatting
	start-dfs.sh		start-dfs.sh
			If you run into trouble, you can look in ''volume/hadoop/logs/'' for clues. The ''tail'' command is your friend. The ''ls -lt'' command lists files sorted by modification time.

	On any instance, test it with		On any instance, test that HDFS is working with
	hdfs dfs -mkdir -p /user/ubuntu/raw-data		hdfs dfs -mkdir -p /user/ubuntu/raw-data
	hdfs dfs -ls /user/ubuntu/ # on any machine in the spark-* cluster		hdfs dfs -ls /user/ubuntu/ # on any machine in the spark-* cluster

Sinoa: /* Install Hadoop */

2022-10-31T12:01:43Z

Install Hadoop

← Older revision		Revision as of 12:01, 31 October 2022
Line 50:		Line 50:
	mkdir -p ${HDFS_DATA} # this is where HDFS will store its data		mkdir -p ${HDFS_DATA} # this is where HDFS will store its data

			== Configure Hadoop ==
	On each instance, extract further environment variables from ''/etc/hosts'':		On each instance, extract further environment variables from ''/etc/hosts'':
	export MASTER_NODE=$(grep spark-driver /etc/hosts \| cut -d" " -f1)		export MASTER_NODE=$(grep spark-driver /etc/hosts \| cut -d" " -f1)
Line 126:		Line 127:
	scp mapred-site.xml spark-driver: # mapred-site.xml is only needed on the driver		scp mapred-site.xml spark-driver: # mapred-site.xml is only needed on the driver

	On each instance:		On each instance, use the ''envsubst'' command to substitute variables names like ''${NUM_WORKERS}'' in the xml-files with their values in the local environment:
	for name in core hdfs yarn mapred; do		for name in core hdfs yarn mapred; do
	cp ${HADOOP_CONF}/${name}-site.xml ${HADOOP_CONF}/${name}-site.xml.original;		cp ${HADOOP_CONF}/${name}-site.xml ${HADOOP_CONF}/${name}-site.xml.original;

Sinoa: /* Web UIs */

2022-10-17T11:28:53Z

Web UIs

← Older revision		Revision as of 11:28, 17 October 2022
Line 173:		Line 173:
	* HDFS' web UI at http://158.39.77.125:9870		* HDFS' web UI at http://158.39.77.125:9870
	* YARN's web UI at http://158.39.77.125:8088		* YARN's web UI at http://158.39.77.125:8088
			(assuming that 158.39.201.197 is the IPv4 address of ''spark-driver'').

Sinoa at 11:06, 17 October 2022

2022-10-17T11:06:55Z

← Older revision		Revision as of 11:06, 17 October 2022
Line 1:		Line 1:
	== Install Java ==		== Install Java ==
	You need Java on each instance. We will use an older, stable version in case some of the tools are not upgraded to more recent versions:		You need Java on each instance. We will use an older, stable version in case some of the tools are not upgraded to more recent versions:
			sudo apt update # ('sudo apt upgrade' is also a good idea on a new instance)
	sudo apt install -y openjdk-8-jdk-headless		sudo apt install -y openjdk-8-jdk-headless
	export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64		export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
Line 20:		Line 21:

	On each instance, set the environment variable HADOOP_HOME and add Hadoop's bin folder to your PATH:		On each instance, set the environment variable HADOOP_HOME and add Hadoop's bin folder to your PATH:
	export HADOOP_HOME=/home/ubuntu/volume/hadoop~~-3.3.4~~		export HADOOP_HOME=/home/ubuntu/volume/hadoop
	export PATH=${PATH}:${HADOOP_HOME}/bin:${HADOOP_HOME}/sbin		export PATH=${PATH}:${HADOOP_HOME}/bin:${HADOOP_HOME}/sbin
	cp ~/.bashrc ~/.bashrc-bkp-$(date --iso-8601=minutes --utc)		cp ~/.bashrc ~/.bashrc-bkp-$(date --iso-8601=minutes --utc)
	echo "export HADOOP_HOME=/home/ubuntu/volume/hadoop~~-3.3.4~~" >> ~/.bashrc		echo "export HADOOP_HOME=/home/ubuntu/volume/hadoop" >> ~/.bashrc
	echo "export PATH=\${PATH}:\${HADOOP_HOME}/bin:\${HADOOP_HOME}/sbin" >> ~/.bashrc		echo "export PATH=\${PATH}:\${HADOOP_HOME}/bin:\${HADOOP_HOME}/sbin" >> ~/.bashrc

Line 46:		Line 47:
	echo "export LD_LIBRARY_PATH=\${HADOOP_HOME}/lib/native:\${LD_LIBRARY_PATH}" >> ~/.bashrc		echo "export LD_LIBRARY_PATH=\${HADOOP_HOME}/lib/native:\${LD_LIBRARY_PATH}" >> ~/.bashrc
	echo "export HDFS_DATA=\${HADOOP_HOME}/hdfs/data" >> ~/.bashrc		echo "export HDFS_DATA=\${HADOOP_HOME}/hdfs/data" >> ~/.bashrc

	mkdir -p ${HDFS_DATA} # this is where HDFS will store its data		mkdir -p ${HDFS_DATA} # this is where HDFS will store its data

Line 123:		Line 124:
	scp {core,hdfs,yarn}-site.xml spark-worker-1:		scp {core,hdfs,yarn}-site.xml spark-worker-1:
	...		...
	scp mapred-site.xml spark-driver: # it is only needed on ~~master~~		scp mapred-site.xml spark-driver: # mapred-site.xml is only needed on the driver

	On each instance:		On each instance:
Line 143:		Line 144:
	== Run HDFS on the cluster ==		== Run HDFS on the cluster ==
	On ''spark-driver'':		On ''spark-driver'':
	hdfs namenode -format # answer 'Y' to reformatting		hdfs namenode -format # answer 'Y' if you are asked about reformatting
	start-dfs.sh		start-dfs.sh

Line 151:		Line 152:

	From your local machine (this file is available at mitt.uib.no):		From your local machine (this file is available at mitt.uib.no):
	scp head-100000-latest-all.json spark-~~driver~~:		scp head-100000-latest-all.json spark-ANY-INSTANCE:

	On ~~any instance~~:		On spark-ANY-INSTANCE:
	hdfs dfs -put ~/head-100000-latest-all.json raw-data		hdfs dfs -put ~/head-100000-latest-all.json raw-data

	~~Stop~~ HDFS again:		From another instance, use ''hdfs dfs -ls ...'' to check that the file is there.

			On ''spark-driver'', stop HDFS again:
	stop-dfs.sh		stop-dfs.sh

	== Run YARN ~~and HDFS~~ on the cluster ==		== Run HDFS and YARN on the cluster ==
	You are now ready to start YARN and HDFS together. On ''spark-drive'' run:		You are now ready to start YARN and HDFS together. On ''spark-drive'' run:
	start-all.sh		start-all.sh
Line 165:		Line 168:
	When you want to stop YARN and HDFS, use:		When you want to stop YARN and HDFS, use:
	stop-all.sh		stop-all.sh

			== Web UIs ==
			If you have opened the ports on ''spark-driver'', you can now access
			* HDFS' web UI at http://158.39.77.125:9870
			* YARN's web UI at http://158.39.77.125:8088

Sinoa at 10:31, 17 October 2022

2022-10-17T10:31:18Z

← Older revision		Revision as of 10:31, 17 October 2022
Line 9:		Line 9:
	* ''export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64:'' sets an environment variable ''temporarily''		* ''export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64:'' sets an environment variable ''temporarily''
	* ''cp ~/.bashrc ~/.bashrc-bkp-$(date --iso-8601=minutes --utc):'' creates a time-stamped backup of the initialisation file ''~/.bashrc''		* ''cp ~/.bashrc ~/.bashrc-bkp-$(date --iso-8601=minutes --utc):'' creates a time-stamped backup of the initialisation file ''~/.bashrc''
	* ''echo "export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64" >> ~/.bashrc:'' adds the ''export'' command to the ~~initialisaiton~~ so the environment variable becomes permanent		* ''echo "export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64" >> ~/.bashrc:'' adds the ''export'' command to the initialisation so the environment variable becomes permanent

	== Install Hadoop ==		== Install Hadoop ==

Sinoa: /* Install Java */

2022-10-12T19:05:22Z

Install Java

← Older revision		Revision as of 19:05, 12 October 2022
Line 1:		Line 1:
	== Install Java ==		== Install Java ==
	You need Java on each instance. We will use ~~and~~ older, stable version in case some of the tools are not upgraded to more recent versions:		You need Java on each instance. We will use an older, stable version in case some of the tools are not upgraded to more recent versions:
	sudo apt install -y openjdk-8-jdk-headless		sudo apt install -y openjdk-8-jdk-headless
	export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64		export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64

Sinoa: Created page with "== Install Java == You need Java on each instance. We will use and older, stable version in case some of the tools are not upgraded to more recent versions: sudo apt install..."

2022-10-12T11:02:56Z

Created page with "== Install Java == You need Java on each instance. We will use and older, stable version in case some of the tools are not upgraded to more recent versions: sudo apt install..."

New page

== Install Java ==
You need Java on each instance. We will use and older, stable version in case some of the tools are not upgraded to more recent versions:
sudo apt install -y openjdk-8-jdk-headless
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
cp ~/.bashrc ~/.bashrc-bkp-$(date --iso-8601=minutes --utc)
echo "export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64" >> ~/.bashrc

Explanation:
* ''export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64:'' sets an environment variable ''temporarily''
* ''cp ~/.bashrc ~/.bashrc-bkp-$(date --iso-8601=minutes --utc):'' creates a time-stamped backup of the initialisation file ''~/.bashrc''
* ''echo "export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64" >> ~/.bashrc:'' adds the ''export'' command to the initialisaiton so the environment variable becomes permanent

== Install Hadoop ==
On each instance, go to http://hadoop.apache.org and download a recent binary distribution to ''~/volume''. For example:
cd ~/volume
wget https://dlcdn.apache.org/hadoop/common/hadoop-3.3.4/hadoop-3.3.4.tar.gz
tar xzvf hadoop-3.3.4.tar.gz
rm hadoop-3.3.4.tar.gz
ln -fns hadoop-3.3.4 hadoop

On each instance, set the environment variable HADOOP_HOME and add Hadoop's bin folder to your PATH:
export HADOOP_HOME=/home/ubuntu/volume/hadoop-3.3.4
export PATH=${PATH}:${HADOOP_HOME}/bin:${HADOOP_HOME}/sbin
cp ~/.bashrc ~/.bashrc-bkp-$(date --iso-8601=minutes --utc)
echo "export HADOOP_HOME=/home/ubuntu/volume/hadoop-3.3.4" >> ~/.bashrc
echo "export PATH=\${PATH}:\${HADOOP_HOME}/bin:\${HADOOP_HOME}/sbin" >> ~/.bashrc

If something goes wrong with a file update, install a text editor to fix it, such as '''sudo apt emacs''' or '''sudo apt nano'''.

On each instance, set additional environment variables for Hadoop:
export HADOOP_MAPRED_HOME=${HADOOP_HOME}
export HADOOP_COMMON_HOME=${HADOOP_HOME}
export HADOOP_HDFS_HOME=${HADOOP_HOME}
export YARN_HOME=${HADOOP_HOME}
export HADOOP_CONF=${HADOOP_HOME}/etc/hadoop
export HADOOP_CONF_DIR=${HADOOP_CONF}
export LD_LIBRARY_PATH=${HADOOP_HOME}/lib/native:${LD_LIBRARY_PATH}
export HDFS_DATA=${HADOOP_HOME}/hdfs/data
cp ~/.bashrc ~/.bashrc-bkp-$(date --iso-8601=minutes --utc)
echo "export HADOOP_MAPRED_HOME=\${HADOOP_HOME}" >> ~/.bashrc
echo "export HADOOP_COMMON_HOME=\${HADOOP_HOME}" >> ~/.bashrc
echo "export HADOOP_HDFS_HOME=\${HADOOP_HOME}" >> ~/.bashrc
echo "export YARN_HOME=\${HADOOP_HOME}" >> ~/.bashrc
echo "export HADOOP_CONF=\${HADOOP_HOME}/etc/hadoop" >> ~/.bashrc
echo "export HADOOP_CONF_DIR=\${HADOOP_CONF}" >> ~/.bashrc
echo "export LD_LIBRARY_PATH=\${HADOOP_HOME}/lib/native:\${LD_LIBRARY_PATH}" >> ~/.bashrc
echo "export HDFS_DATA=\${HADOOP_HOME}/hdfs/data" >> ~/.bashrc

mkdir -p ${HDFS_DATA} # this is where HDFS will store its data

On each instance, extract further environment variables from ''/etc/hosts'':
export MASTER_NODE=$(grep spark-driver /etc/hosts | cut -d" " -f1)
export HADOOP_NAMENODE=${MASTER_NODE}
export NUM_WORKERS=$(grep spark- /etc/hosts | wc -l | cut -d" " -f1)
cp ~/.bashrc ~/.bashrc-bkp-$(date --iso-8601=minutes --utc)
echo "export MASTER_NODE=\$(grep spark-driver /etc/hosts | cut -d\" \" -f1)" >> ~/.bashrc
echo "export HADOOP_NAMENODE=\${MASTER_NODE}" >> ~/.bashrc
echo "export NUM_WORKERS=\$(grep spark- /etc/hosts | wc -l | cut -d\" \" -f1)" >> ~/.bashrc

Create these four files on your local computer:

File "core-site.xml":

<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://${HADOOP_NAMENODE}:9000</value>
</property>
</configuration>

File "hdfs-site.xml":

<configuration>
<property>
<name>dfs.replication</name>
<value>${NUM_WORKERS}</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file://${HDFS_DATA}</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file://${HDFS_DATA}</value>
</property>
</configuration>

File "yarn-site.xml":

<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>${HADOOP_NAMENODE}</value>
</property>
</configuration>

File mapred-site.xml:

<configuration>
<property>
<name>mapreduce.jobtracker.address</name>
<value>${HADOOP_NAMENODE}</value>
</property>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>

On your local machine:
scp {core,hdfs,yarn}-site.xml spark-driver:
scp {core,hdfs,yarn}-site.xml spark-worker-1:
...
scp mapred-site.xml spark-driver: # it is only needed on master

On each instance:
for name in core hdfs yarn mapred; do
cp ${HADOOP_CONF}/${name}-site.xml ${HADOOP_CONF}/${name}-site.xml.original;
done
cd ~
for name in *-site.xml; do
envsubst < ${name} > ${HADOOP_CONF}/${name};
done
rm ~/*-site.xml

On each instance, define the master and worker IPv4 addresses, and do a few other preparations:
grep spark-driver /etc/hosts | cut -d" " -f2 > ${HADOOP_CONF}/masters
grep spark- /etc/hosts | cut -d" " -f2 > ${HADOOP_CONF}/workers
echo "JAVA_HOME=$JAVA_HOME" >> $HADOOP_CONF/hadoop-env.sh
mkdir -p -m 0777 $HADOOP_HOME/logs

== Run HDFS on the cluster ==
On ''spark-driver'':
hdfs namenode -format # answer 'Y' to reformatting
start-dfs.sh

On any instance, test it with
hdfs dfs -mkdir -p /user/ubuntu/raw-data
hdfs dfs -ls /user/ubuntu/ # on any machine in the spark-* cluster

From your local machine (this file is available at mitt.uib.no):
scp head-100000-latest-all.json spark-driver:

On any instance:
hdfs dfs -put ~/head-100000-latest-all.json raw-data

Stop HDFS again:
stop-dfs.sh

== Run YARN and HDFS on the cluster ==
You are now ready to start YARN and HDFS together. On ''spark-drive'' run:
start-all.sh

When you want to stop YARN and HDFS, use:
stop-all.sh