Install Kafka on the cluster - Revision history

Sinoa: /* Test run Kafka */

2022-12-01T11:15:11Z

Test run Kafka

← Older revision		Revision as of 11:15, 1 December 2022
Line 99:		Line 99:

	Run these two commands on different instances to see that lines you type into the ''producer'' console show up in the ''consumer'':		Run these two commands on different instances to see that lines you type into the ''producer'' console show up in the ''consumer'':
	~~bin/~~kafka-console-producer.sh --topic test --bootstrap-server ${MASTER_NODE}:9092		kafka-console-producer.sh --topic test --bootstrap-server ${MASTER_NODE}:9092
	~~bin/~~kafka-console-consumer.sh --topic test --bootstrap-server ${MASTER_NODE}:9092 --from-beginning		kafka-console-consumer.sh --topic test --bootstrap-server ${MASTER_NODE}:9092 --from-beginning

	'''Task 2:''' Run the full Twitter pipeline from Exercise 3. Tips:		'''Task 2:''' Run the full Twitter pipeline from Exercise 3. Tips:

2022-12-01T11:13:29Z

Configure Kafka

← Older revision		Revision as of 11:13, 1 December 2022
Line 75:		Line 75:
	On each instance, configure Kafka:		On each instance, configure Kafka:
	cp ~/volume/kafka/config/server.properties ~/volume/kafka/config/server.properties.original		cp ~/volume/kafka/config/server.properties ~/volume/kafka/config/server.properties.original
	sed -i "s/broker.id=0/broker.id=$(cat /tmp/zookeeper/myid)/g" ~/volume/kafka/config/server.properties		sed -i "s/broker.id=/broker.id=$(cat /tmp/zookeeper/myid)/g" ~/volume/kafka/config/server.properties
	export ZOOKEEPER_HOSTLIST=$(grep spark- /etc/hosts \| cut -d" " -f1 \| head -3 \| paste -sd,)		export ZOOKEEPER_HOSTLIST=$(grep spark- /etc/hosts \| cut -d" " -f1 \| head -3 \| paste -sd,)
	echo "export ZOOKEEPER_HOSTLIST=\$(grep spark- /etc/hosts \| cut -d\" \" -f1 \| head -3 \| paste -sd,)" >> ~/.bashrc		echo "export ZOOKEEPER_HOSTLIST=\$(grep spark- /etc/hosts \| cut -d\" \" -f1 \| head -3 \| paste -sd,)" >> ~/.bashrc

2022-12-01T11:09:03Z

Test run zookeeper

← Older revision		Revision as of 11:09, 1 December 2022
Line 57:		Line 57:

	To stop zookeeper, do		To stop zookeeper, do
	zkServer.sh ~~status~~ ${ZOOKEEPER_HOME}/conf/zookeeper.properties		zkServer.sh stop ${ZOOKEEPER_HOME}/conf/zookeeper.properties
	on each host.		on each host.

2022-10-31T12:35:42Z

Test run Kafka

← Older revision		Revision as of 12:35, 31 October 2022
Line 103:		Line 103:

	'''Task 2:''' Run the full Twitter pipeline from Exercise 3. Tips:		'''Task 2:''' Run the full Twitter pipeline from Exercise 3. Tips:
	* You ~~need~~ to create virtual environment to ~~download~~ packages~~, specifically~~ ''tweepy'', ''kafka-python'', and ''afinn''.		* You may want to create a virtual environment to install Python packages such as ''tweepy'', ''kafka-python'', and ''afinn''.
	* You need to create ''keys/bearer_token'' with restricted access.		* You need to create ''keys/bearer_token'' with restricted access.
	* With the default settings, it can take time before the final join is written to console. Be patient...		* With the default settings, it can take time before the final join is written to console. Be patient...

2022-10-31T12:32:11Z

Web UIs

← Older revision		Revision as of 12:32, 31 October 2022
Line 123:		Line 123:

	== Web UIs ==		== Web UIs ==
	Assuming that 158.39.77.227 is the IPv4 address of ''spark-driver'', you can access ~~zookeeper~~'s very simple web UI at http://158.39.77.227:8080/commands/stat .		Assuming that 158.39.77.227 is the IPv4 address of ''spark-driver'', you can access Zookeeper's very simple web UI at http://158.39.77.227:8080/commands/stat .

	Kafka has no built in web UI, but third-party UIs are available. <!-- If you [https://docs.docker.com/engine/install/ install Docker] on ''spark-driver'', you can run something like:		Kafka has no built in web UI, but third-party UIs are available. <!-- If you [https://docs.docker.com/engine/install/ install Docker] on ''spark-driver'', you can run something like:

2022-10-31T12:30:33Z

Test run Kafka

← Older revision		Revision as of 12:30, 31 October 2022
Line 98:		Line 98:
	kafka-topics.sh --list --bootstrap-server ${MASTER_NODE}:9092		kafka-topics.sh --list --bootstrap-server ${MASTER_NODE}:9092

	Run these two commands on different instances:		Run these two commands on different instances to see that lines you type into the ''producer'' console show up in the ''consumer'':
	bin/kafka-console-producer.sh --topic test --bootstrap-server ${MASTER_NODE}:9092		bin/kafka-console-producer.sh --topic test --bootstrap-server ${MASTER_NODE}:9092
	bin/kafka-console-consumer.sh --topic test --bootstrap-server ${MASTER_NODE}:9092 --from-beginning		bin/kafka-console-consumer.sh --topic test --bootstrap-server ${MASTER_NODE}:9092 --from-beginning

2022-10-31T12:29:04Z

Test run Kafka

← Older revision		Revision as of 12:29, 31 October 2022
Line 94:		Line 94:
	(It is a good sign if nothing happens.)		(It is a good sign if nothing happens.)

	~~Test~~ commands to run on the different instances:		Here are some test commands you can run on the different instances:
	kafka-topics.sh --create --topic test --replication-factor 2 --partitions 3 --bootstrap-server ${MASTER_NODE}:9092		kafka-topics.sh --create --topic test --replication-factor 2 --partitions 3 --bootstrap-server ${MASTER_NODE}:9092
	kafka-topics.sh --list --bootstrap-server ${MASTER_NODE}:9092		kafka-topics.sh --list --bootstrap-server ${MASTER_NODE}:9092

2022-10-31T12:28:22Z

Install Kafka

← Older revision		Revision as of 12:28, 31 October 2022
Line 72:		Line 72:
	ln -fns kafka_2.12-3.3.1 kafka		ln -fns kafka_2.12-3.3.1 kafka

			== Configure Kafka ==
	On each instance, configure Kafka:		On each instance, configure Kafka:
	cp ~/volume/kafka/config/server.properties ~/volume/kafka/config/server.properties.original		cp ~/volume/kafka/config/server.properties ~/volume/kafka/config/server.properties.original

2022-10-31T12:27:25Z

Test run zookeeper

← Older revision		Revision as of 12:27, 31 October 2022
Line 54:		Line 54:
	This will start zookeeper in the background on each host. Afterwards, check its status:		This will start zookeeper in the background on each host. Afterwards, check its status:
	zkServer.sh status ${ZOOKEEPER_HOME}/conf/zookeeper.properties		zkServer.sh status ${ZOOKEEPER_HOME}/conf/zookeeper.properties
	On of the zookeepers is the leader, the two others are followers. In case of trouble, check ${ZOOKEEPER_HOME}/logs/ .		One of the zookeepers is the leader, the two others are followers. In case of trouble, check ${ZOOKEEPER_HOME}/logs/ .

	To stop zookeeper, do		To stop zookeeper, do

2022-10-31T12:24:37Z

Configure Zookeeper

← Older revision		Revision as of 12:24, 31 October 2022
Line 16:		Line 16:
	syncLimit=10		syncLimit=10

	From your local computer, upload		From your local computer, upload:
	scp zookeeper.properties spark-driver:		scp zookeeper.properties spark-driver:
	scp zookeeper.properties spark-worker-1:		scp zookeeper.properties spark-worker-1: