Streaming tweets with Kafka and Spark - Revision history

Sinoa: /* Building a distributed architecture */

2022-10-13T15:22:39Z

Building a distributed architecture

← Older revision		Revision as of 15:22, 13 October 2022
Line 43:		Line 43:

	== Building a distributed architecture ==		== Building a distributed architecture ==
	The following illustration shows one way to organise the solution to this exercise: [[File:Twitterpipe.png]]		The following illustration shows one way to organise the solution to this exercise:

			[[File:Twitterpipe.png]]

	== Receive tweets from Kafka with streaming Spark ==		== Receive tweets from Kafka with streaming Spark ==

Sinoa: /* Building a distributed architecture */

2022-10-13T15:21:27Z

Building a distributed architecture

← Older revision		Revision as of 15:21, 13 October 2022
Line 43:		Line 43:

	== Building a distributed architecture ==		== Building a distributed architecture ==
	The following illustration shows one way to organise the solution to this exercise:		The following illustration shows one way to organise the solution to this exercise: [[File:Twitterpipe.png]]

	== Receive tweets from Kafka with streaming Spark ==		== Receive tweets from Kafka with streaming Spark ==

Sinoa: /* Receive tweets from Kafka with streaming Spark */

2022-10-13T15:19:39Z

Receive tweets from Kafka with streaming Spark

← Older revision		Revision as of 15:19, 13 October 2022
Line 41:		Line 41:

	This guide presents more [https://towardsdatascience.com/3-libraries-you-should-know-to-master-apache-kafka-in-python-c95fdf8700f2 examples of using kafka-python].		This guide presents more [https://towardsdatascience.com/3-libraries-you-should-know-to-master-apache-kafka-in-python-c95fdf8700f2 examples of using kafka-python].

			== Building a distributed architecture ==
			The following illustration shows one way to organise the solution to this exercise:

	== Receive tweets from Kafka with streaming Spark ==		== Receive tweets from Kafka with streaming Spark ==

Sinoa: /* Streaming tweets to Kafka */

2022-10-13T11:50:03Z

Streaming tweets to Kafka

← Older revision		Revision as of 11:50, 13 October 2022
Line 23:		Line 23:
	== Streaming tweets to Kafka ==		== Streaming tweets to Kafka ==
	Create a Kafka topic called ''tweets'':		Create a Kafka topic called ''tweets'':
	$ bin/kafka-~~console-consumer~~.sh --topic tweets ~~--from-beginning~~ --bootstrap-server localhost:9092		$ bin/kafka-topics.sh --create --topic tweets --bootstrap-server localhost:9092

	'''Task 1:''' Go back to '''tweet_harvester.py''' from Exercise 2, and change the code to write the stream of tweets to a Kafka topic instead of a socket. Here are some basic steps and code pieces you will need:		'''Task 1:''' Go back to '''tweet_harvester.py''' from Exercise 2, and change the code to write the stream of tweets to a Kafka topic instead of a socket. Here are some basic steps and code pieces you will need:

Sinoa at 09:10, 13 October 2022

2022-10-13T09:10:58Z

← Older revision		Revision as of 09:10, 13 October 2022
Line 1:		Line 1:
	''This ~~exercise assumes~~ you are running Ubuntu Linux, either natively or through WSL 2.''		''This and later exercises assume you are running Ubuntu Linux, either natively or through WSL 2.''

	== Download and install Kafka ==		== Download and install Kafka ==

Sinoa: /* Spark join */

2022-10-12T15:00:08Z

Spark join

← Older revision		Revision as of 15:00, 12 October 2022
Line 182:		Line 182:
	# pick out the ''timestamp''-s in addition to the ''value''-s from the incoming Kafka streams of tweets and photos,		# pick out the ''timestamp''-s in addition to the ''value''-s from the incoming Kafka streams of tweets and photos,
	# rename the ''id'' and ''timestamp'' columns in the streaming photo DataFrame,		# rename the ''id'' and ''timestamp'' columns in the streaming photo DataFrame,
	# define the ''timestamp'' and (renamed) ''photo_timestamp'' columns as watermarks, for example with a 10 minute ~~treshold~~, and		# define the ''timestamp'' and (renamed) ''photo_timestamp'' columns as watermarks, for example with a 10 minute threshold, and
	# add an explicit '''outerLeft''' join expression over the ''id''-s and ''timestamp''-s.		# add an explicit '''outerLeft''' join expression over the ''id''-s and ''timestamp''-s.

	Tips:		Tips:
	* to pick out timestamps from incoming Kafka streams:		* to pick out timestamps from incoming Kafka streams, for example:
	.select(photo_sdf.timestamp,		.select(photo_sdf.timestamp,
	decode(photo_sdf.value, 'utf-8').alias('json_str')) \		decode(photo_sdf.value, 'utf-8').alias('json_str')) \
	* to rename columns:		* to rename columns, for example:
	.withColumnRenamed('timestamp', 'photo_timestamp') \		.withColumnRenamed('timestamp', 'photo_timestamp') \
	* to add a watermark:		* to add a watermark, for example:
	.withWatermark('photo_timestamp', '10 minutes')		.withWatermark('photo_timestamp', '10 minutes')

Sinoa: /* Spark join */

2022-10-12T14:57:58Z

Spark join

← Older revision		Revision as of 14:57, 12 October 2022
Line 177:		Line 177:
	'''Task 10:''' Add a Spark pipeline to read tweet ''id'' and photo ''size'' from the ''photos'' topic. Perform an '''inner''' join on the ''id'' column to combine the results of sentiment analysis and external photo analysis.		'''Task 10:''' Add a Spark pipeline to read tweet ''id'' and photo ''size'' from the ''photos'' topic. Perform an '''inner''' join on the ''id'' column to combine the results of sentiment analysis and external photo analysis.

	'''Task 11:''' The inner join in task 10 only returns tweets with photos. A ''left outer join'' of the sentiment analysis frame (left) and photo analysis frame (right) will return a Row for every ~~tweets~~, but with some null values for the tweets without photos.		'''Task 11:''' The inner join in task 10 only returns tweets with photos. A ''left outer join'' of the sentiment analysis frame (left) and photo analysis frame (right) will return a Row for every tweet, but with some null values for the tweets without photos.

	Perform such an '''outerLeft''' join on the tweet ids. To achieve this, you need to:		Perform such an '''outerLeft''' join on the tweet ids. To achieve this, you need to:

Sinoa: /* Spark join */

2022-10-12T14:57:34Z

Spark join

← Older revision		Revision as of 14:57, 12 October 2022
Line 177:		Line 177:
	'''Task 10:''' Add a Spark pipeline to read tweet ''id'' and photo ''size'' from the ''photos'' topic. Perform an '''inner''' join on the ''id'' column to combine the results of sentiment analysis and external photo analysis.		'''Task 10:''' Add a Spark pipeline to read tweet ''id'' and photo ''size'' from the ''photos'' topic. Perform an '''inner''' join on the ''id'' column to combine the results of sentiment analysis and external photo analysis.

	'''Task 11:''' The inner join in task 10 only returns tweets with photos. A ''left outer join'' of the sentiment analysis frame (left) and photo analysis frame (right) will return ~~all~~ tweets, with some null values for tweets without photos~~. Perform such an '''outerLeft''' join on the tweet ids~~.		'''Task 11:''' The inner join in task 10 only returns tweets with photos. A ''left outer join'' of the sentiment analysis frame (left) and photo analysis frame (right) will return a Row for every tweets, but with some null values for the tweets without photos.

	To achieve this, you need to:		Perform such an '''outerLeft''' join on the tweet ids. To achieve this, you need to:
	# pick out the ''timestamp''-s in addition to the ''value''-s from the incoming Kafka streams of tweets and photos,		# pick out the ''timestamp''-s in addition to the ''value''-s from the incoming Kafka streams of tweets and photos,
	# rename the ''id'' and ''timestamp'' columns in the streaming photo DataFrame,		# rename the ''id'' and ''timestamp'' columns in the streaming photo DataFrame,

Sinoa: /* Spark join */

2022-10-12T14:46:44Z

Spark join

← Older revision		Revision as of 14:46, 12 October 2022
Line 176:		Line 176:

	'''Task 10:''' Add a Spark pipeline to read tweet ''id'' and photo ''size'' from the ''photos'' topic. Perform an '''inner''' join on the ''id'' column to combine the results of sentiment analysis and external photo analysis.		'''Task 10:''' Add a Spark pipeline to read tweet ''id'' and photo ''size'' from the ''photos'' topic. Perform an '''inner''' join on the ''id'' column to combine the results of sentiment analysis and external photo analysis.

			'''Task 11:''' The inner join in task 10 only returns tweets with photos. A ''left outer join'' of the sentiment analysis frame (left) and photo analysis frame (right) will return all tweets, with some null values for tweets without photos. Perform such an '''outerLeft''' join on the tweet ids.

			To achieve this, you need to:
			# pick out the ''timestamp''-s in addition to the ''value''-s from the incoming Kafka streams of tweets and photos,
			# rename the ''id'' and ''timestamp'' columns in the streaming photo DataFrame,
			# define the ''timestamp'' and (renamed) ''photo_timestamp'' columns as watermarks, for example with a 10 minute treshold, and
			# add an explicit '''outerLeft''' join expression over the ''id''-s and ''timestamp''-s.

			Tips:
			* to pick out timestamps from incoming Kafka streams:
			.select(photo_sdf.timestamp,
			decode(photo_sdf.value, 'utf-8').alias('json_str')) \
			* to rename columns:
			.withColumnRenamed('timestamp', 'photo_timestamp') \
			* to add a watermark:
			.withWatermark('photo_timestamp', '10 minutes')

			For more inforation, see [https://spark.apache.org/docs/3.3.0/structured-streaming-programming-guide.html#handling-late-data-and-watermarking Handling Late Data and Watermarking] and [https://spark.apache.org/docs/3.3.0/structured-streaming-programming-guide.html#outer-joins-with-watermarking Outer Joins with Watermarking] in the [https://spark.apache.org/docs/3.3.0/structured-streaming-programming-guide.html Structured Streaming Programming Guide].

	== Some practical advice ==		== Some practical advice ==

Sinoa: /* Practical advice */

2022-10-12T14:30:33Z

Practical advice

← Older revision		Revision as of 14:30, 12 October 2022
Line 177:		Line 177:
	'''Task 10:''' Add a Spark pipeline to read tweet ''id'' and photo ''size'' from the ''photos'' topic. Perform an '''inner''' join on the ''id'' column to combine the results of sentiment analysis and external photo analysis.		'''Task 10:''' Add a Spark pipeline to read tweet ''id'' and photo ''size'' from the ''photos'' topic. Perform an '''inner''' join on the ''id'' column to combine the results of sentiment analysis and external photo analysis.

	== ~~Practical~~ advice ==		== Some practical advice ==

	* Do all of this incrementally, making sure that each step works completely before you start the next. Keep the whole pipeline running as you proceed.		* Do all of this incrementally, making sure that each step works completely before you start the next. Keep the whole pipeline running as you proceed.