Processing tweets with Spark - Revision history

Sinoa: /* Processing tweets with Spark */

2022-09-03T09:04:01Z

Processing tweets with Spark

← Older revision		Revision as of 09:04, 3 September 2022
Line 5:		Line 5:
	* collect only the texts from the tweets		* collect only the texts from the tweets
	* split the texts into words and select all the hashtags		* split the texts into words and select all the hashtags
	* the step where you go from a column of lists-of-words to a columns of words is ~~a little harder~~		* the step where you go from a column of lists-of-words to a columns of words is harded in plain Spark
	** ''import pyspark.sql.functions import ~~export~~'' is the ~~simplest~~ way ~~to do this~~		** using the ''explode()''-function (''import pyspark.sql.functions import explode'') seems to be the easiest way
	* split the tweets into two sets of 80% and 20% size		* split the tweets into two sets of 80% and 20% size
	* find URLs in the texts and download a few image files		* find URLs in the texts and download a few image files

Sinoa: /* Processing tweets with Spark */

2022-09-03T08:49:09Z

Processing tweets with Spark

← Older revision		Revision as of 08:49, 3 September 2022
Line 6:		Line 6:
	* split the texts into words and select all the hashtags		* split the texts into words and select all the hashtags
	* the step where you go from a column of lists-of-words to a columns of words is a little harder		* the step where you go from a column of lists-of-words to a columns of words is a little harder
	** ~~you can write this step in Python (using collect() and then creating a new DataFrame)~~		** ''import pyspark.sql.functions import export'' is the simplest way to do this
	** you can also go via a Pandas frame, but like the Python solution, this breaks the parallel processing
	** you can also use the file all-tweet-words.~~txt in mitt~~.~~uib.no~~ to ~~skip~~ this ~~step~~
	** two other solutions, which we will present later, are:
	** going via an RDD and using a flatMap()
	** writing a user-defined Spark function (UDF)
	* split the tweets into two sets of 80% and 20% size		* split the tweets into two sets of 80% and 20% size
	* find URLs in the texts and download a few image files		* find URLs in the texts and download a few image files

Sinoa at 14:14, 1 September 2022

2022-09-01T14:14:14Z

← Older revision		Revision as of 14:14, 1 September 2022
Line 6:		Line 6:
	* split the texts into words and select all the hashtags		* split the texts into words and select all the hashtags
	* the step where you go from a column of lists-of-words to a columns of words is a little harder		* the step where you go from a column of lists-of-words to a columns of words is a little harder
	* you can write this step in Python (using collect() and then creating a new DataFrame)		** you can write this step in Python (using collect() and then creating a new DataFrame)
	* you can also go via a Pandas frame, but like the Python solution, this breaks the parallel processing		** you can also go via a Pandas frame, but like the Python solution, this breaks the parallel processing
	* you can also use the file all-tweet-words.txt in mitt.uib.no to skip this step		** you can also use the file all-tweet-words.txt in mitt.uib.no to skip this step
	* two other solutions, which we will present later, are:		** two other solutions, which we will present later, are:
	* going via an RDD and using a flatMap()		** going via an RDD and using a flatMap()
	* writing a user-defined Spark function (UDF)		** writing a user-defined Spark function (UDF)
	* split the tweets into two sets of 80% and 20% size		* split the tweets into two sets of 80% and 20% size
	* find URLs in the texts and download a few image files		* find URLs in the texts and download a few image files

Sinoa: /* Processing tweets with Spark = */

2022-09-01T14:13:18Z

Processing tweets with Spark =

← Older revision		Revision as of 14:13, 1 September 2022
Line 1:		Line 1:
	== Processing tweets with Spark ===		== Processing tweets with Spark ==

	Continuing the examples from [Getting started with Apache Spark]:		Continuing the examples from [[Getting started with Apache Spark]]:
	* load the tweets in ‘tweet-id-text-345/’ as JSON objects		* load the tweets in ‘tweet-id-text-345/’ as JSON objects
	* collect only the texts from the tweets		* collect only the texts from the tweets

Sinoa: Created page with "== Processing tweets with Spark === Continuing the examples from [Getting started with Apache Spark]: * load the tweets in ‘tweet-id-text-345/’ as JSON objects * collect..."

2022-09-01T14:12:59Z

Created page with "== Processing tweets with Spark === Continuing the examples from [Getting started with Apache Spark]: * load the tweets in ‘tweet-id-text-345/’ as JSON objects * collect..."

New page

== Processing tweets with Spark ===

Continuing the examples from [Getting started with Apache Spark]:
* load the tweets in ‘tweet-id-text-345/’ as JSON objects
* collect only the texts from the tweets
* split the texts into words and select all the hashtags
* the step where you go from a column of lists-of-words to a columns of words is a little harder
* you can write this step in Python (using collect() and then creating a new DataFrame)
* you can also go via a Pandas frame, but like the Python solution, this breaks the parallel processing
* you can also use the file all-tweet-words.txt in mitt.uib.no to skip this step
* two other solutions, which we will present later, are:
* going via an RDD and using a flatMap()
* writing a user-defined Spark function (UDF)
* split the tweets into two sets of 80% and 20% size
* find URLs in the texts and download a few image files