Getting started with Apache Spark - Revision history

Sinoa at 17:41, 5 September 2022

2022-09-05T17:41:54Z

← Older revision		Revision as of 17:41, 5 September 2022
Line 117:		Line 117:

	=== Running Spark in Jupyter and VS Code ===		=== Running Spark in Jupyter and VS Code ===
	You can also run PySpark in a Jupyter notebook, either standalone or inside an Integrated Development Environment (IDE).		You can also run PySpark in a Jupyter notebook, either standalone or inside an Integrated Development Environment (IDE). On Windows, you may also need to install the [https://code.visualstudio.com/docs/remote/wsl Visual Studio Code Remote - WSL] plugin.

	The example below runs PySpark in Jupyter inside [https://code.visualstudio.com/ VS Code (Visual Studio Code)]. See Microsoft's own guide to [https://code.visualstudio.com/docs/introvideos/basics Getting started with Visual Studio Code]. See this guide for [https://code.visualstudio.com/docs/datascience/jupyter-notebooks running Jupyter notebooks in VS Code].		The example below runs PySpark in Jupyter inside [https://code.visualstudio.com/ VS Code (Visual Studio Code)]. See Microsoft's own guide to [https://code.visualstudio.com/docs/introvideos/basics Getting started with Visual Studio Code]. See this guide for [https://code.visualstudio.com/docs/datascience/jupyter-notebooks running Jupyter notebooks in VS Code].
Line 123:		Line 123:
	The example code below should work on other Jupyters too, such as [https://jupyter.org/ JupyterLab].		The example code below should work on other Jupyters too, such as [https://jupyter.org/ JupyterLab].

	Install and start [https://code.visualstudio.com/ VS Code]. Select '''File -> Open folder...''' and open 'info319-exercise' from before. Create a new file with extension ''.ipynb'', for example ''exercise1.ipynb''. Doubleclick the new file, which should open it in a Jupyter notebook inside VS Code.		Install and start [https://code.visualstudio.com/ VS Code]. Install the

			Select '''File -> Open folder...''' and open 'info319-exercise' from before. Create a new file with extension ''.ipynb'', for example ''exercise1.ipynb''. Doubleclick the new file, which should open it in a Jupyter notebook inside VS Code.

	Activate the new ''exercise1.ipynb'' tab and then click the 'Python' button near the upper right corner of the Python notebook. A window should pop up and ask you to ''Change kernel for 'exercise1.ipynb'''. Type in or select ''./venv/bin/python''. Now you are ready to use Jupter inside VS Code.		Activate the new ''exercise1.ipynb'' tab and then click the 'Python' button near the upper right corner of the Python notebook. A window should pop up and ask you to ''Change kernel for 'exercise1.ipynb'''. Type in or select ''./venv/bin/python''. Now you are ready to use Jupter inside VS Code.

Sinoa at 14:11, 1 September 2022

2022-09-01T14:11:01Z

@@ Line 100: / Line 100: @@
   <class 'pyspark.sql.types.Row'>
 [https://spark.apache.org/docs/latest/sql-programming-guide.html DataFrame Rows] are another central Spark data structure. Can you get the rows out as Python dicts?
 === Set up git (optional) ===

Sinoa at 14:10, 1 September 2022

2022-09-01T14:10:18Z

← Older revision		Revision as of 14:10, 1 September 2022
Line 102:		Line 102:

	=== Exploring tweets in Spark ===		=== Exploring tweets in Spark ===
	~~In Session 1 we will look at more things to do with Spark DataFrames. Here are some possible things to do in Exercise 1 (this is not final)~~:		Continuing the examples:
	* ~~Load~~ the tweets as JSON objects.		* load the tweets in ‘tweet-id-text-345/’ as JSON objects
	* ~~Collect~~ only the texts from the tweets.		* collect only the texts from the tweets
	* ~~Split~~ the texts into words and select all the hashtags.		* split the texts into words and select all the hashtags
	* ~~Build~~ a ~~graph of retweets~~		* the step where you go from a column of lists-of-words to a columns of words is a little harder
	* ~~Split~~ the tweets into two sets of 80% and 20% size.		* you can write this step in Python (using collect() and then creating a new DataFrame)
	* ~~Find~~ URLs in the texts and download a few image files.		* you can also go via a Pandas frame, but like the Python solution, this breaks the parallel processing
	* Work on a folder with more tweets.		* you can also use the file all-tweet-words.txt in mitt.uib.no to skip this step
	* Open the Spark context Web UI (see PySpark's start-up banner)		* two other solutions, which we will present later, are:
	* Experiment with different numbers of partitions and executors.		* going via an RDD and using a flatMap()
	~~Of course, we will do these things ''in Spark'', without going via plain Python.~~		* writing a user-defined Spark function (UDF)
			* split the tweets into two sets of 80% and 20% size
			* find URLs in the texts and download a few image files

	=== Set up git (optional) ===		=== Set up git (optional) ===

Sinoa at 14:32, 24 August 2022

2022-08-24T14:32:00Z

← Older revision		Revision as of 14:32, 24 August 2022
Line 76:		Line 76:

	=== Processing tweets ===		=== Processing tweets ===
	Download [https://~~drive~~.~~google~~.~~com~~/~~file~~/d/~~175MWQp_D_eFqP3DcRhY9AwrHOvHMP0vT~~/~~view~~?~~usp~~=~~sharing~~ this archive of random tweets], and unpack in your exercise folder:		Download [https://mitt.uib.no/courses/37204/files/4406276/download?download_frd=1 this archive of random tweets] (available from Files -> Datafiles in https://mitt.uib.no), and unpack in your exercise folder:
	(venv) $ tar xjf tweet-id-text-345.tar.bz2		(venv) $ tar xjf tweet-id-text-345.tar.bz2
	(venv) $ ls tweet-id-text-345		(venv) $ ls tweet-id-text-345

Sinoa at 14:23, 24 August 2022

2022-08-24T14:23:14Z

← Older revision		Revision as of 14:23, 24 August 2022
Line 112:		Line 112:
	* Open the Spark context Web UI (see PySpark's start-up banner)		* Open the Spark context Web UI (see PySpark's start-up banner)
	* Experiment with different numbers of partitions and executors.		* Experiment with different numbers of partitions and executors.
	Of course, we will do these things ~~_in Spark_~~, without going via plain Python.		Of course, we will do these things ''in Spark'', without going via plain Python.

	=== Set up git (optional) ===		=== Set up git (optional) ===
Line 127:		Line 127:
	$ git branch -M main		$ git branch -M main
	$ git push -uf origin main		$ git push -uf origin main
	(The push will not contain many files since we haven't written any Spark ~~_program_~~ yet.)		(The push will not contain many files since we haven't written any Spark ''program'' yet.)

	=== Running Spark in Jupyter and VS Code ===		=== Running Spark in Jupyter and VS Code ===

Sinoa at 14:22, 24 August 2022

2022-08-24T14:22:04Z

← Older revision		Revision as of 14:22, 24 August 2022
Line 19:		Line 19:

	In addition, Spark will need a Java Runtime Environment - a JRE or JDK - somewhere on your PATH. For example:		In addition, Spark will need a Java Runtime Environment - a JRE or JDK - somewhere on your PATH. For example:
	$ sudo apt install openjdk-8-jdk-headless/		$ sudo apt install openjdk-17-jdk-headless/

	And in your '~/.bashrc' file add the line (check the location first, it could depend on your system):		And in your '~/.bashrc' file add the line (check the location first, it could depend on your system):
	export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64		export JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64
			(We may use packages later in the course that have problems with the newest Java versions, but let us start with 17.)
	(~~This is an old Java version, but we~~ may use packages later in the course that have problems with ~~never~~ Java-s.)

	Create a new folder for running Spark:		Create a new folder for running Spark:

Sinoa at 14:20, 24 August 2022

2022-08-24T14:20:00Z

← Older revision		Revision as of 14:20, 24 August 2022
Line 3:		Line 3:
	* Getting up and running with Apache Spark		* Getting up and running with Apache Spark
	* Using VS Code (or your IDE of choice) with Spark		* Using VS Code (or your IDE of choice) with Spark
	* ~~Writing and running~~ your own first Spark instructions		* Running your own first Spark instructions
	~~For a general introduction, see the slides to Session 1 on Apache Spark. There is a useful tutorial at [https://www.tutorialspoint.com/spark_sql/spark_introduction.htm TutorialsPoint].~~

	==Preparations==		==Preparations==
	In the first exercise, you will run Spark standalone on your own computer, first in a ''console'' (or ''terminal window'') and then in your favourite IDE (Integrated Development Environment).		In the first exercise, you will run Spark standalone on your own computer, first in a ''console'' (or ''terminal window'') and then in your favourite IDE (Integrated Development Environment).
			* [https://spark.apache.org/docs/latest/quick-start.html Spark Quick Start]
			* [https://spark.apache.org/docs/latest/sql-programming-guide.html Spark SQL, DataFrames, and Datasets Guide]
			* [https://spark.apache.org/docs/latest/api/python/getting_started/quickstart_df.html PySpark QuickStart: DataFrame]

	=== Run PySpark from console ===		=== Run PySpark from console ===
Line 88:		Line 90:
	>>> type(tweets)		>>> type(tweets)
	<class 'pyspark.sql.dataframe.DataFrame'>		<class 'pyspark.sql.dataframe.DataFrame'>
	DataFrame is a very central data structure in Spark.		[https://spark.apache.org/docs/latest/sql-programming-guide.html DataFrame] is a very central data structure in Spark. Here are [https://spark.apache.org/docs/latest/api/python/getting_started/quickstart_df.html more PySpark examples of DataFrames].

	>>> tweets.count()		>>> tweets.count()
Line 98:		Line 100:
	>>> type(tweet_list[13])		>>> type(tweet_list[13])
	<class 'pyspark.sql.types.Row'>		<class 'pyspark.sql.types.Row'>
	DataFrame Rows are another central Spark data structure. Can you get the rows out as Python dicts?		[https://spark.apache.org/docs/latest/sql-programming-guide.html DataFrame Rows] are another central Spark data structure. Can you get the rows out as Python dicts?

	=== Exploring tweets in Spark ===		=== Exploring tweets in Spark ===
Line 111:		Line 113:
	* Open the Spark context Web UI (see PySpark's start-up banner)		* Open the Spark context Web UI (see PySpark's start-up banner)
	* Experiment with different numbers of partitions and executors.		* Experiment with different numbers of partitions and executors.

	Of course, we will do these things _in Spark_, without going via plain Python.		Of course, we will do these things _in Spark_, without going via plain Python.

Line 159:		Line 160:
	tweets = spark.read.text(folder)		tweets = spark.read.text(folder)
	tweets.count()		tweets.count()

			Spark is lazily evaluated, so if you enter transformations, you will normally not get any output before you combine them with actions (such as ''count()'', ''collect()'', or ''show()''). But you can instruct Spark to evaluate eagerly instead:
			spark.conf.set('spark.sql.repl.eagerEval.enabled', True)
			spark.conf.set('spark.sql.repl.eagerEval.maxNumRows', 5)

	An important advantage over running PySpark directly in a console/terminal window is that autocompletion and other help functions now work. You can also easily save and load notebooks, and benefit from other useful IDE functions.		An important advantage over running PySpark directly in a console/terminal window is that autocompletion and other help functions now work. You can also easily save and load notebooks, and benefit from other useful IDE functions.

Sinoa at 10:53, 24 August 2022

2022-08-24T10:53:11Z

← Older revision		Revision as of 10:53, 24 August 2022
Line 22:		Line 22:
	export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64		export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64

	(This is an old Java version, but we may use packages later in the course that have problems with never ~~Javas~~.)		(This is an old Java version, but we may use packages later in the course that have problems with never Java-s.)

	Create a new folder for running Spark:		Create a new folder for running Spark:

Sinoa at 10:50, 24 August 2022

2022-08-24T10:50:35Z

← Older revision		Revision as of 10:50, 24 August 2022
Line 64:		Line 64:
	/__ / .__/\_,_/_/ /_/\_\ version 3.3.0		/__ / .__/\_,_/_/ /_/\_\ version 3.3.0
	/_/		/_/

	Using Python version 3.8.10 (default, Jun 22 2022 20:18:18)		Using Python version 3.8.10 (default, Jun 22 2022 20:18:18)
	Spark context Web UI available at http://172.23.240.233:4040		Spark context Web UI available at http://172.23.240.233:4040

Sinoa at 10:49, 24 August 2022

2022-08-24T10:49:45Z

Show changes