View source for Apache Hadoop and MapReduce

==Hadoop and MapReduce==
===Purpose===
* Getting up and running with Apache Hadoop and MapReduce
* Getting experience with non-trivial installation
* Writing and running your own first program

For a general introduction, see the slides to [[:File:S3-EM_datasources.pdf | Session 3]]. We will follow this tutorial closely: https://hadoop.apache.org/docs/r2.8.0/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html .

===Preparations===
You will run Hadoop standalone on your computers. Running Hadoop on a cluster of many computers is harder to set up (and you will need a cluster of computers), but after that, the coding and running of code is the same. So the jobs we will run on small datasets on a single computer will scale to large datasets on clusters of many powerful computers.

Follow these [[Hadoop preparations | preparations]] to install Hadoop on your '''Linux''' or '''Windows'''-machine. If you are on '''MacOS''', it runs BSD Unix under the hood, so most Linux-commands should work in a ''Terminal'' window on your Mac too.

===Tasks===
* [[Running Hadoop | Getting started with Hadoop]]
* [[Running Spark cluster in the cloud | Getting started with Spark cluster]]