Explain the reasoning behind your code so that it is clear what each block is intended to achieve, and why.
This assignment is designed on the basis of five learning outcome criteria`s, which you can find in this article. These 5 learning outcomes are very essential in order to clear this subject. You have study each and every requirements of these five outcomes and provide a brief result according to your research.
Aims
The aim of this assignment is to introduce a practical application of Big Data and Cloud Computing using a realistic big data problem. Students will implement a solution using an industry leading Cloud computing provider together with the distributed processing environment Apache Spark. This will involve the selection of problem appropriate Machine Learning algorithms and methods.
Learning Outcome 1: Apply big data analytic algorithms, including those for visualization and cloud computing techniques to multi-terabyte datasets.
Learning Outcome 2: Critically assess data analytic and machine learning algorithms to identify those that satisfy given big data problem requirements
Learning Outcome 3: Critically evaluate and select appropriate big data analytic algorithms to solve a given problem, considering the processing time available and other aspects of the problem.
Learning Outcome 4: Design and develop advanced big data applications that integrate with third party cloud computing services Personal Values Attributes (Global / Cultural awareness, Ethics, Curiosity) (PVA):
Learning Outcome 5: Critically assess the relationship between knowledge and the ethical and social interpretation of primary research using big data.
Definitions
Portfolio Assignment: A collection of pieces of work
Individual Work: Work carried out by one person only
Group Work: Work carried out collaboratively seeking to improve each other`s elements
Peer Review: Critical analysis and subsequent grading of a social equal`s work
Semi-Formative: Training tasks assigned course credit to reward and ensure engagement.
Big Data Product: Weapons and Drugs
In the television documentary "Ross Kemp and the Armed Police" broadcast 6th September 2018 by ITV, multiple claims were made regarding violent crime in the UK.
These claims were:
1. Violent Crime is increasing
2. There are more firearms incidents per head in Birmingham than anywhere else in the UK.
3. Crimes involving firearms are closely associated with drugs offences
In this assignment you will investigate these claims using real, publicly available data sets that will be made available to you and placed in Amazon S3. These include, but are not limited to:
1. Street Level Crime Data published by the UK Home Office. This dataset contains 19 million data rows giving a crime type, together with their location as a latitude and longitude.
2. Land Registry Price Paid Data: This gives the postcode of a property, the property type from an enumeration of D (Detached), S (Semi-Detached), T (Terraced), F (Flats/Maisonettes) and the price paid.
3. Postcode Data: This data set is based on material provided by the Ordinance Survey. It gives a latitude and longitude to every postcode. This is useful as it relates between the Land Registry Price Paid dataset postcode, and the original crime dataset
latitude/longitude.