Some of the key features of Apache Spark are the following:

• Supports multiple Programming Languages – Spark code can be written in any of the four programming languages like Python, Java, Scala, and R and also supports high-level APIs in them.
• Machine Learning – Apache Spark’s MLib is the machine learning component that is very useful for Big Data processing. It eradicates the need to use distinct engines for machine learning and processing. For data scientists and data engineers, Apache Spark supports a powerful and unified engine that is both fast and very easy to manage.
• Lazy Evaluation – Apache Spark supports lazy evaluation, which is too useful for delaying the evaluation time until the point it becomes absolutely compulsory.
• Real-Time Computation – Apache Spark computation is real-time and has less latency due to its in-memory computation. It is designed especially for massive scalability requirements.
• Supports Multiple Formats – Apache Spark offers support for multiple data sources like Hive, Cassandra, Parquet, and JSON. To access structured data though Spark SQL, data sources API provides a pluggable mechanism, and they can be much more than simple pipes for converting and pulling data into Spark.
• Hadoop Integration – For Hadoop, Apache Spark provides smooth compatibility. It can run on top of a Hadoop cluster using YARN for resource scheduling.
• Speed – Apache Spark is 100 times faster for extensive scale data processing compared to Hadoop and MapReduce. It achieves tremendous speed through controlled portioning, which helps in parallelizing distributed data processing with minimal network traffic.

To support Python with Spark, the Spark community has released a tool called PySpark. It is primarily used to process structured and semi-structured datasets and also supports an optimized API to read data from the multiple data sources containing different file formats. Using PySpark, you can also work with RDDs in the Python programming language using its library name Py4j.

The main characteristics of PySpark are listed below:

• Nodes are Abstracted.
• Based on MapReduce.
• API for Spark.
• The network is abstracted.

RDD stands for Resilient Distribution Datasets, a fault-tolerant set of operational elements that is capable of running in parallel. In general, RDDs are portions of data, which are stored in the memory and distributed over many nodes.

All partitioned data in an RDD is distributed and immutable.

Primarily two types of RDDs are available:

1. Hadoop datasets: Those who perform a function on each file record in Hadoop Distributed File System (HDFS) or any other storage system.
2. Parallelized collections: Those existing RDDs which run in parallel with one another.

The major advantages of using Apache Spark are the following:

• It’s simple to write parallelized code.
• Manages synchronization points as well as errors.
• Many vital algorithms are already implemented in Spark.

Some of the limitations of using Apache Spark are listed below:

• Sometimes, it’s challenging to manage a problem in MapReduce.
• Compared to other programming models, it’s not efficient.

SparkContext is referred to as an entry point for Spark functionality. For running any Spark application, automatically a program driver is created, which includes the main function and SparkContext will get initiated there. Then, the driver program runs operations inside executors on worker nodes.

SparkContext uses Py4J(library) for launching a JVM in PySpark. By default, PySpark contains SparkContext as ‘sc’.

. SparkConf helps in setting a few configurations and parameters to run a Spark application on the local/cluster. In simple words, it provides configurations to run a Spark application.

PySpark offers the credibility to upload our files in Apache Spark. This is done, using a sc.addFile, where sc is default SparkContext. We get the path of the directory using SparkFiles.net.

For resolving the path to the files added through SparkContext.addFile(), we use the below-mentioned methods in SparkFiles:

• get(filename)
• getrootdirectory()

In general, Apache Spark is a graph execution engine that enables users to analyze massive data sets with high performance. For this, Spark first needs to be held in memory to improve performance drastically, if data needs to be manipulated with multiple stages of processing.

Resilient Distributed Datasets are a set of multiple data items which are huge in size such that they are not suitable for a single node and have to be partitioned across several nodes. For this, Spark automatically partitions RDD and distributes the partitions across different nodes. Partition in Spark referred to as an atomic chunk of data stored on a node in the cluster. RDDs in Apache Spark are sets of partitions.

get (filename) helps to achieve the correct path of a file that is added through SparkContext.addFile(). Whereas, getrootdirectory() helps to get the root directory which consists of the files that are added through SparkContext.addFile().

PySpark Storage Level controls storage of an RDD. It also manages how to store RDD in the memory or over the disk, or sometimes both. Moreover, it even controls the replicate or serializes RDD partitions.

We use a broadcast variable, for the purpose of saving data copy across all nodes. It is represented with SparkContext.broadcast().

We use Accumulator variables in order to aggregate the information through associative and commutative operations.

The program that runs on the master node of a machine and declares actions and transformations on data RDDs is called Spark Driver. In simple words, a driver.in Spark develops SparkContext, connected to a given Spark Master.

Spark Driver also delivers RDD graphs to Master, when the standalone Cluster Manager runs.

The frequently used Spark ecosystems are:

• Spark SQL (Shark) for developers
• GraphX for generating and computing graphs
• Spark Streaming for processing live data streams
• SparkR to promote R Programming in the Spark engine
• MLlib (Machine Learning Algorithms)

Stream processing is supported by Spark, which is an extension to the Spark API that lets stream processing of live data streams. Data from multiple sources like Flume, Kafka, Kinesis, etc., is processed and then pushed to live dashboards, file systems, and databases. Compared to the terms of input data, it is just similar to batch processing, and data is segregated into streams like batches in processing.

For improving performance, PySpark supports custom serializers to transfer data. They are:

• MarshalSerializer – It supports only fewer data types, but compared to PickleSerializer, it is faster.
• PickleSerializer – It is by default used for serializing objects. Supports any Python object but in slow speed.

PySpark supports custom profiles that are used for creating predictive models. Profilers are in general, calculated using min and max values of each column.

As a useful data review tool, it is used for ensuring the data is valid and fit for further consumption.

For a custom profiler, you should define or inherit the following methods:

• profile – Similar to system profile.
• add – Helps to add a profile to the existing accumulated profile
• dump – Dumps the profiles to a path.
• stats – Returns the collected stats.

Some of the Algorithms supported in ApacheSpark are:

• mllib.classification
• mllib.fpm
• Mllib.linalg
• mllib.clustering
• spark.mllib
• mllib.recommendation
• Mllib.regression

Following are the parameters of a SparkContext:

• Master – It’s the URL of the cluster from which it connects.
• pyFiles – It is the .zip or .py files, in order to send to the cluster and also to add to the PYTHONPATH.
• Environment – Worker nodes environment variables.
• sparkHome – Spark installation directory.
• Conf – to set all the Spark properties, an object of L{SparkConf}.
• appName – It denotes the name of our job.
• Serializer – RDD serializer.
• JSC – It is the JavaSparkContext instance.

The significant attributes of SparkConf are listed below:

• set(key, value) – This attribute helps in setting the configuration property.
• setSparkHome(value) – This attribute enables in setting Spark installation path on worker nodes.
• setAppName(value) – This attribute helps in setting the application name.
• setMaster(value) – This attribute helps in setting the master URL.
• get(key, defaultValue=None) – This attribute supports in getting a configuration value of a key.

In ApacheSpark, the columnar format file supported by various other data processing systems is called a Parquet file. Spark SQL executes both operations which include read and write using Parquet file and determines it to be one of the great Big Data Analytics formats on whole.

The advantages of having columnar storage are as follows:

• Columnar storage helps to limit IO operations.
• It fetches particular columns that you need to access.
• It supports better-summarized data and follows type-specific encoding.
• It consumes less space.

The functions which are applied on RDD for resulting in another RDD are called transformations. They don’t execute till action occurs.

Examples: map() and filter()

Actions help to bring back the data from RDD to the local machine. The execution of the action is the output of all previously created transformations.

Actions trigger execution using a lineage graph for loading the data into original RDD, carrying out all intermediate transformations and returning final results to the Driver program or write it out to the file system.

Examples:

• take() action – It takes all the values from RDD to a local node.
• reduce() action – It executes the function passed again and again until one value is left.

The module used is Spark SQL, which integrates relational processing with Spark’s functional programming API. It helps to query data either through Hive Query Language or SQL.

The below mentioned are the four libraries of Spark SQL.

• Data Source API
• Interpreter & Optimizer
• DataFrame API
• SQL Service

SparkCore implements several key functions such as memory management, fault-tolerance, monitoring jobs, job scheduling and interaction with storage systems. Moreover, additional libraries, built atop the core let diverse workloads for streaming, machine learning, and SQL.

This is useful for:

• Memory management
• fault recovery
• Interacting with storage systems
• Scheduling and monitoring jobs on a cluster

Yes, Apache Spark can be run on the hardware clusters that are administered by ApacheMesos.

By setting the parameter ‘spark.cleaner.ttl’ we can trigger the automatic clean-ups. Also, by segregating the long-running jobs into various batches and writing the intermediary results to the disk.

Spark uses Akka for scheduling. When the workers request a task to master after registering, then the master just assigns a task. For this, Spark uses Akka to message between the workers and masters.

MLlib is a scalable Machine Learning library offered by Spark. It supports making Machine Learning secure and scalable with standard learning algorithms and use cases such as regression filtering, clustering, dimensional reduction, and the like.