Introduction

Spark integration

By default Detective has been configed to connect spark master "local[200]", which means you don't need config anything, a new spark "cluster" will start up in your local machine with 200 threads capacity.

Run with spark local server

bin/spark-submit --class detective.core.distribute.SparkDriver --master local[8] [YourJarFile] [YourPackageOrSotry] [YourApplicationName]

Build a true Spark cluster

To build your own spark cluster, please reference to Spark documentation

however, we require you do one change as we have to use a newer HttpClient component but Spark shipped a older one. To overcome this issue, for now please make a change in