[Repost] Top 5 Mistakes to Avoid When Writing Apache Spark Applications

Top 5 Mistakes to Avoid When Writing Apache Spark Applications from Cloudera, Inc.

Spark Summit East 2016 presentation by Mark Grover and Ted Malaska

Intro to Graph Databases Neo4j

Enterprise Integration Patterns : What, Why, How

How to : working with Graph database, Tinkerpop and Gremlin

Intro to Graph Databases Using Tinkerpop, TitanDB, and Gremlin from Caleb Jones https://www.slideshare.net/calebwjones/intro-to-graph-databases-using-tinkerpops-titandb-and-gremlin

SparkR with Zeppelin

SparkR + Zeppelin from felixcss

Algorithm : Types, Classification and Definition

1. Simple recursive algorithms
2. Backtracking algorithms
3. Divide and conquer algorithms
4. Dynamic programming algorithms
5. Greedy algorithms
6. Branch and bound algorithms
7. Brute force algorithms
8. Randomized algorithms

Complexity of Algorithms : 

1. Constant        - O(1)
2. Logarithmic  - O(log(N))
3. Linear           - O(N)
4. Quadratic      - O(N*N)
5. Cubic            - O(N*N*N)
6. ...
7. Exponential  - O(N!) or O(2^N) or O(N^K) or many others.

Road to Machine Learning

1. You start with a dataset to analyse. - Purchase / Social / Medical / Travel
2. Many variable are typically collected. - Categorical / Continuous /  Geo
3. Majority of them can be irrelevant and cause noise.
4. Data Mining is Statistics at Scale and Speed.
5. Applications in Intelligence / Genetics / Natural Sc. / Bussiness.
6. Data Mining has origin with Categorical data whereas Statistics deals with Continuous data.
7. Large model overfits the training dataset and may lead to higher prediction error with new situations.
8. Consider if predictor variable would be available and relationship holds in future data.
9. Cluster analysis is example for Unsupervised learning
10. Dimension Reduction
11. Association Rules
12. Classification is example of Supervised learning
13. Regression, Regression Trees, Nearest Neighbour - Continuous response.
14. Logistic Regression, Classification Trees, Nearest Neighbour, Discriminant analysis and Naive Bayes methods are well suited for Categorical response.  
15. Data Mining should be viewed as a process :
1. Data Storage & PreProcessing
2. Identify variables for investigation
3. Screen the outliers and missing values from data
4. Data need to be partitioned for training, test and evaluation set.
5. Use Sampling for Large datasets.
6. Visualize your data - Line, Bar, Scatter, Box, Histogram, Map, Geo
7. Summary of data - Mean, Median, Mode, Standard Deviation, Correlation, Principal Components
8. Apply appropriate model - Linear, Logistic, Trees, K-means ...
9. Verify finding against evaluation set.
10. Get the insights, Apply the findings! Plan - do - check - act !!

More on the next blog! Thank you!!
https://www.linkedin.com/in/alokawi ( Data Engineer, Analytics Engineer, Data Science )