Sunday, October 21, 2018

[Repost] Top 5 Mistakes to Avoid When Writing Apache Spark Applications


Spark Summit East 2016 presentation by Mark Grover and Ted Malaska

Saturday, January 2, 2016

Algorithm : Types, Classification and Definition


  1. Simple recursive algorithms
  2. Backtracking algorithms
  3. Divide and conquer algorithms
  4. Dynamic programming algorithms
  5. Greedy algorithms
  6. Branch and bound algorithms
  7. Brute force algorithms
  8. Randomized algorithms

Complexity of Algorithms : 
  1. Constant        - O(1)
  2. Logarithmic  - O(log(N))
  3. Linear           - O(N)
  4. Quadratic      - O(N*N)
  5. Cubic            - O(N*N*N)
  6. ...
  7. Exponential  - O(N!) or O(2^N) or O(N^K) or many others.

Tuesday, December 29, 2015

Road to Machine Learning

  1. You start with a dataset to analyse. - Purchase / Social / Medical / Travel
  2. Many variable are typically collected. - Categorical / Continuous /  Geo
  3. Majority of them can be irrelevant and cause noise.
  4. Data Mining is Statistics at Scale and Speed.
  5. Applications in Intelligence / Genetics / Natural Sc. / Bussiness.
  6. Data Mining has origin with Categorical data whereas Statistics deals with Continuous data.
  7. Large model overfits the training dataset and may lead to higher prediction error with new situations.
  8. Consider if predictor variable would be available and relationship holds in future data.
  9. Cluster analysis is example for Unsupervised learning
  10. Dimension Reduction
  11. Association Rules
  12. Classification is example of Supervised learning
  13. Regression, Regression Trees, Nearest Neighbour - Continuous response.
  14. Logistic Regression, Classification Trees, Nearest Neighbour, Discriminant analysis and Naive Bayes methods are well suited for Categorical response.  
  15. Data Mining should be viewed as a process :
    1. Data Storage & PreProcessing
    2. Identify variables for investigation
    3. Screen the outliers and missing values from data
    4. Data need to be partitioned for training, test and evaluation set.
    5. Use Sampling for Large datasets.
    6. Visualize your data - Line, Bar, Scatter, Box, Histogram, Map, Geo
    7. Summary of data - Mean, Median, Mode, Standard Deviation, Correlation, Principal Components
    8. Apply appropriate model - Linear, Logistic, Trees, K-means ...
    9. Verify finding against evaluation set.
    10. Get the insights, Apply the findings! Plan - do - check - act !!
More on the next blog! Thank you!!
https://www.linkedin.com/in/alokawi ( Data Engineer, Analytics Engineer, Data Science )