Friday, March 18, 2016

SparkR with Zeppelin

Saturday, January 2, 2016

Algorithm : Types, Classification and Definition

  1. Simple recursive algorithms
  2. Backtracking algorithms
  3. Divide and conquer algorithms
  4. Dynamic programming algorithms
  5. Greedy algorithms
  6. Branch and bound algorithms
  7. Brute force algorithms
  8. Randomized algorithms

Complexity of Algorithms : 
  1. Constant        - O(1)
  2. Logarithmic  - O(log(N))
  3. Linear           - O(N)
  4. Quadratic      - O(N*N)
  5. Cubic            - O(N*N*N)
  6. ...
  7. Exponential  - O(N!) or O(2^N) or O(N^K) or many others.

Tuesday, December 29, 2015

Road to Machine Learning

  1. You start with a dataset to analyse. - Purchase / Social / Medical / Travel
  2. Many variable are typically collected. - Categorical / Continuous /  Geo
  3. Majority of them can be irrelevant and cause noise.
  4. Data Mining is Statistics at Scale and Speed.
  5. Applications in Intelligence / Genetics / Natural Sc. / Bussiness.
  6. Data Mining has origin with Categorical data whereas Statistics deals with Continuous data.
  7. Large model overfits the training dataset and may lead to higher prediction error with new situations.
  8. Consider if predictor variable would be available and relationship holds in future data.
  9. Cluster analysis is example for Unsupervised learning
  10. Dimension Reduction
  11. Association Rules
  12. Classification is example of Supervised learning
  13. Regression, Regression Trees, Nearest Neighbour - Continuous response.
  14. Logistic Regression, Classification Trees, Nearest Neighbour, Discriminant analysis and Naive Bayes methods are well suited for Categorical response.  
  15. Data Mining should be viewed as a process :
    1. Data Storage & PreProcessing
    2. Identify variables for investigation
    3. Screen the outliers and missing values from data
    4. Data need to be partitioned for training, test and evaluation set.
    5. Use Sampling for Large datasets.
    6. Visualize your data - Line, Bar, Scatter, Box, Histogram, Map, Geo
    7. Summary of data - Mean, Median, Mode, Standard Deviation, Correlation, Principal Components
    8. Apply appropriate model - Linear, Logistic, Trees, K-means ...
    9. Verify finding against evaluation set.
    10. Get the insights, Apply the findings! Plan - do - check - act !!
More on the next blog! Thank you!! ( Data Engineer, Analytics Engineer, Data Science )

Friday, October 16, 2015

Correlation, Regression and Causation ?

What is a Correlation in statistics?

Correlation is a statistical technique that can show whether and how strongly pairs of variables are related. For example, height and weight are related; taller people tend to be heavier than shorter people. The relationship isn't perfect.

What is a Regression in statistics?

In statistics, linear regression is an approach for modeling the relationship between a scalar dependent variable y and one or more explanatory variables (or independent variables) denoted X. The case of one explanatory variable is called simple linear regression.

What is a Causation in statistics?
When an article says that causation was found, this means that the researchers found that changes in one variable they measured directly caused changes in the other.