We have been dealing with different machine learning models. The aim of model building is to predict the value of target variable(dependent) based on feature values. If we have only one feature (independent/explanatory) variable we use Simple Linear Regression model. If not (that is more than one feature) we use Multiple Linear Regression model. Feature variables can be grouped together and call X random variables which are independent of each other. Before go into the subject matter of Feature engineering let us understand the difference between inference and prediction. Under statistics we have already seen descriptive and inferential statistics.
Inference
Under Inference, we focus on understanding and explaining the relationships between the target and feature variables. We use Correlation Coefficient measure for this. The relationship could be either positive or negative or zero(no relationship). If the relationship is positive then it means the target values increases when independent variable increases in the positive direction(upward). If the relationship is negative, it is called downward relationship where target value either increases or decreases when independent value decreases or increases. (That is in the opposite direction). Generally we ask “why” and “how” the relationship between target and independent variable behave.
Prediction
Under prediction, we focus our attention to accurate forecasting outcomes for new or test datasets/data points. It is concerned with “what will happen” when one unit of independent variable changes (increase or decrease). We use complex models to achieve high degree of accuracy even if the models we propose are harder to interpret.
While constructing models we have to consider the following two aspects, Accuracy and interpretability, irrespective of the fact whether the model is simple or complex. Simple model is easy to interpret and it is less prone to overfitting. However, complex models are prone to overfitting. We can not compromise Accuracy for the sake of simplicity. The key consideration of a model building lies on fine Trade-off between accuracy and interpretabililty. Additional parameters play critical role in getting the accuracy and interpretability aspects in the right mix.
Feature Engineering:
Feature engineering is nothing but a process of creating representations of data that increases the effectiveness of the model
- Some models can not tolerate predictors(features) that measure the same underlying quantity(multi collinearity or correlation between independent features). The beauty is that two or more independent variables may be highly correlated with each other. it may miss lead to wrong prediction of y value.
- Missing values may lead wrong prediction. So data preprocessing is required to remove missing values or substitute missing values
- We have been dealing with different machine learning models. The aim of model building is to predict the value of target variable(dependent) based on feature values. If we have only one feature (independent/explanatory) variable we use Simple Linear Regression model. If not (that is more than one feature) we use Multiple Linear Regression model. Feature variables can be grouped together and call X random variables which are independent of each other. Before go into the subject matter of Feature engineering let us understand the difference between inference and prediction. Under statistics we have already seen descriptive and inferential statistics.
Inference
Under Inference, we focus on understanding and explaining the relationships between the target and feature variables. We use Correlation Coefficient measure for this. The relationship could be either positive or negative or zero(no relationship). If the relationship is positive then it means the target values increases when independent variable increases in the positive direction(upward). If the relationship is negative, it is called downward relationship where target value either increases or decreases when independent value decreases or increases. (That is in the opposite direction). Generally we ask “why” and “how” the relationship between target and independent variable behave.
Prediction
Under prediction, we focus our attention to accurate forecasting outcomes for new or test datasets/data points. It is concerned with “what will happen” when one unit of independent variable changes (increase or decrease). We use complex models to achieve high degree of accuracy even if the models we propose are harder to interpret.
While constructing models we have to consider the following two aspects, Accuracy and interpretability, irrespective of the fact whether the model is simple or complex. Simple model is easy to interpret and it is less prone to overfitting. However, complex models are prone to overfitting. We can not compromise Accuracy for the sake of simplicity. The key consideration of a model building lies on fine Trade-off between accuracy and interpretabililty. Additional parameters play critical role in getting the accuracy and interpretability aspects in the right mix.
Feature Engineering:
Feature engineering is nothing but a process of creating representations of data that increases the effectiveness of the model
- Some models can not tolerate predictors(features) that measure the same underlying quantity(multi collinearity or correlation between independent features). The beauty is that two or more independent variables may be highly correlated with each other. it may miss lead to wrong prediction of y value.
- Missing values may lead wrong prediction. So data preprocessing is required to remove missing values or subsitute missing values
- Irrelevant predictors(features) available in the data may confuse and provide wrong prediction.
Feature Engineering and Variable (feature) Selection will solve this type of problems. Dimensionality Reduction or reduction of feature enables you to maintain the simplicity and interpretability
Why Feature Selection plays vital role in model building?
- Curse of Dimensionality: When we deal with huge number of features we suffer from different issues which may complicate out data analysis and machine learning tasks. The sparsity of data points may prevent you to find meaningful patterns and relationships. Dimensionality reduction provides simplicity.
- Computational Complexity: When dimensionality increases we may require high computational storage, process and analysis. Algorithm may be lengthy and unexplainable at one stage and it may be expensive also
- Interpretability: Complexity of the model prevent you to have the easy interpretability characteristics. High-dimensional models are to prone to capture noise. Overfitted models are complex in nature. You may not be able to judge which features are influential.
- Distance Matrix: This one works well with low dimensional data. In high-dimensional data it may lose its interpretive power. You may not be able to explain why certain data points are considered and others are not considered.
Feature Selection Methods:
- Filter based
- Wrapper methods
- Embedded Techniques
- Hybrid Techniques
predictors(features) available in the data may confuse and provide wrong prediction.
Feature Engineering and Variable (feature) Selection will solve this type of problems. Dimensionality Reduction or reduction of feature enables you to maintain the simplicity and interpretability.
Why Feature Selection plays vital role in model building?
- Curse of Dimensionality: When we deal with huge number of features we suffer from different issues which may complicate out data analysis and machine learning tasks. The sparsity of data points may prevent you to find meaningful patterns and relationships. Dimensionality reduction provides simplicity.
- Computational Complexity: When dimensionality increases we may require high computational storage, process and analysis. Algorithm may be lengthy and unexplainable at one stage and it may be expensive also
- Interpretability: Complexity of the model prevent you to have the easy interpretability characteristics. High-dimensional models are to prone to capture noise. Overfitted models are complex in nature. You may not be able to judge which features are influential.
- Distance Matrix: This one works well with low dimensional data. In high-dimensional data it may lose its interpretive power. You may not be able to explain why certain data points are considered and others are not considered.
Feature Selection Methods:
- Filter based
- Wrapper methods
- Embedded Techniques
- Hybrid Techniques