Regression Explained for Dummies

 

What is regression? 

  Regression refers to a statistical method applied in various disciplines, such as finance and investing, to estimate and understand the extent of the relationship between a dependent variable and one or more independent variables, also referred to as predictable variables.  The term ‘regression’ derives from ‘regress’ which in turn derives from the Latin word ‘regressus,’ meaning to go back (to something).   The terms ‘regression’ and ‘regression analysis’ can be used interchangeably.   

🏆10 Best Forex Brokers in South Africa

Roots of the term ‘regression’ 

  According to historians, Francis Galton (1822 - 1911), an English polymath, anthropologist, and statistician, coined the term ‘regression’ when he was experimenting with sweet-pea seeds in 1875, according to the MacTutor History of Mathematics Archive.     MacTutor describes the experimenting process as follows: ‘He [Galton] used 100 seeds of each of seven different diameters and constructed a two-way plot of diameters of the original seeds against the diameters of the seeds of the next generation. He noticed that the median diameter of the offspring of the large seeds was less than that of their parents while the median diameter of the offspring of the small seeds was greater than that of their parents. Galton realised that the offspring tended to revert towards the mean size. Certainly, he did not understand at this stage that his findings would apply to any two-way plot, thinking rather that it was peculiar to the situation with which he was experimenting. At first, he called the phenomena ‘reversion,’ but later changed the name to regression.’ (Accentuations in the quotation are by the article writer.)    As a statistical method, regression should not be confused with the concept of ‘regression to the mean,’ which refers to the idea that extreme events are likely to be followed by more moderate ones.    

Types of regression 

  Regression is categorised into two main categories: linear regression and non-linear regression.    

Linear regression

Linear regression analysis is graphically depicted by using a straight line, referred to as the regression line or a line of best fit, where the slope indicates how the change in one variable impacts a change in the other variable.   The y-intercept of a linear regression relationship represents the value of one variable when the value of the other one is zero. Put differently, the y-intercept, also known as the constant, is the constant term used in regression analysis to indicate the value at which the regression line crosses the y-axis.  Regression analysis is helpful to determine which factors can be ignored, and which factors are the most important factors.   

• Four fundamental assumptions of linear regression

There are four fundamental assumptions of linear regression that must be met to qualify as a valid linear model. If one or more of these assumptions are disregarded, the results of the linear regression may be inaccurate and even deceptive.    1. Linear relationship, also known as linearity, implies there exists a linear relationship between the independent variable, X, and the dependent variable, Y.    When a variable is independent, it is not affected in any way by any other variables. The independent variable is also called the predictor variable which provides information on a related dependent variable with regard to a specific outcome.    The dependent variable is also referred to as the response variable or the variable that changes the results.    2. Put simply, homoscedasticity, also called homogeneity of variances, refers to the assumption that the variance of residuals is constant at every level of X, the independent variable.     In simple terms, a residual is ‘the error between a predicted value and the observed actual value,’ according to Towards Data Science.    Vexpower mentions that homoscedasticity essentially means ‘same variance,’ referring to it as an important concept in linear regression and defining the concept as follows: ‘Homoscedasticity describes how the error term (the noise or disturbance between independent and dependent variable) is the same across the values of the independent variables. So, in homoscedasticity, the residual term is constant across observations, i.e., the variance is constant. In simple terms, as the value of the dependent variable changes, the error term does not vary much.’    3. Independence of residuals, meaning observations of residuals (errors) are independent of each other.    4. Normality: The residual (error) values of the model are normally distributed.   

• Types of linear regression 

Linear regression (linear regression analysis) consists of two basic types of regression, namely linear regression, and multiple linear regression. These two types are considered traditional linear regression methods.    1. Linear regression    Linear regression, also called simple linear regression, is a model that analyses the relationship between a dependent variable and an independent variable.  The formula to express the simple linear model is as follows:  Y = a + bX + u  Where:  

• Y = the dependent variable that the analyser is trying to predict or explain. 

• X = the independent (explanatory) variable which the analyser is using to associate with Y. 

The terms ‘independent variable’ and ‘explanatory variable’ are often used interchangeably. However, there is a small difference between the two terms. When a variable is considered independent, it is not affected in any way by any other variables. But when a variable is not independent for sure, it is referred to as an explanatory variable. 

• a = the y-intercept, also known as the constant.  

• b = the slope of the independent (explanatory) variable.  

Dummies.com explains that ‘in a regression context, the slope is the heart and soul of the equation because it tells you how much you can expect Y [dependent variable] to change as X [independent variable] increases.’ (Accentuation and insertions in the explanation are by the article writer.) 

• u = the residual or error term.

  2. Multiple linear regression    Multiple linear regression analysis is the same as the simple linear regression model, except that multiple independent variables are used in the model.    However, the presence of several independent variables in the model requires another compulsory condition, namely non-collinearity. This means that the independent variables should show the minimum correlation with each other. If the independent variables are multicollinear (highly correlated to each other), it will be quite difficult to determine the true relationships between the dependent and independent variables.     The equation for multiple linear regression is:  Y = a + bX1 + cX2 + dX3 + u  Where: 

• Y = the dependent variable. 

• X1, X2, X3 = the independent (explanatory) variables. 

• a = the y-intercept, also known as the constant.  

• b, c, d = the slopes of the independent (explanatory) variable(s).  

• u = the residual or error term. 

 

Non-linear regression

Non-linear regression models also exist but are considerably more complicated. They are typically used for more complicated data sets where the independent and dependent variables display a non-linear relationship using a curve, making it non-linear in the parameter.    Put differently, the relationship between the predictor (independent) variable and response (dependent) variable represents a non-linear pattern.    While traditional linear regression is restricted to estimating linear models, non-linear regression ‘can estimate models with arbitrary relationships between independent and dependent variables. This is accomplished using iterative estimation algorithms,’ according to IBM.    Although significantly complicated, the non-linear regression model creates accurate results.    

Purposes of regression 

  Regression is used in statistical analysis to identify the relations between one or more variables that occur in some facts and figures.    Regression analysis can display both the significance of these relations and also determine the statistical importance of the relations, meaning whether or not the relations are likely due to chance.    Regression analysis has often been used to try to predict future outcomes based on past observations. In addition, it is a powerful tool for statistical inference, which is the process where a random sample is used to infer the properties of a whole population.      Also, regression analysis enables analysts, managers, and other people, to understand which are the most important factors, and how they are affecting each other. However, regression analysis cannot easily indicate causation, which is ‘the process of causing something to happen or exist,’ according to the Cambridge Dictionary.    

Applications of regression analysis 

  Regression analysis is applied in numerous ways by analysts in various industries. For example:    

• Financial industry and businesses

Managers and financial analysts use linear regression models to evaluate a business’s operational performance and to optimise its business processes. For instance, the management of a company may want to evaluate the efficiency of a call centre, determining what the relationship is between the wait times of clients and other callers and the number of complaints.    One of the most common uses of regression analysis in the business world is to predict events that have yet to happen. For instance, demand analysis is used to forecast how many units of a certain product consumers will purchase.    Regression is applied in the marketing of products: to understand the effectiveness of marketing campaigns, and to predict pricing and sales of products. This is done by establishing the relationships between measurable variables, such as the engagement of people via social media, product pricing and quantity of sales. The information obtained allows businesses to apply specific marketing strategies to improve and maximise sales and increase revenue. For instance, a business can apply a simple linear regression model to determine how price influences sales and use, evaluating the strength of the relationship between the two variables.    Short-term insurance companies use regression analysis to a considerable degree to evaluate risks. For example, estimating how many policy holders will be involved in vehicle accidents or be victims of hijacking or burglaries.    In terms of investments, financial analysts and investment managers use regression analysis for various reasons. For instance: 

1. To understand the trend in stock (share) prices and to predict future prices. 
2. To allow investment managers to value assets and understand the association between factors like commodity prices, and the shares of businesses dealing in those commodities. 
3. To predict returns on investments. 
4. It is used in the capital asset pricing model (CAPM), which examines the relationship between the expected rate of return and the associated market risks. 

 

• In manufacturing, regression analysis is used by businesses as a tool to identify root causes of defects, waste, and errors. By reducing defects, errors, and waste, the management of a business will be able to enhance performance. 

• In the medical industry, researchers can apply the regression model to determine the relationship between independent variables, such as body weight and age, and dependent ones, such as blood pressure and cholesterol levels. This can be helpful to identify the risk factors associated with diseases. The information obtained can also be used to identify high-risk patients, advancing healthy lifestyles. 

Regression is also used to predict the different combinations of medicines, enabling major companies to manufacture generic medicines to combat diseases.   

• Environmental Health 

Researchers in this industry use the regression analysis model to examine and evaluate the relationship between natural elements like soil, air, and water. For instance, what is the relationship between the quality of air and the quality of vegetables?  Also, what is the relationship between the amount and quality of water and plant growth?     This can enable environmentalists to forecast the influence of air pollution and water pollution on environmental health. 

Recommended brokers