If you’re diving into data analysis, you’ve likely heard about Ordinary Least Squares (OLS) regression. This fundamental statistical tool allows us to understand relationships between variables. In this post, we’ll break down key concepts and guide you through interpreting regression results, from single-variable to multi-variable models.
1. Understanding the Basics: Dependent and Independent Variables
Dependent Variable (Y): The outcome variable we’re trying to explain or predict. For example, in a study on education, the dependent variable might be ‘Student Test Scores’. This means that we want to explain the determinants of student test scores.
Independent Variable (X): The variable we believe impacts the dependent variable. For instance, ‘Hours Studied’ could be an independent variable influencing ‘Student Test Scores’.
Think of the dependent variable as the ‘outcome’ and the independent variables as factors that influence this outcome.
2. A Simple Model with One Independent Variable
Imagine a model where we examine how ‘Hours Studied’ (independent variable) affects ‘Student Test Scores’ (dependent variable). Our simple linear regression model can be expressed as:
Student Test Scores = β0 + β1 (Hours Studied) + ε
Where:
β0 is the intercept, showing the expected score when ‘Hours Studied’ is zero.
β1 is the coefficient for ‘Hours Studied,’ representing how each additional hour studied affects test scores.
ε is the error term, accounting for other unobserved factors.
2.1. Interpreting the Results of the Simple Model
Below is the output of the linear regression analysis run in the statistical software STATA for the simple model presented above. The number of observations is 50. Let’s proceed step by step to understand what to look for when interpreting this output.
Check R-squared
Look at the value of R-squared to get a sense of the model fit. In our model, the R-squared is 0.7662, indicating that 76.62% of the variation in test scores is explained by the number of hours studied. This is a strong fit.
Check Statistical Significance
Look at the p-value (or t-statistic) to see if the coefficient is statistically significant. Typically, a p-value less than 0.05 or a t-statistic greater than 2 suggests significance. Here, the p-value of 0.000 for ‘Hours Studied’ indicates that it significantly predicts ‘Student Test Scores’.
In published articles, you’ll often see stars (*) next to significant coefficients, which is a common shorthand to quickly communicate the strength of statistical significance.
Examine the Coefficient’s Direction and Size
The coefficient for ‘Hours Studied’ (4.947) is positive, indicating that as hours studied increases, test scores also increases.
Interpretation
Intercept (48.03): When ‘Hours Studied’ is zero, the predicted average test score is 48.03.
Coefficient for Hours Studied (4.947): For each additional unit (hour studied in our model), the test score is expected to increase by around 4.5 points.
2.2. The Omitted Variable Problem
While our simple model gives insight, it may miss key factors affecting test scores, such as class attendance or part-time job hours. Omitting such variables can bias our estimates, leading to incorrect conclusions about the effect of hours studied.
3. Expanding to Multiple Independent Variables
To get a clearer picture, let’s add ‘Class Attendance’, ‘Part-Time Job Hours’ and 'Library Visits' to our model:
Student Test Scores = β0 + β1 (Hours Studied) + β2 (Class Attendance) + β3 (Part-Time Job Hours) + β4 (Library Visits) + ϵ
3.1. Interpreting the Results of the Multi-Variable Model
Below is the output of the linear regression analysis run for the multi-variable model presented above.
Check R-squared
The R-squared has increased from 0.7662 to 0.9264 when we added new variables to the model. This substantial increase indicates that the new variables are significantly helping to explain the variation in the dependent variable (i.e. 'Student Test Scores').
Check Statistical Significance
The p-values of each predictor are less than 0.05, indicating that all variables significantly affect 'Student Test Scores' —except for Library Visits, which has a high p-value (0.529). This suggests 'Library Visits' doesn’t contribute much to the model.
Examine the Coefficient’s Direction
Hours Studied (3.54): A positive coefficient, meaning more study hours lead to higher test scores.
Class Attendance (2.24): Also positive, showing that attending class increases test scores.
Part-Time Job Hours (-1.17): A negative coefficient, indicating that working more hours negatively impacts test scores. Each additional hour of work reduces the score by around 1.2 points.
Interpretation
Adding ‘Class Attendance’ and ‘Part-Time Job Hours’ helps refine our model. For instance, the coefficient for ‘Hours Studied’ dropped from 4.97 to 3.54, offering a more accurate estimate of its impact after accounting for other variables. In other words, in the simple model, 'Hours Studied' may have been capturing the effects of these correlated variables, leading to an inflated estimate of its own impact.
In multi-variable models, each coefficient reflects the variable’s impact on test scores, holding all other factors constant (ceteris paribus). This phrase means "all else being equal," emphasizing that we’re isolating each variable’s effect on the outcome.
3.2. What Happens When We Add New Variables?
Adding new independent variables can have various effects:
Refine Existing Coefficients
Including ‘Class Attendance’ and ‘Part-Time Job Hours’ adjusted the effect of ‘Hours Studied,’ giving a more accurate picture of its true impact. The simple model might have "confounded" the effect of 'Hours Studied' with those of the other variables, which is why the coefficient changes when those variables are included.
Improve Model Fit
Additional relevant variables can increase the model’s explanatory power, often seen in an increase in the R-squared value, which shows how well the model explains the variation in the dependent variable.
BUT !!!
Risk of a Spurious Relationship
Sometimes, adding variables can introduce a spurious relationship—a misleading correlation due to an unobserved variable. For instance, if we added ‘Coffee Consumption’ to predict ‘Test Scores,’ we might observe an association. However, the relationship could simply reflect that students who study more often drink more coffee, not that coffee directly affects scores.
Risk of Over-fitting
Adding too many variables, especially if they are not meaningful, can lead to over-fitting—where the model becomes too complex and starts to capture noise rather than true relationships. This can make the model less generalizable to new data.
3.3. When to Add or Remove Variables? Caution !!!
When deciding which variables to include, theory is essential. A strong theoretical basis ensures that each variable logically contributes to the model, preventing the introduction of irrelevant or misleading predictors.
In Sum
When building your regression model, make sure that it is guided by a theory.
When interpreting the results, remember to check statistical significance, analyze the direction of each coefficient, and be cautious of potential spurious relationships.
Content Usage
This content is intended for educational purposes. If you wish to use or cite it, please provide proper attribution to Dissertation Roadmap.
Comments