Tutorial 9: Regression Modelling II

Checking for the assumptions of a statistical model, like linear regression, is fundamental for several reasons. Ensuring these assumptions are met is critical for the accuracy, reliability, and validity of the conclusions drawn from the model. Here's why checking for assumptions matters:

• Validity of Model Inferences: For the statistical inferences of a model to be valid, such as hypothesis tests and confidence intervals for the model parameters, the underlying assumptions of the model must be satisfied. If assumptions are violated, the inferences made about the population from the sample data may not be accurate.

• Accuracy of Predictions: The predictive accuracy of a model relies on how well the model's assumptions are met. Violations of these assumptions can lead to biased or inaccurate predictions. Checking assumptions helps ensure the model correctly captures the relationship between the variables. 

• Efficiency of Estimators: Many statistical models, including linear regression, aim to provide the best possible estimates of the relationship between variables. When assumptions such as homoscedasticity and normality of residuals are met, the ordinary least squares (OLS) estimators are the most efficient linear unbiased estimators available (according to the Gauss-Markov theorem). Violating these assumptions can lead to less efficient estimators, meaning there could be other estimators that have lower variance.

• Identification of Model Limitations: Checking assumptions can help identify limitations or weaknesses in your model, pointing to areas where the model could be improved. For example, if residuals are not normally distributed or there is evidence of heteroscedasticity, this might indicate missing variables, the need for variable transformation, or the presence of outliers.

• Guidance on Model Selection: The process of checking assumptions can guide the selection of an appropriate model or the need for model adjustments. If certain assumptions are not met, alternative models that do not have these assumptions (e.g., generalized linear models for non-normal data) might be more appropriate.

Checking that the models' assumptions are met is a critical step in any statistical analysis. It ensures that the model is appropriate for the data and that the conclusions drawn from the analysis are valid and reliable.

-----------------------------------------------------------------------------------------------------------------------------

When assumptions are not met: 

If the assumptions are not met, consider the following options:

1. Add More Explanatory Variables to Your Model: Determine if there are additional variables that could help explain your dependent variable. If so, incorporate these into your model to see if the assumptions are now satisfied. However, be cautious not to indiscriminately add too many variables, as this can lead to other issues such as overfitting.

2. Non-linear Relationships: The relationship between variables may be non-linear rather than linear. In such cases, you can model this non-linear relationship. Doing so is relatively straightforward, but interpreting the results can be more complex. For guidance on modelling non-linear relationships, you might find [this link] helpful (Note: Link placeholder for actual URL).

3. Transforming Your Dependent and/or Independent Variable: If assumptions continue to be violated, consider transforming your dependent variable into an ordinal or nominal variable. This transformed variable can then be analyzed using logistic regression.

4. Quantile or Robust Regression: If the above strategies do not address the issue, quantile regression or robust regression methods may be appropriate. These techniques are designed to be more resilient to outliers and heteroscedasticity but are not covered in this tutorial.

5. Logistic Regression for Categorical Outcomes: Alternatively, if your dependent variable is or can be converted into an ordinal or nominal variable, logistic regression can be a suitable analytical approach.

Addressing Linearity Issues

When the linearity assumption is violated, you can use several strategies:

• Transformation of Variables: Applying transformations to the dependent and/or independent variables can sometimes linearize the relationship. Common transformations include logarithmic, square root, and power transformations. The choice of transformation depends on the nature of the non-linearity.

• Adding Polynomial Terms: If the relationship between variables is curvilinear, adding polynomial terms (e.g., the square or cube of an independent variable) to the model can capture the non-linear relationship.

• Piecewise Regression (Segmented Regression): If the data suggests different linear relationships in different ranges of the independent variable, piecewise regression can model these distinct phases separately.

• Non-linear Modeling: When linear models are insufficient to capture the complexity of the relationship, moving to non-linear models or generalized additive models (GAMs) may be more appropriate (These are not covered currently)

Important Considerations

Transformations and the addition of polynomial terms can make interpreting model coefficients more complex. It's essential to understand how these changes affect the relationship between variables. Adding polynomial terms or segmenting the data increases the model's complexity. It's crucial to balance the need for capturing non-linearity with the risk of overfitting, where the model fits the training data too closely and performs poorly on new data. Ultimately, the choice between transforming variables, adding polynomial terms, or switching to a non-linear model depends on the specific research question, the nature of the data, and the interpretability of the model.

While the assumption of linearity simplifies modelling and interpretation, real-world data often exhibit complex relationships that require careful assessment and potentially sophisticated modelling strategies to accurately capture.

The assumption of independence stipulates that the observations (data points) are independent of each other; that is, the value of one observation does not influence or predict the value of another. This assumption is foundational for the validity of statistical tests, as many inferential statistics rely on the independence of observations to produce accurate standard errors, confidence intervals, and significance tests.

How to Check for Independence

• Durbin-Watson Statistic: This test specifically checks for autocorrelation in residuals from a regression analysis. A Durbin-Watson statistic near 2 suggests no autocorrelation; values deviating substantially from 2 indicate positive or negative autocorrelation.

• Examination of Study Design: Often, the best way to assess independence is by examining the study design. For example, in experiments, randomization ensures independence. In observational studies, you'll need to consider whether the data collection method might introduce dependence.

Addressing Independence Issues

Violations of the independence assumption can often be addressed through modifications to the model or the data. Generalized Estimating Equations can be used for correlated data, often found in longitudinal studies, where observations from the same subject are correlated.

Use a Random Effects or Multi-Level Model. These models are particularly useful for handling data where observations are nested within higher-level groups (e.g., students within schools). By introducing random effects, these models can account for the lack of independence within groups while still assuming independence between groups. This is correct; using multi-level (hierarchical) regression is a way to address independence issues, especially in data with a nested or hierarchical structure

Alternatively, use fixed Fixed Effects Models in Panel data to control for unobserved heterogeneity when this heterogeneity is constant over time and specific to individuals or entities, thus addressing potential non-independence.

Ensuring the assumption of independence is crucial for the integrity of your statistical analysis. When this assumption is violated, it means the standard errors, p-values, and confidence intervals may not be reliable. Using appropriate models like multi-level regression or adjusting the analysis technique can help mitigate these issues and lead to more accurate and trustworthy conclusions.

Interpreting the Residual Plot:

The residuals are the differences between the observed values and the values predicted by the model. Analyzing these differences can provide insights into the adequacy of the model, the presence of outliers, and whether the assumptions of linear regression are being met (e.g., linearity, homoscedasticity, independence, and normality of residuals).

A fitted residual plot is a specific type of residual plot that plots the residuals on the y-axis against the fitted values (or predicted values) on the x-axis. This plot can help assess:

• Linearity: If the relationship between the variables is linear, the plot should show no specific pattern. A systematic pattern (like a curve) suggests a non-linear relationship that the linear model can't capture.

• Homoscedasticity: This assumption means that the variance of the error terms is constant across all levels of the independent variables. In the plot, this is indicated by the residuals spreading equally across all levels of the fitted values without forming a funnel shape (where the spread increases or decreases with the fitted values).

• Outliers: Points that are far away from the zero line could indicate outliers in the data that could potentially influence the regression model disproportionately.

• Leverage points: While not directly indicated by the fitted residual plot, by identifying unusual patterns or outliers, you may also indirectly identify data points with high leverage.

While the fitted residual plot is a powerful diagnostic tool, it's usually not enough on its own. It is best used in conjunction with other diagnostic plots and statistics, such as:

• QQ Plot (Quantile-Quantile Plot): Helps to assess if the residuals are approximately normally distributed.

• Leverage Plot: Helps to identify influential observations that have a significant impact on the model's parameters.

• Cook’s Distance: A measure that combines the information of leverage and residuals to identify influential data points.

Together, these diagnostics give a comprehensive view of the model's performance and assumptions. It's crucial to use multiple diagnostics because each highlights different aspects of the data and model fitting process.

-----------------------------------------------------------------------------------------------------------------------------

Addressing Heteroscedasticity

When homoscedasticity is violated, there are several approaches to address the issue:

• Transformations: Applying a transformation to the dependent variable (e.g., log, square root) can sometimes stabilize the variance across the range of independent variables.

• Weighted Least Squares (WLS): Instead of ordinary least squares (OLS), WLS can be used where each observation is weighted inversely to its variance, helping to mitigate the impact of heteroscedasticity (not covered here)

• Robust Standard Errors: Utilizing robust standard errors can adjust the estimates to account for heteroscedasticity, making the model's inference more reliable without transforming the data or changing the estimation technique. (not covered here).

The assumption of homoscedasticity is integral to many statistical analyses for ensuring the accuracy and validity of the model's inferences. Identifying and addressing heteroscedasticity is crucial for the reliability of the conclusions drawn from statistical models.

Interept Q/Q Plots

Q-Q plots are graphical tools used to assess if a dataset follows a certain theoretical distribution, usually the normal distribution. When interpreting Q-Q plots, the main thing to look for is how well the points follow the straight line drawn on the plot. If the points lie on or very close to the line, it suggests that the data conforms well to the theoretical distribution. Deviations from this line indicate departures from the expected distribution: 

• If the points form a curve that is higher than the line at the ends, it suggests "heavy tails" - meaning there are more extreme values than expected.

• If the curve is below the line at the ends, the data has light tails - fewer extreme values than expected.

• A systematic deviation to the left or right suggests skewness in the data.

By assessing how closely the data points follow this reference line, Q-Q plots provide a visual means to evaluate the distribution assumptions underlying many statistical tests and models, guiding further analysis decisions.

------------------------------------------------------------------------------------------------------------------------Addressing Non-normality of Residuals

When residuals do not appear to be normally distributed, several strategies can be employed:

• Transformations: Applying transformations to the dependent variable (e.g., logarithmic, square root, or Box-Cox transformation) can help achieve normality in the residuals.

• Adding Missing Variables: Non-normality can sometimes result from omitting relevant variables that capture the non-linear effects or interactions in the data.

• Alternative Models: For some types of data, particularly count data or binary outcomes, linear regression may not be appropriate. Generalized linear models (GLMs) offer alternative link functions and error distributions that might better suit the data structure.

While the assumption of the normality of residuals is less critical for large sample sizes due to the Central Limit Theorem, which suggests that the sampling distribution of the estimate becomes approximately normal with sufficient sample size, it remains a vital diagnostic tool. Assessing and addressing the normality of residuals ensures the integrity of statistical inferences drawn from the model, enhancing the credibility and reliability of the findings.

Addressing Multicollinearity

When multicollinearity is detected, several strategies can be considered to mitigate its impact:

• Remove Highly Correlated Predictors: If two variables are highly correlated, consider removing one of them, especially if it's not crucial for your model.

• Combine Variables: If the correlated variables represent similar concepts or measures, combining them into a single variable or using principal component analysis (PCA) to reduce dimensionality might be effective.

• Increase Sample Size: Sometimes, increasing the sample size can help reduce the impact of multicollinearity, although this is not always feasible.

• Ridge Regression: This technique introduces a small bias in the regression coefficients to significantly reduce their variance, making the model more interpretable despite multicollinearity.

The assumption of no multicollinearity is critical for ensuring the reliability and interpretability of the regression coefficients in a multiple regression model. By detecting and addressing multicollinearity, researchers can make more confident inferences about the relationships between independent variables and the dependent variable, enhancing the model's overall validity and usefulness.

-----------------------------------------------------------------------------------------------------------------------------

General Guidelines for Cook's Distance

Low Cook's Distance: A low value suggests that the observation has little to no influence on the regression model's coefficients. Typically, values of Cook's Distance below 0.5 are considered low, indicating that removing the observation would not significantly alter the model.

Moderate Cook's Distance: Values between 0.5 and 1 can be considered moderate. Observations falling in this range may warrant closer examination, as they could start to influence the model's predictions.

High Cook's Distance: Observations with Cook's Distance values greater than 1 are often considered highly influential. These points have a substantial impact on the model and could potentially skew the results. A common, more conservative threshold used to flag potentially influential points is 4/n, where n is the total number of observations in the dataset. This threshold adjusts for the size of the dataset, making it particularly useful for datasets of varying sizes.

Interpreting Cook's Distance

Interpreting Cook's Distance involves both quantitative assessment using these guidelines and qualitative judgment about the data and the context of the study. It's essential to:

• Examine observations with high Cook's Distance to determine if they are outliers or leverage points that unduly influence the model.

• Consider the substantive context: Is there a reason to expect certain data points to be influential? Are they errors, or do they represent important but rare events?

• Decide on appropriate action: Depending on the analysis and the impact of the influential points, you might choose to investigate further, remove them, or model them separately.

In summary, while values of Cook's Distance greater than 1 are commonly flagged as potentially influential, the decision on what constitutes low or high should be informed by the specific context of your analysis, the distribution of Cook's Distance values across your data, and the size of your dataset.