In this tutorial, we consider an advanced technique for building a statistical model which can be used to explain phenomena, understand relationships, and predict outcomes. If you think about the inference portion of applied statistics so far, we’ve discussed how to conduct inference on a single numerical or categorical variable and then moved on to discuss inference on a numerical or categorical variable with an additional grouping variable. In the latter cases, the grouping variable needed to be a categorical variable (gender, automatic or manual transmission, type of diet eaten, etc.) – the questions we asked and answered were of the form, given observations which differ by the categorical variable \(X\), do they differ in another variable of interest \(Y\)?

Another way of phrasing this is that our inference applications seek to answer the question, “within our population of interest, is there an association between the variable \(X\) and the variable \(Y\)?”. If both variables \(X\) and \(Y\) are numerical, then we develop a new statistical inference technique called linear regression.

Simple linear regression uses a single numerical feature (predictor variable) to predict a numerical response. Simple linear regression uses the form of a straight line \(y = mx + b\), where \(m\) denotes slope of the relationship and \(b\) denotes the intercept (the value of \(y\) if \(x\) is 0). With regression, we are fitting a straight line to data, where noise is present – that is, the line we fit is not expected to pass through all of the data points. The form for a simple regression model is \[\displaystyle{y = \beta_0 + \beta_1x + \varepsilon}\]

Notice that \(\beta_0\) is the intercept, \(\beta_1\) is the slope, and \(\varepsilon\) denotes the unexplained error (noise). We typically do not write \(\varepsilon\) as part of the model, since we assume that it is random noise with a mean of \(0\) and a constant standard deviation (\(\sigma\)) – in our earlier notation, we assume \(\varepsilon \sim N\left(\mu = 0, \sigma\right)\). Instead, we often write the regression model as \[\displaystyle{\mathbb{E}\left[y\right] = \beta_0 + \beta_1x}\]

We can use regression models to predict an expected average response (\(y\)) for a given value of the feature \(x\). Regression models are only as good as the data they are trained on. Typically we can feel comfortable using a regression model to interpolate predictions (make predictions within the range of observed feature values) but not for extrapolation – making predictions for values of the predictor variable outside of the range of its observed values.

Let’s see regression in action as we consider an application to understanding biases in course evaluations.

The data were gathered from end of semester student evaluations for a large sample of professors from the University of Texas at Austin. In addition, six students rated the professors’ physical appearance. (This is a slightly modified version of the original data set that was released as part of the replication data for Data Analysis Using Regression and Multilevel/Hierarchical Models (Gelman and Hill, 2007).) The result is a data frame where each row contains a different course and columns represent variables about the courses and professors.

Before we draw conclusions about the trend, compare the number of observations in the data frame with the approximate number of points on the scatterplot. Is anything awry?

evals%>%
  ggplot(aes(x = bty_avg, y = score))+
  geom_jitter()

We add a line to your plot by adding a layer with geom_abline() using the appropriate slope and intercept arguments.

evals%>%
  ggplot(aes(x = bty_avg, y = score))+
  geom_point()+
  geom_abline(intercept = 3.88, slope = 0.067)