In this section, we use least squares regression as a rigorous approach for fitting data using a linear model.

• Draw the scatter plot, then observe if there is a linear pattern.

• The least square regression line: \(y=b_0 + b_1x\), where the coefficients \(b_0\), \(b_1\) are determined by minimizing \[D = \sum{(\text{Observed response - Predicted response})^2}\\ =\sum{(y_i - b_0 - b_1x)^2} = \sum{e_i}^2\]

• There are formulas to compute \(b_0\) and \(b_0\).

• We want a line that has small residuals

  1. Option 1: Minimize the sum of magnitudes (absolute values) of residuals. \(|e_1| + |e_2|+ ... + |e_n|\)
  2. Option 2: Minimize the sum of squared residuals – least squares. \(e_1^2 + e_2^2 + ... + e_n^2\)

• Why least squares?

  1. Most commonly used.

  2. Easier to compute by hand and using software.

1. Linearity

2. Nearly normal residuals

3. Constant variability

4. Independent observations

• The relationship between the explanatory and the response variable should be linear.

• Check using a scatter plot of the data, or a residuals plot.

• The residuals should be nearly normal (bell shaped, symmetric).

• Check using a histogram or normal probability plot of residuals

Given \(\bar{x}= 7.5\), \(\bar{y} = 14.265\), \(s_x= 0.909\), \(s_y= 1.849\), \(R= 0.8976\).

1. Calculate the slope of the regression line \(\color{blue}{b_1 = \frac{s_y}{s_x}R}\)
2. Calculate the intercept of the regression line \(\color{blue}{b_0 = \bar{y} - b_1 \bar{x}}\)
3. Write the equation of the regression line.
4. Use this model to predict the value of y if x= 100
5. Find the coefficient of determination (\(R^2\))

Check Answers:

1. \(b_1 = \frac{s_y}{s_x}R = 1.8258\)
   
2. \(b_0 = \bar{y} - b_1 \bar{x}= 0.5714\)
   
3. \(y= 0.5714 + 1.8258x\)
   
4. 18.8294
   
5. 0.8057
   Discussion: if know \(s_x\), \(s_y\) and the slope, how to find R?
   Answer: \(R = b_1\frac{s_x}{s_y}\)

A researcher measures the wrist circumference and height of a random sample of individuals. The data summary is displayed below.

Residual standard error: 4.0929 on 28 degrees of freedom. R-squared: 0.6683, Adjusted R-squared: 0.6564.

1. The equation of the best-fit line is(Use x(Wrist circumference) and y (height) for variables:

\(y= 27.1107 + 6.4843x\)

1. Identify the coefficient of determination

\(R^2 = 0.6683\) (variability of height that can be explained using the model)

1. Calculate the correlation coefficient

\(R= \sqrt{0.6683} = 0.8174961774 \hspace{0.2cm} (\text{pay attention to sign of} \hspace{0.2cm}b_1)\)

1. Typical error using the model to predict heights

\(4.0929\)

1. Find the residual for one observation (7.3,84)

\(e= y-\hat{y} = 84- (27.1107 + 6.4843 \times 7.3)= 9.55391\)

Interpretation of \(\hat{y}= 64.68- 0.62x\)

1. Slope \(b_1=−0.62\): For each additional % point in HS graduate rate, we would expect the % living in poverty to be lower on average by 0.62% points.
2. Intercept \(b_0=64.68\): States with no HS graduates are expected on average to have 64.68% of residents living below the poverty line.

• Note: Since there are no states in the data set with no HS graduates, the intercept is of no interest, not very useful, and not reliable. In fact, the predicted value of the intercept is so far from the bulk of the data.

• Using the linear model to predict the value of the response variable for a given value of the explanatory variable is called prediction, simply by plugging in the value of x in the linear model equation.

• There will be some uncertainty associated with the predicted value.

• Applying a model estimate to values outside of the realm of the original data is called extrapolation (Extrapolation is used to estimate values that go beyond a set of given data or observations).

• In our example, the intercept is an extrapolation.

• The correlation

  \(R = \frac{1}{n-1}\sum_{i=1}^n \frac{x_i-\bar{x}}{s_x}\frac{y_i-\bar{y}}{s_y}\hspace{0.2cm} {\text{OR}}\hspace{0.2cm} R= \frac{\sum(x_i-\bar{x})(y_i-\bar{y})}{\sqrt{\sum(x_i-\bar{x})^2(y_i-\bar{y})^2}}\)

• The quantity \(R^2\) is called the coefficient of determination (see.in 8.1)

• It is a statistical measure in a linear regression model that determines the proportion of variance in the response variable that can be explained by the explanatory variable(or input,or predictor variable).

• The larger \(R^2\) (Close to 1), the better fit;

• The smaller \(R^2\) (close to 0), the more noise; the model does not give a good fit: due to either little association or nonlinear association.

We can allow to use categorical variable to predict outcomes.

If we divide the US into East and West, and use the model \[ \hat{poverty} = 11.17 + 0.38 \times west \]

• Explanatory variable: region, reference level: east

• Intercept: The estimated average poverty percentage in eastern states is 11.17%.

  • This is the value we get if we plug in 0 for the explanatory variable.

• Slope: The estimated average poverty percentage in western states is 0.38% higher than eastern states.

  • Then, the estimated average poverty percentage in western states is 11.17 + 0.38 = 11.55%.
  • This is the value we get if we plug in 1 for the explanatory variable