• This chapter focuses on summarizing data

  • Numerical data

  • Categorical data

• There are three sections in this chapter

  • 2.1 Examining numerical data

  • 2.2 Considering categorical data

  • 2.3 Case study

In this section, we explore techniques for summarizing numerical variables.

• Graphical Summary

  • Scatterplot (for two numerical variables)
  • Dot plot
  • Histogram and shape (modality and skewness)
  • Boxplots & Outliers

• Numerical Summary

  • Mean
  • Variance and Standard Deviation
  • Median
  • Quartiles and IQR

A scatterplot (also called scatter graph, scatter chart, scattergram, or scatter diagram) is a type of plot using Cartesian coordinates to display values for typically two numerical variables for a set of data.

Scatterplots are useful for visualizing the relationship between two numerical variables.

Example 1. The figure below is a scatterplot for a data set (50 cases) of the total income and borrowed loan amount.

Practice: Read two pair of data and interpret.

• When two variables show some connection with one another, they are called associated variables.
  • Linear(positive, negative) association.
  • Nonlinear association.
• If two variables are not associated, i.e. there is no evident connection between the two, then they are said to independent.

Back to Example 1. Is the association linear or nonlinear? Nonlinear

Visualize the types of associations.

• Dot Plot : shows a dot for each observation placed above its value on a number line.

• Dot Plot is useful for visualizing one numerical variable.

• Darker colors represent areas where there are more observations.

Higher bars represent areas where there are more observations, makes it a little easier to judge the center and the shape of the distribution.

• The mean, also called the average (marked with a triangle in the above plot), is one way to measure the center of a distribution of data.

• The mean GPA is 3.59.

• The sample mean, denoted as \(\bar{x}\), can be calculated as

where \(x_1+x_2+ \dots + x_n\) represent the n observed values.

• The population mean is also computed the same way but is denotes as \(\mu\). It is often not possible to calculate \(\mu\) since population data are rarely available.

• This estimate may not be perfect, but if the sample is good (representative of the population), it is usually a pretty good estimate.

• Histogram provide a view of the data density. Higher bars represent where the data are relatively more common.

• Histograms are especially convenient for describing the shape of the data distribution.

• The chosen bin width can alter the story the histogram is telling.

Histograms are said to be skewed to the side of the long tail.

In order to determine modality, step back and imagine a smooth curve over the histogram. (Continue to see next slide).

Unimodal and right skewed, with a potentially unusual observation at 60 hours/week.

\[ s^2 = \frac{\sum_{i=1}^{n}(x_i-\bar{x})^2}{n-1}\\ x : 1,2,5,20 \\ \bar{x} =7\\ (x-\bar{x}): -6,-5,-2,13 \\ (x-\bar{x})^2 : 36,25,4,169 \\ s^2 = \frac{234}{3}= 78 \] We use the squared deviation in the calculation of variance:

• To get rid of negatives so that observations equally distant from the mean are weighted equally.

• To weigh larger deviations more heavily.

\(s^2 = \frac{(5-6.71)^2+(9-6.71)^2+\dots+(7-6.71)^2}{217-1} = 4.11 \text{ } hours^2\)

The standard deviation is the square root of the variance, and has the same units as the data.

\[s = \sqrt{s^2}\] \[ s = \sqrt{\frac{\sum_{i=1}^n(x_i - \bar{x})^2}{n-1}}\] Standard deviation measures the variability of data:

1. if \(\color{blue}{s}\) is small, the data is concentrated around the mean \(\color{blue}{\bar{x}}\);

2. if \(\color{blue}{s}\) is large, the data is spread further from the mean \(\color{blue}{\bar{x}}\).

\[ s = \sqrt{\frac{\sum_{i=1}^n(x_i - \bar{x})^2}{n-1}}\]

• The standard derivation 𝑠 represents a type of average distance of observations from the mean.

• The standard deviation is zero (𝑠 = 0) only when all observations have the same value, otherwise 𝑠 > 0.

• The larger the value of standard deviation 𝑠, the greater the variability of the data. As the spread of the data increases,𝑠 gets larger.

• s has the same units of measurement as the original observations, while variance 𝑠^2 does not.

• The standard deviation𝑠is not resistant. That is, strong skewness or a few outliers can greatly increase𝑠.

• The median is the value that splits the data in half when ordered in ascending order.

\[0, 1, \textbf{2}, 3, 4\]

• If there are an even number of observations, then the median is the average of the two values in the middle.

\[0, 1, \underline{2, 3}, 4, 5 \rightarrow \frac{2+3}{2} = \textbf{2.5}\]

• Since the \(\color{red}{median}\) is the midpoint of the data, 50% of the values are below it. Hence, it is also the \(\color{red}{50^{\textbf{th}}}\) percentile.

Example. the data below gives the per capita CO2 emissions in 9 largest nations measured in metric tons per person. Find the value of the median.

China 5.9; India 1.4; U.S. 16.9; Indonesia 1.8; Brazil 2.1; Pakistan 0.8; Nigeria 0.3; Bangladesh 0.4; Russia 11.6.

Solution.

First, put the 9 observations in ascending order 0.3, 0.4, 0.8, 1.4, 1.8, 2.1, 5.9, 11.6, 16.9

Since 𝑛 = 9 is odd, the median is the middle observation:

median= 1.8 metric tons.

• In R, we calculate the median by applying the median() command to the raw (unordered) values:

median(c(5.9, 1.4, 16.9, 1.8, 2.1, 0.8, 0.3, 0.4, 11.6))

• If we drop out the U.S. value, what is the new median? New mean? 0.3, 0.4, 0.8, 1.4, 1.8, 2.1, 5.9, 11.6 Now the new median is \(\frac{(1.4+1.8)}2=1.6\)

The new mean is 3.04.

• If we change the last value, 0.3, 0.4, 0.8, 1.4, 1.8, 2.1, 5.9, 11.6, \(\color{red}{106.9}\)
  • what is the new median? New mean?
  • Now the new median is still 1.8, the new mean is \(\color{red}{13.04}\)

Steps:

• Arrange the data in order.

• Find the median. This is the second quartile, Q2.

• Consider the lower half of the observations (excluding the median itself if n is odd). The median of these observations is the first quartile, Q1.

• Consider the upper half of the observations (excluding the median itself if n is odd). Their median is the third quartile, Q3.

• The median of the 20 values is the average of the 10th and 11th observations, 180 and 180, which is Q2 = 180 mg.

• The first quartile Q1 is the median of the 10 smallest observations (in the top row), which is the average of 130 and 140, Q1 = 135 mg.

• The third quartile Q3 is the median of the 10 largest observations (in the bottom row), which is the average of 200 and 210, Q3 = 205 mg.

• Five-Number Summary:
  minimum; first quartile; median; third quartile; maximum

• In this example, the five number summary is: 0 135 180 205 340

The interquartile range is the distance between the third quartile and first quartile: \[IQR = Q3-Q1\] IQR gives the spread of middle 50% of the data distribution.

Boxplot uses quartiles to draw a box and two whiskers to summarize data. (Boxplot can be horizonal or vertical.)

1. The box represents the middle 50% of the data, from Q1 to Q3. The thick line in the box is the median.

2. A line goes from the lower end of the box to the smallest observation that is not a potential outlier (lower whisker).

3. A line goes from the upper end of the box to the largest observation that is not a potential outlier (upper whisker).

4. Indicate potential (suspected) outliers if there are any.

Below is a boxplot of interest rate from from loan dataset. From the plot,

1. Find (estimate) quartiles, IQR and five number summary.
2. Is the data skewed?.

When there is no outliers (or in a simple way), the boxplot is determined by five number summary.

Multiple boxplots may put by side-by-side to compare the data sets.

Example. The following are two box plots for heights (in inches) of boys and girls. Use the boxplots to answer the following questions:

Match the histograms to Boxplot.

• Identify extreme skew in the distribution.

• Identify data collection and entry errors.

• Provide insight into interesting features of the data.

Median and IQR are more robust to skewness and outliers than mean and SD. Therefore,

Median -b/c household income is skewed to the right.

• If the distribution is symmetric, center is often defined as the mean: mean \(\approx\) median.

• If the distribution is skewed or has extreme outliers, center is often defined as the median.

  • Right-skewed: mean > median
  • left-skewed: mean < median

1. Mean > Median
2. Mean < Median
3. Mean \(\approx\) Median
4. Impossible to tell

1. Mean > Median
2. Mean < Median
3. Mean \(\approx\) Median
4. Impossible to tell

• Median: 80%
• Mean: 76%

When data are extremely skewed, transforming them might make modeling easier. A common transformation is the log transformation.

The histograms on the left shows the distribution of number of basketball games attended by students. The histogram on the right shows the distribution of log of number of games attended.(Here is the natural log)

• Skewed data are easier to model with when they are transformed because outliers tend to become far less prominent after an appropriate transformation. \[ \begin{split} \# of games & 70 & 50 & 25 & \dots \\ log(\# of games) & 4.25 & 3.91 & 3.22 & \dots \\ \end{split} \]
• However, results of an analysis in log units of the measured variable might be difficult to interpret.

Salary, housing prices, etc.

In this section, we introduce tables and other tools for categorical data.

• Frequency table, relative frequency table.
• Bar Plots
• Contingency Tables, Row and Column Proportions
• Mosaic plots

Example. A study of aphasia measured a sample of 22 adult aphasics with identified three types of aphasia (loss of ability to understand or express speech). The resulting data is given below:

• The variable “Type of Aphasia” has 3 categories (also called classes).
• We can construct the following frequency table to summarize the data.

We can use the R function table() to construct the frequency table to summarize the data.

• First, put the data in a vector using the c() function. We will name that vector aphasia_type
  • You may choose any other name but note that names of objects in R cannot contain any spaces;
  • Notice that we used underscore to create a two-word name
  • Also, notice that the values are typed within double quotes because we are working with non-numerical or character data
• Then, use the table() function to get the frequencies of the different aphasia types

aphasia_type = c("Broca's", "Anomic", "Anomic", "Conduction", "Broca's", "Conduction", "Conduction", "Anomic", 
                 "Conduction", "Anomic", "Conduction", "Broca's", "Anomic", "Broca's", "Anomic", "Anomic", "Anomic",
                 "Conduction", "Broca's", "Anomic", "Conduction", "Anomic")
table(aphasia_type)

A bar plot is a common way to display a single categorical variable.

• The categories of a categorical variable are represented by bars.

• The height of each bar is either the frequency, or the relative frequency (or the percentage), of the represented category.

Here is an example of a bar plot (frequency). (check the annotation and the scale of the vertical axis)

A table that summarizes data for two categorical variable is called a contingency table.

The contingency table below shows the distribution of survival and ages of passengers on the Titanic.

\[ \begin{align} && && {Survival} & \\ & && Died & & Survived & & Total \\ {Age} & Adult && 1438 && 654 && 2092 \\ & Child && 52 && 57 && 109 \\ & Total && 1490 && 711 && 2201 \\ \end{align} \]

A bar plot is a common way to display a single categorical variable. A bar plots below are frequency bar plots* (left) and relative frequency bar plot (right) for the group of Adult on Titanic.

\[ \begin{align} && && {Survival} & \\ & && Died & & Survived & & Total \\ {Age} & Adult && 1438 && 654 && 2092 \\ & Child && 52 && 57 && 109 \\ & Total && 1490 && 711 && 2201 \\ \end{align} \] check: \(\frac{1438}{2092}\approx 0.69\hspace{0.2cm} \text{or} \hspace{0.2cm} 69%\)

The x-axis in a histogram is a number line, hence the order of the bars cannot be changed. In a bar plot, the categories can be listed in any order(though some ordering make more sense than others, especially for ordinal variables)

\[ \begin{align} && && {Survival} & \\ & && Died & & Survived & & Total \\ {Age} & Adult && 1438 && 654 && 2092 \\ & Child && 52 && 57 && 109 \\ & Total && 1490 && 711 && 2201 \\ \end{align} \]

To answer this question we examine the row proportions:

• % Adults who survived: 654 / 2091 \(\approx\) 0.31

• % Children who survived: 57 / 109 \(\approx\) 0.52

What do the column proportion tell us? \(\frac{654}{711}\approx 0.92 \hspace{0.2cm} \text {% adults survived in the total survivors on Titanic}\) \(\frac{57}{711}\approx 0.08 \hspace{0.2cm} \text{(Give a description)}\)

The percentage of passengers who were survived and were adults? \(\frac{654}{2201} \approx 0.30\)

A Mosaic plot is a special type of stacked bar plot. For two variables, the width of the columns is proportional to the number of observations in each level of the variable plotted on the horizontal axis. (here 2092:109 \(\approx\) 19:1)

In this section, we consider a case study on gender discrimination.

• In 1972, as a part of a study on gender discrimination, 48 male bank supervisors were each given the same personnel file and asked to judge whether the person should be promoted to a branch manager job that was described as “routine”.

• The files were identical except that half of the supervisors had files showing the person was male while the other half had files showing the person was female.

• It was randomly determined which supervisors got “male” applications and which got “female” applications.

• Of the 48 files reviewed, 35 were promoted.

• The study is testing whether females are unfairly discriminated against.

• as the files marked with male/female were randomly given to supervisors, the results can be used to evaluate a casual relationship between gender and promotion.

\[ \begin{align} & && Promoted && Not Promoted && Total \\ {Gender} & Male && 21 && 3 && 24 \\ & Female && 14 && 10 && 24 \\ & Total && 35 && 13 && 48 \\ \end{align} \] % of males promoted: 21/24 = 0.875

% of females promoted: 14/24 = 0.583

1. If we were to repeat the experiment we will definitely see that more female files get promoted. This was a fluke.
2. Promotion is dependent on gender, males are more likely to be promoted, and hence there is gender discrimination against women in promotion decisions.
3. The difference in the proportions of promoted male and female files is due to chance, this is not evidence of gender discrimination against women in promotion decisions.
4. Women are less qualified than men, and this is why fewer females get promoted.

1. If we were to repeat the experiment we will definitely see that more female files get promoted. This was a fluke.

2. Promotion is dependent on gender, males are more likely to be promoted, and hence there is gender discrimination against women in promotion decisions.

   Maybe

3. The difference in the proportions of promoted male and female files is due to chance, this is not evidence of gender discrimination against women in promotion decisions.

   Maybe

4. Women are less qualified than men, and this is why fewer females get promoted.

1. “There is nothing going on.”
   • Promotion and gender are independent , no gender discrimination, observed difference in proportions is simply due to chance. \(\rightarrow\) Null hypothesis
2. “There is something going on.” - Promotion and gender are dependent, there is gender discrimination, observed difference in proportions is not due to chance. \(\rightarrow\) Alternative hypothesis

• Then we judge the evidence - “Could these data plausibly have happened by chance if the null hypothesis were true?”

• If they were very unlikely to have occurred, then the evidence raises more than a reasonable doubt in out minds about the null hypothesis.

• Ultimately we must make a decision. How unlikely is unlikely?

• If the evidence is not strong enough to reject then assumption of innocence, the jury returns with a verdict of “not guilty”.

  • The jury does not say that the defendant is innocent, just that there is not enough evidence to convict.

  • The defendant may, in fact, be innocent, but the jury has no way of being sure.

• Said statistically, we fail to reject the null hypothesis.

  • We never declare the null hypothesis to be true, because we simply do not know whether it’s true or not.

  • Therefore we never “accept the null hypothesis”.

• In a trial, the burden of proof is on the prosecution.

• In a hypothesis test, the burden of proof is on the unusual claim.

• The null hypothesis is the ordinary state of affairs (the status quo), so it’s the alternative hypothesis that we consider unusual and for which we must gather evidence.

• We start with a null hypothesis (\(H_0\)) that represents the status quo.

• We also have an alternative hypothesis (\(H_A\)) that represents our research question, i.e. what we’re testing for.

• We conduct a hypothesis test under the assumption that the null hypothesis is true, either via simulation (today) or theoretical methods (later in the courses).

• if the test results suggest that the data do not provide convincing evidence for the alternative hypothesis, we stick with the null hypothesis, If they do, then we reject the null hypothesis in favor of the alternative.

… under the assumption of independence, i.e. leave things up to chance.

If results from the simulations based on the chance model look like the data, then we can determine that the difference between the proportions of promoted files between males and females was simply due to chance (promotion and gender are independent).

If the results from the simulations based on the chance model do not look like the data, then we can determine that the difference between the proportions of promoted filed between males and females was not due to chance, but due to an actual effect of gender (promotion and gender are dependent).

Use a deck of playing cards to simulate the experiment.

1. Let a face card represent \(\textit{not}\) promoted and a non-face card represent a \(\textit{promoted}\). Consider aces as face cards.

   • Set aside the jokers.
   • Take out 3 aces \(\rightarrow\) there are exactly 13 face cards left in the deck (face cards: A, K, Q, J).
   • Take out a number card \(\rightarrow\) there are exactly 35 number (non-face) cards left in the deck (number cards: 2-10).

2. Shuffle the cards and deal them into two groups of size 24, representing males and females.

3. Count and record how many files in each group are promoted (number cards).

4. Calculate the proportion of promoted files in each group and take the difference (male - female), and record this value.

5. Repeat steps 2 - 4 many times.

1. No, the data do not provide convincing evidence for the alternative hypothesis, therefore we can’t reject the null hypothesis of independence between gender and promotion decisions. The observed difference between the two proportions was due to chance.

2. Yes, the data provide convincing evidence for the alternative hypothesis of gender discrimination against women in promotion decisions. The observed difference between the two proportions was due to a real effect of gender.

1. No, the data do not provide convincing evidence for the alternative hypothesis, therefore we can’t reject the null hypothesis of independence between gender and promotion decisions. The observed difference between the two proportions was due to chance.

2. Yes, the data provide convincing evidence for the alternative hypothesis of gender discrimination against women in promotion decisions. The observed difference between the two proportions was due to a real effect of gender.

These simulations are tedious and slow to run using the method described earlier. In reality, we use software to generate the simulations. The dot plot below shows the distribution of simulated differences in promotion rates based on 100 simulations.