Correlation and regression

Scatter diagram first. Before computing any number, plot the data. The picture tells you whether a linear model is even sensible. A clear curve, U-shape, or fan-out should make you cautious.

The PMCC.

$r = \dfrac{S_{xy}}{\sqrt{S_{xx} S_{yy}}}, \qquad -1 \le r \le 1.$

Value of $r$	Meaning
$r = 1$	Perfect positive linear pattern
$r \approx 0.8$	Strong positive linear pattern
$r \approx 0.5$	Moderate positive
$r \approx 0$	No linear pattern
$r \approx -0.7$	Strong negative
$r = -1$	Perfect negative

Compute via the booklet sums. Plug summary statistics ($\sum x$, $\sum y$, $\sum x^{2}$, $\sum y^{2}$, $\sum xy$, $n$) into

$S_{xx} = \sum x^{2} - \dfrac{(\sum x)^{2}}{n}, \;\; S_{yy} = \sum y^{2} - \dfrac{(\sum y)^{2}}{n}, \;\; S_{xy} = \sum xy - \dfrac{\sum x \sum y}{n}.$

Then plug into the PMCC formula.

Worked example. $n = 10$, $\sum x = 50$, $\sum y = 70$, $\sum x^{2} = 285$, $\sum y^{2} = 540$, $\sum xy = 386$.

• $S_{xx} = 285 - 250 = 35$.
• $S_{yy} = 540 - 490 = 50$.
• $S_{xy} = 386 - 350 = 36$.
• $r = 36/\sqrt{1750} \approx 0.861$ — strong positive linear correlation.

Goal. Find the straight line that best summarises the linear relationship for predicting $y$ from $x$.

Method (least squares). Choose $a, b$ to minimise

$\sum (y_i - (a + bx_i))^{2}.$

The minimisers (in the formula booklet) are

$b = \dfrac{S_{xy}}{S_{xx}}, \qquad a = \bar{y} - b\bar{x}.$

So the regression line is

$y = a + bx = \bar{y} + b(x - \bar{x}).$

It always passes through the mean point $(\bar{x}, \bar{y})$.

Worked example. Same data as before: $\bar{x} = 5$, $\bar{y} = 7$, $S_{xy} = 36$, $S_{xx} = 35$.

• $b = 36/35 \approx 1.029$.
• $a = 7 - 1.029 \times 5 \approx 1.857$.
• Line: $y \approx 1.86 + 1.03 x$.

Slope interpretation. A one-unit increase in $x$ is associated with an estimated increase of $b$ units in $y$. Phrase it in context: 'each extra hour studied is associated with an estimated $1.03$-mark increase in test score'.

$y$ on $x$ vs $x$ on $y$. These are DIFFERENT lines. $y$ on $x$ has slope $S_{xy}/S_{xx}$ and minimises vertical residuals; $x$ on $y$ has slope $S_{xy}/S_{yy}$ and minimises horizontal residuals. They coincide only when $r = \pm 1$.

Interpolation. Substituting an $x$ within the observed range into the regression line gives an estimate of the corresponding $y$. This is interpolation and is generally reliable, especially when $r$ is high.

Extrapolation. Substituting an $x$ outside the observed range gives an estimate that may be far from reality. The linear pattern observed in the data is not guaranteed to continue. Examples of failure modes:

• Saturation: an effect levels off at high $x$.
• Sign change: the relationship reverses outside the range.
• Non-linear regime: the true relationship is quadratic or exponential, which looks linear over a small window.

Worked example. Regression $y = 12 + 1.4x$ from data with $x \in [5, 25]$, $r = 0.96$.

• $x = 18$: $y = 37.2$ — within range, interpolation, reliable.
• $x = 60$: $y = 96$ — well outside range, extrapolation, unreliable.

Best-practice template for an exam answer.

'This estimate uses the regression line at $x = X$. Since $X$ lies within / outside the data range $[a, b]$, the estimate is reliable / unreliable. [If unreliable]: the linear model may not extend beyond the data, so the prediction at $X$ is an extrapolation and should be treated with caution.'

Coding simplifies arithmetic when raw data has large magnitudes or awkward offsets. For example $u = x - 100$, $v = (y - 50)/2$ converts heights around $100$ cm and weights around $50$ kg to small whole numbers.

Effect on $r$. For linear codings $u = \alpha x + \beta$, $v = \gamma y + \delta$ with $\alpha, \gamma > 0$:

$r_{uv} = r_{xy}.$

The PMCC is dimensionless and unaffected by shift or positive scaling. (A negative scale factor flips the sign of $r$.)

Effect on the regression line. Slope and intercept DO change.

Method.

1. Find the regression line of $v$ on $u$ from the coded data.
2. Substitute the coding relations $u = \alpha x + \beta$, $v = \gamma y + \delta$.
3. Rearrange to express $y$ in terms of $x$.

Worked example. $u = x - 100$, $v = (y - 50)/2$. Coded regression: $v = 1.5 + 0.4 u$.

• Substitute: $(y - 50)/2 = 1.5 + 0.4(x - 100)$.
• Multiply by $2$: $y - 50 = 3 + 0.8(x - 100) = 3 + 0.8x - 80$.
• Rearrange: $y = -27 + 0.8 x$.

So the regression line of $y$ on $x$ is $y = -27 + 0.8x$. PMCC unchanged at whatever value it had on the coded data.

Correlation does not imply causation. A high $r$ between two variables can arise from:

1. $X$ causes $Y$ (the interpretation we'd like).
2. $Y$ causes $X$ (reverse causation).
3. A confounding variable $Z$ causes both — neither $X$ nor $Y$ causes the other.
4. Pure coincidence in a small sample.

Famous example. Ice cream sales and drowning deaths are strongly correlated across summers. Neither causes the other; hot weather (confounder) causes both.

Outliers. A single extreme point can shift $r$ substantially. Always check the scatter diagram, identify outliers, and consider whether they are real measurements, recording errors, or come from a different population.

Non-linearity. $r \approx 0$ does not mean 'no relationship' — only 'no linear relationship'. Quadratic, exponential, sinusoidal relationships can have $r = 0$ over a symmetric range.

Extrapolation. Reliable predictions require interpolation within the observed range of $x$.

Linearity assumption. The regression line is only sensible if the scatter looks roughly linear. Otherwise transform the variables (e.g. fit $\log y$ vs $x$ for exponential data) or use a different model.