Statistics is not just the realm of data scientists. All types of jobs use statistics. Statistics are important for making decisions, new discoveries, investments, and predictions. Whether the subject is political races, sports rankings, shopping trends, or healthcare advancements, statistics is an instrument for understanding your favorite topic at a deeper level. In this course, Professor Eddie Davila offers beginner-level lessons, so you too can master the terms, formulas, and techniques needed to perform the most common types of statistics. Eddie covers several examples of data and charts. He explains how to find the middle of your data set, as well as the mean and median. He introduces range, then goes into standard deviation and what to do with outliers. These techniques help you understand your data, prove theories, and save time, money, and other valuable resources—all by understanding the numbers.
With Charts/Graphs/Tables
- meaning of data becomes clearer, data starts to tell stories (statistical story telling), and it can educate and empower us
- it illustrates dominance and reveal changes in data, facilitating better data organization
- drives curiosity to look for deeper insights and collect better datasets
- can be used to learn, decide, persuade
Data can be visualized as - raw tables / frequency table / dot plot / bar plot or histogram / relative frequency / relative frequency as % distribution / relative frequency histogram / pie chart
With charts, the objective is to not look for "right answers", but read a story behind. Think what it means to you and embrace questions as an opportunity to open a discussion about the data and decisions it might provoke.
- Every new set of data is filled with mystery: Is there something odd, interesting, common knowledge?
- We typically start reading book from beginning, but for data, start with middle / center (specially with large dataset)
- Statistical tools to know center of data: mean (average), median (middle), mode (most common)
- Do not rely on mode too much to make significant decisions from data
- Together mean, median and mode help you understand the data at hand
- For a dataset if median is 40 and mean is 70, it can be interpreted as: half the datapoints are under 40, but numbers above the median must have been very large as they push average to a higher value (i.e. 70)
- For a dataset if median is 40, and mode is 20 (provided there are 7 data points in total and 20 occur three times), it can be interpreted as: all occurrences of most frequent value (i.e. 20) are below median which is likely to push the average down significantly
- If no value appears more than once, dataset has no mode
- If two values with same frequency appears in dataset, it is referred to as bimodal data and both are considered modes
- If even number of values are there in the dataset, we take average of two middle values to calculate median
- It is possible that for some data mean is lower than median, if many data points are below median which causes lower average
Mean of the very diverse datasets can be identical (ex. {29,30} and {0,60}). Therefore, range and standard deviation helps to understand how datasets are spread out (statistically called "variability")
- Range = largest - smallest data point
- For a dataset if mean = 60, median = 58, and range = 70, it can be interpreted as: There might be one rogue datapoint (outlier) due to which the range is quite large. Visualizing histogram can be helpful in this case but more generally, range alone is not much useful as a measure of variability.
- Z-score is a data point's distance from the mean (can be helpful to identify/describe datapoints which are very small or very large)
- Z-score = (data point - mean) / standard deviation, which can be interpreted as: the given data point is {Z-score} standard deviation away from the mean in the positive/negative direction.
- Standard deviation is an average squared distance from the dataset's mean to all individual datapoints
- Standard deviation = sqrt ( (sum(data point - mean))^2 / (sample size - 1) )
- You cannot infer variation of datasets only based on the standard deviation. It depends on what you are actually measuring e.g. weather (more variable) vs. salaries (less variable)
- Standard deviations can be helpful in understanding datasets, their individual datapoints and how the datapoints relate to central tendency
- Standard deviations can be helpful in generating interesting questions about data collection methods, entire pool of data and even individual datapoints
- Empirical rule is helpful for understanding the distribution of data points in our datasets provided it is symmetrical or normally distributed data (bell shaped data distribution symmetrically distributed/mirrored across mean)
- Empirical rule: "most data points in a given dataset fall within three standard deviations of the mean"
- I.e. 68% datapoints within +/-1 SD, 95% datapoints within +/-2 SD, 99.7% datapoints within +/-3 SD and only 0.3% data are beyond 3 SD (which is considered to be outlier)
- Empirical rule, bell curve and z-scores of data points are great tools in understanding dataset as a whole as well as individual datapoints in the dataset
- What exactly makes something an outlier?: datapoint at an abnormal (no definitive definition) distance from the other values in the dataset
- Rule based outlier definition ex. more than 2 SD away datapoint, new or unprecedented value i.e. 25% one-day drop in stock market
- What to do with outlier?: ignore (considering it as an unlikely event) or consider it as a learning opportunity (to further investigate as a cause of beginning of a new trend)
- Factors to consider:
(i) knowledge (ex. person knows something which others havn't figured out yet, strange weather aided the attempt)
(ii) training and equipment (ex. did they use different training or equipment/technology)
(iii) ask questions: is this really an outlier, how did this happen, what can we learn, what needs to change - Sometimes a mass of closely distributed datapoints can be very instructive but sometimes lone outlier can provide us with a brand new perspective
Two sets of temperature data are identical, except one is in °C and the other is in °F. The standard deviations are 20°C for set 1 and 68°F for set 2. Which set has the greater variability?: Since both sets contain the same data (just using different scales), the variability is identical.