Midterm 1 Review

1. Introduction:
   1. Potential concepts:
      1. Data Exploration
      2. Data Visualization
   2. Skills expected
      1. You know the name and purpose of each step we’ve covered so far
2. Experiment Design: 
   1. Concepts:
      1. The role of optimization criteria, proxy measures, and Goodhart’s law
      2. Types of experiments and all associated vocab : Observational studies (Cross-sectional, Retrospective, Prospective), Experiments, Synthetic data
   2. Skills expected
      1. Understand potential issues with experimental design, including sampling, as well other potential sources of error. You should be comfortable designing an experiment, or pointing out flaws in an experiment you are shown.
      2. Generally, the potential issues with surveys and self-reported data (Max only)
      3. Understand what blinding and placebo are for
3. Pandas
   1. Concepts:
      1. Pandas: Understand what a dataframe and series are, and what pandas is generally used for.
   2. Skills expected
      1. Be able to write pandas statements of similar difficulty to the easier parts of the homework. You will not be provided with any sort of pandas cheat sheet.
4. Data Types
   1. Concepts
      1. Data definition
      2. Types of Data and their subtypes
      3. Data Formats - TSV, CSV, JSON, HTML, XML
      4. Images: Gif, Lossless and Lossy compression
      5. Databases and what they’re for
   2. Skills expected
      1. Be able to describe or identify any of the listed filetypes and their purposes.
5. Probability
   1. Concepts
   2. Bayes Rule
   3. Conditional Probabilities
   4. Expected Value
   5. All types of distributions - Uniform, Gaussian, Poisson, Zero-Inflated Poisson Distribution, Bernoulli Trial, Binomial Distribution
   6. Central Limit Theorem
   7. Summary Statistics
      1. Measure of Central Tendency: Mean, median, mode (Including equation for arithmetic mean)
      2. Measures of Variance: Standard deviation, variance (You should know what these are, how they work and what they represent)
      3. Other Descriptors - Skew, Modality
   8. Skills expected
      1. Understand distributions conceptually
      2. Probability and statistics questions akin to those on the homework
      3. Name and identify distributions based on descriptions
      4. Understand and apply the central limit theorem
      5. Understand how mean, median and mode relate to each other, and how to use them to learn about data
6. Hypothesis Testing
   1. Concepts
      1. Hypothesis Testing
      2. Null and Alternative Hypothesis
      3. Statistical Significance
      4. P-Values
      5. Type I and Type II Errors
      6. Different Types of Statistical Tests and when to apply them
         1. Z-Test
         2. T-Test ( two sample, one sample, paired)
         3. Chi Square Test
         4. ANOVA test
   2. Skills expected
      1. Given a scenario, you need to identify which hypothesis test will be applicable, its null and alternate hypothesis
      2. Understand what a p value is and what it represents
      3. Understand why hypothesis tests are important and what they do
7. Data Visualization
   1. Concepts
      1. What Type of Visualization to Use?
         1. Comparison
         2. Correlation
         3. Part-To-Whole
         4. Data Over Time
         5. Distribution
      2. Design Principles
   2. Skills expected
      1. Given a scenario, identify the best type of visualization to use
      2. Be able to read, understand, and draw graphs
8. Data Exploration
   1. Concepts
      1. What is exploratory data analysis (EDA)
      2. Pearson’s correlation coefficient and what it means
9. Data Cleaning
   1. Concepts
      1. Duplicated records
      2. Evolving data practices
      3. Outlier detection, z-score
      4. Data missing at random / Data missing not at random
      5. Different types of imputation
      6. Boundary conditions
      7. Data errors and what causes them
      8. How to detect corrupt or incorrect data
   2. Skills expected
      1. Be able to reason about missing data
      2. Know when to replace / when not to replace outliers
      3. Be able to pick the correct imputation type for the situation
      4. Be able to identify data errors from graphs
      5. Knowing what type of information people might lie about