{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Week 6 Lab: Model Comparisons\n",
    "\n",
    "This lab is about the *tools* for comparing models, in two flavors:\n",
    "\n",
    "- **Part 1 (Regression):** compare four delay-discounting models -- exponential,\n",
    "  hyperbolic, Green & Myerson's hyperboloid, and Rachlin's hyperboloid -- using\n",
    "  R-squared, MAE, RMSE, AIC, BIC, and AICc.\n",
    "- **Part 2 (Classification):** compare logistic regression, a decision tree, a\n",
    "  random forest, and an SVM at predicting challenging behavior, using accuracy,\n",
    "  precision, recall, F1, MCC, and ROC-AUC.\n",
    "\n",
    "Information criteria (AIC/BIC/AICc) matter when models differ in parameter count:\n",
    "they penalize complexity so the comparison is fair."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup\n",
    "\n",
    "Import everything you need: `scipy.optimize` (fitting), `sklearn.metrics`, and the four classifier families from scikit-learn."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "# Imports for fitting, metrics, and the four classifiers\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 1: Discounting Model Comparison (Regression)\n",
    "\n",
    "## Task 1: Load the discounting data\n",
    "\n",
    "Each row is one participant/commodity with indifference points at 7 delays plus the `Amount` and `Commodity`."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "# Load participant_discounting_data.csv\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 2: Define the four discounting models\n",
    "\n",
    "Each returns predicted value as a proportion of the amount (so indifference points are normalized to 0-1).\n",
    "\n",
    "- Exponential: $V = e^{-kD}$\n",
    "- Hyperbolic (Mazur): $V = 1/(1+kD)$\n",
    "- Green-Myerson: $V = 1/(1+kD)^s$\n",
    "- Rachlin: $V = 1/(1+kD^s)$"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "# Define exponential_model, hyperbolic_model, green_myerson_hyperboloid, rachlin_hyperboloid\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 3: Helpers to fit a model and compute fit metrics\n",
    "\n",
    "`fit_model_robust` uses global optimization (`differential_evolution`) to minimize squared error. `calculate_fit_metrics` returns R-squared, MAE, RMSE plus the information criteria AIC, BIC, and AICc (small-sample corrected)."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "# Define prepare_data_for_fitting, fit_model_robust, calculate_fit_metrics, and a MODELS dict\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 4: Fit every model to every participant\n",
    "\n",
    "Loop over participants, fit all four models, and collect the metrics into a tidy summary DataFrame. (Fitting all participants on 7 points each takes a moment.)"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "# Loop participants x commodities x models; build summary_df of metrics\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 5: Compare the models across metrics\n",
    "\n",
    "For each fit metric, show its distribution by model so you can see which model wins on which metric. Note how the information criteria treat the 1- vs 2-parameter models differently."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "# Summarize mean metrics by model and plot each metric's distribution by model\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 2: Challenging Behavior Prediction (Classification)\n",
    "\n",
    "## Task 6: Load and prepare the data\n",
    "\n",
    "Load `challenging_behavior_data.csv`, split features/label, make a train/test split, and standardize features (needed by logistic regression and the SVM). The SVM is expensive on large data, so we sample a subset for training speed."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "# Load challenging_behavior_data.csv, split X/y, train/test split, scale features\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 7: Fit four classifiers and score them\n",
    "\n",
    "Fit logistic regression, a decision tree, a random forest, and an RBF SVM. For each, compute accuracy, precision, recall, F1, MCC, and ROC-AUC, and add its ROC curve to a shared plot. Scale-sensitive models (logistic, SVM) use the scaled features."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "# Fit the four classifiers, compute the six metrics each, plot ROC curves\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 8: Results table and interpretation\n",
    "\n",
    "Put the classification metrics in one table. Which model would you choose, and on which metric -- and why might accuracy alone be misleading if challenging behavior is rare?"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "# Assemble the metrics table and interpret\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Wrap-up\n",
    "\n",
    "Across both parts: when does a more complex model (extra parameter, ensemble) actually earn its complexity? Tie your answer to what the information criteria showed in Part 1 and to the precision/recall trade-off in Part 2."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}