{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Machine Learning Lab: Classifying Behavioral Function from Functional Analysis Data\n",
    "\n",
    "In this lab, you will use supervised machine learning to classify the function of problem behavior from functional analysis (FA) summary data. This mirrors a task clinicians perform routinely: examining response rates across FA conditions to determine whether behavior is maintained by attention, escape, tangible reinforcement, or automatic reinforcement.\n",
    "\n",
    "You will:\n",
    "1. Explore the dataset and understand its structure\n",
    "2. Build a decision tree classifier\n",
    "3. Evaluate model performance with a confusion matrix and classification report\n",
    "4. Investigate overfitting across different tree depths\n",
    "5. Build a random forest and compare to the single tree\n",
    "6. Discuss the prediction-explanation tradeoff\n",
    "\n",
    "**Objectives:**\n",
    "- Gain hands-on experience with scikit-learn's classification tools\n",
    "- Understand how decision trees make classification decisions\n",
    "- Appreciate the tradeoff between model complexity and interpretability"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup\n",
    "\n",
    "Run the cell below to import the libraries you will need."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.tree import DecisionTreeClassifier, plot_tree\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.metrics import (\n",
    "    accuracy_score,\n",
    "    confusion_matrix,\n",
    "    ConfusionMatrixDisplay,\n",
    "    classification_report,\n",
    ")\n",
    "\n",
    "np.random.seed(42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 1: Load and Explore the Data\n",
    "\n",
    "Load the file `fa_classification_data.csv`. This dataset contains simulated functional analysis summary data for 60 participants. Each participant has:\n",
    "\n",
    "- `attention_rate`: response rate during the attention condition\n",
    "- `escape_rate`: response rate during the escape condition\n",
    "- `tangible_rate`: response rate during the tangible condition\n",
    "- `play_rate`: response rate during the play (control) condition\n",
    "- `function`: the true behavioral function (attention, escape, tangible, or automatic)\n",
    "\n",
    "Tasks:\n",
    "1. Load the CSV file into a pandas DataFrame\n",
    "2. Print the shape and first few rows\n",
    "3. Check the distribution of the `function` column\n",
    "4. Compute descriptive statistics for each rate column grouped by function"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "df = pd.read_csv(\"fa_classification_data.csv\")\n",
    "print(\"Shape:\", df.shape)\n",
    "print(df.head())"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "feature_cols = ['attention_rate', 'escape_rate', 'tangible_rate', 'play_rate']\n",
    "\n",
    "print(\"Class distribution:\")\n",
    "print(df['function'].value_counts())\n",
    "print(\"\\nMean rate by function:\")\n",
    "print(df.groupby('function')[feature_cols].mean())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 2: Prepare Features and Labels\n",
    "\n",
    "Separate the data into:\n",
    "- **X** (features): the four rate columns\n",
    "- **y** (labels): the `function` column\n",
    "\n",
    "Then split into training (75%) and test (25%) sets using `train_test_split` with `random_state=42` and `stratify=y` to ensure each function is represented proportionally in both sets.\n",
    "\n",
    "Print the size of each set and verify the class distribution in both."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "X = df[feature_cols]\n",
    "y = df['function']\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    X, y, test_size=0.25, random_state=42, stratify=y)\n",
    "\n",
    "print(\"Train set:\", X_train.shape, \" Test set:\", X_test.shape)\n",
    "print(\"\\nTrain class counts:\\n\", y_train.value_counts())\n",
    "print(\"\\nTest class counts:\\n\", y_test.value_counts())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 3: Train a Decision Tree Classifier\n",
    "\n",
    "Train a `DecisionTreeClassifier` with `random_state=42` (use default hyperparameters for now).\n",
    "\n",
    "After training:\n",
    "1. Print the training accuracy\n",
    "2. Print the test accuracy\n",
    "3. Note whether there is a gap between the two (this is your first signal about overfitting)"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "tree = DecisionTreeClassifier(random_state=42)\n",
    "tree.fit(X_train, y_train)\n",
    "\n",
    "train_acc = accuracy_score(y_train, tree.predict(X_train))\n",
    "test_acc = accuracy_score(y_test, tree.predict(X_test))\n",
    "print(f\"Training accuracy: {train_acc:.3f}\")\n",
    "print(f\"Test accuracy:     {test_acc:.3f}\")\n",
    "print(f\"Gap (train - test): {train_acc - test_acc:.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 4: Visualize the Decision Tree\n",
    "\n",
    "Use `sklearn.tree.plot_tree` to visualize the trained decision tree.\n",
    "\n",
    "Tips:\n",
    "- Use `filled=True` for color-coded nodes\n",
    "- Use `feature_names` and `class_names` for readability\n",
    "- Set `figsize` large enough to read the tree (e.g., 20x10)\n",
    "\n",
    "Examine the tree: What features does it split on first? Does the tree structure make intuitive sense given how FA data are typically interpreted?"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(figsize=(20, 10))\n",
    "plot_tree(tree, filled=True, feature_names=list(X.columns),\n",
    "          class_names=list(tree.classes_), fontsize=9, ax=ax)\n",
    "plt.title(\"Default Decision Tree\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Describe what you observe about the tree structure here.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 5: Confusion Matrix and Classification Report\n",
    "\n",
    "Evaluate the decision tree on the **test set**:\n",
    "\n",
    "1. Compute and display the confusion matrix using `ConfusionMatrixDisplay`\n",
    "2. Print the full `classification_report` (precision, recall, F1 for each class)\n",
    "\n",
    "Which function is easiest to classify? Which is hardest? Why might that be?"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "y_pred = tree.predict(X_test)\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(6, 5))\n",
    "ConfusionMatrixDisplay.from_predictions(y_test, y_pred, ax=ax)\n",
    "ax.set_title(\"Decision Tree - Test Set\")\n",
    "plt.show()\n",
    "\n",
    "print(classification_report(y_test, y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Interpret the confusion matrix and classification report here.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 6: Investigate Overfitting Across Tree Depths\n",
    "\n",
    "Decision trees are prone to overfitting -- they can memorize the training data perfectly but perform poorly on new data. The `max_depth` parameter controls the tree's complexity.\n",
    "\n",
    "1. Train decision trees with `max_depth` values from 1 to 10\n",
    "2. For each depth, record the training accuracy and test accuracy\n",
    "3. Plot both curves on the same figure\n",
    "\n",
    "Identify the `max_depth` value that gives the best test accuracy. What happens to the gap between training and test accuracy as depth increases?"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "depths = list(range(1, 11))\n",
    "train_scores, test_scores = [], []\n",
    "for d in depths:\n",
    "    t = DecisionTreeClassifier(max_depth=d, random_state=42)\n",
    "    t.fit(X_train, y_train)\n",
    "    train_scores.append(accuracy_score(y_train, t.predict(X_train)))\n",
    "    test_scores.append(accuracy_score(y_test, t.predict(X_test)))\n",
    "\n",
    "best_depth = depths[int(np.argmax(test_scores))]\n",
    "\n",
    "plt.figure(figsize=(8, 5))\n",
    "plt.plot(depths, train_scores, 'o-', label='Training accuracy')\n",
    "plt.plot(depths, test_scores, 's-', label='Test accuracy')\n",
    "plt.axvline(best_depth, color='gray', linestyle='--', alpha=0.7,\n",
    "            label=f'Best depth = {best_depth}')\n",
    "plt.xlabel('max_depth'); plt.ylabel('Accuracy')\n",
    "plt.title('Overfitting vs. Tree Depth'); plt.legend()\n",
    "plt.show()\n",
    "print(\"Best max_depth by test accuracy:\", best_depth,\n",
    "      f\"(test acc = {max(test_scores):.3f})\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Describe the overfitting pattern and the optimal max_depth here.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 7: Build a Random Forest\n",
    "\n",
    "A random forest is an ensemble of many decision trees, each trained on a random subset of the data and features. The ensemble averages over individual trees' errors, typically producing better generalization.\n",
    "\n",
    "1. Train a `RandomForestClassifier` with `n_estimators=100` and `random_state=42`\n",
    "2. Report training and test accuracy\n",
    "3. Compare to the best single decision tree from Task 6\n",
    "4. Print the feature importances -- which FA condition rates are most important for classification?"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "rf = RandomForestClassifier(n_estimators=100, random_state=42)\n",
    "rf.fit(X_train, y_train)\n",
    "\n",
    "rf_train = accuracy_score(y_train, rf.predict(X_train))\n",
    "rf_test = accuracy_score(y_test, rf.predict(X_test))\n",
    "print(f\"Random Forest training accuracy: {rf_train:.3f}\")\n",
    "print(f\"Random Forest test accuracy:     {rf_test:.3f}\")\n",
    "print(f\"(Best single tree test accuracy was {max(test_scores):.3f})\")\n",
    "\n",
    "print(\"\\nFeature importances:\")\n",
    "for name, imp in zip(X.columns, rf.feature_importances_):\n",
    "    print(f\"  {name}: {imp:.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 8: Visualize Feature Importances\n",
    "\n",
    "Create a bar chart of the random forest's feature importances. Label the bars with the feature names.\n",
    "\n",
    "Do the importances align with your intuition about how clinicians interpret FA data? Are all four condition rates equally useful, or do some carry more information?"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "importances = rf.feature_importances_\n",
    "order = np.argsort(importances)[::-1]\n",
    "\n",
    "plt.figure(figsize=(8, 5))\n",
    "plt.bar([X.columns[i] for i in order], importances[order], color='steelblue')\n",
    "plt.ylabel('Importance')\n",
    "plt.title('Random Forest Feature Importances')\n",
    "plt.xticks(rotation=20)\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Interpret the feature importances here.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 9: The Prediction-Explanation Tradeoff\n",
    "\n",
    "Summarize your results in a comparison table:\n",
    "\n",
    "| Model | Training Accuracy | Test Accuracy | Interpretable? |\n",
    "|-------|-------------------|---------------|----------------|\n",
    "| Decision Tree (default) | ? | ? | ? |\n",
    "| Decision Tree (best depth) | ? | ? | ? |\n",
    "| Random Forest | ? | ? | ? |\n",
    "\n",
    "Then write a discussion (3-4 paragraphs) addressing:\n",
    "\n",
    "1. **Prediction vs. explanation**: The random forest likely predicts better, but can you explain *why* it classifies a given case the way it does? How does this compare to the single decision tree? In clinical behavior analysis, is prediction or explanation more important?\n",
    "\n",
    "2. **Clinical utility**: Could a classifier like this be useful in practice? What are the limitations of training on simulated data? What real-world complications would arise (e.g., undifferentiated functions, multiply-maintained behavior, variable FA protocols)?\n",
    "\n",
    "3. **Modeling philosophy**: How does the ML approach here differ from the parametric models (e.g., matching law, demand curves) we have used earlier in the course? What do we gain and lose by moving from theory-driven to data-driven modeling?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Write your discussion here.*"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}