Precision-Recall Tradeoff
The trilemma encoded in a 2x2 confusion matrix. Improve precision, reduce recall (and vice versa). Choose based on false positive vs. false negative costs.
Intent & Description
🎯 Intent
Optimize classification threshold based on business costs of different error types. Precision = how many predicted positives are actually positive. Recall = how many actual positives were captured. They move in opposite directions as threshold changes.
📋 Context
You are building a binary classifier. Lower the threshold to catch more positives = higher recall, more false positives. Raise the threshold to be more selective = higher precision, more false negatives. The confusion matrix tells the story: TP, FP, FN, TN trade off against each other.
💡 Solution
Start with default threshold (0.5). Plot precision-recall curve to see the tradeoff landscape. Calculate business cost for different threshold values. Choose threshold that minimizes expected cost. Consider precision-recall AUC for model selection (especially for imbalanced datasets). Use F1-score when precision and recall are equally important.
Real-world Use Case
Source
📌 TL;DR
Precision-recall = classification threshold tradeoff. Lower threshold = higher recall, more false positives. Raise threshold = higher precision, more false negatives. Choose based on error costs.
Advantages
- Directly maps to business costs and user experience
- More informative than accuracy for imbalanced datasets
- Precision-recall curve shows complete tradeoff landscape
- F1-score provides single metric when both matter equally
Disadvantages
- Does not capture true negatives (often important in practice)
- Threshold choice depends on cost estimates which may be uncertain
- Can be gamed by extreme thresholds (e.g., predict all positive = perfect recall)
- ROC curves sometimes preferred for balanced datasets
// Precision-Recall Tradeoff: Threshold optimization
from sklearn.metrics import precision_recall_curve, f1_score
import numpy as np
# Get predicted probabilities
y_proba = model.predict_proba(X_test)[:, 1]
# Calculate precision-recall at different thresholds
precisions, recalls, thresholds = precision_recall_curve(y_test, y_proba)
# Find optimal threshold based on business costs
# Cost = FP_cost * FP + FN_cost * FN
def total_cost(y_true, y_pred, fp_cost=1, fn_cost=10):
fp = ((y_pred == 1) & (y_true == 0)).sum()
fn = ((y_pred == 0) & (y_true == 1)).sum()
return fp_cost * fp + fn_cost * fn
costs = []
for threshold in thresholds:
y_pred = (y_proba >= threshold).astype(int)
cost = total_cost(y_test, y_pred, fp_cost=1, fn_cost=10)
costs.append(cost)
optimal_threshold = thresholds[np.argmin(costs)]
print(f"Optimal threshold: {optimal_threshold:.3f}")
# Alternative: Maximize F1-score
f1_scores = 2 * (precisions * recalls) / (precisions + recalls)
optimal_threshold_f1 = thresholds[np.argmax(f1_scores)]
print(f"F1-optimal threshold: {optimal_threshold_f1:.3f}")