Proposal to Add Penalized Brier Score (PBS) and Penalized Log Loss (PLL) Methods

[Ostovane]

Hello @frazane ,

First, thank you for maintaining the excellent scoringrules package! I want to propose integrating the Penalized Brier Score (PBS) and Penalized Log Loss (PLL) methods from Ahmadian et al.'s work, which directly address limitations of traditional probabilistic scoring rules in classification tasks.

Key Advantages

1. Fixes Critical Flaws

   • Resolves Brier Score/Log Loss failure to guarantee proper scoring in finite-sample regimes (Theorems/Proofs)
   • Introduces penalty term: `penalty = ensuring incorrect predictions always score worse than correct ones

2. Theoretical Guarantees

   • Strictly proper by construction
   • Consistent model selection even with imperfect models

3. Practical Benefits

   • Better guidance for early stopping and checkpointing
   • Robust to class imbalance
   • Computationally equivalent to traditional scores

Implementation Details

The reference Python implementation provides:

• Vectorized PBS/PLL calculations
• Unit tests verifying propriety conditions
• Benchmarks against standard metrics

Would you consider adding these to scoringrules?

Paper: "Superior scoring rules..." (Ahmadian et al., Int. J. Approx. Reason. 2024)

Thank you for considering this contribution to improve probabilistic scoring!

[sallen12]

Hi @Ostovane,

Thanks a lot for the suggestion, the penalised scores are very interesting. At the moment, we are working towards v.1.0 of the package, which will essentially replicate the functionality of the scoringRules R package, so are currently not looking to include additional scoring rules. But we may later add other scores, in which case the penalised scores could be a useful addition - I'll therefore leave this issue open for now.

I disagree though with the critical flaws of the Brier score and log score that you mention. I would argue that there is no "correct" probabilistic prediction. If interest is on assessing the binary classifier defined by the argmax of the probability vector, then scoring rules can be derived to achieve this, but it is not necessarily a flaw if a scoring rule does not satisfy the notion of "superiority" defined in the paper. An alternative scoring rule to assess the ability of the forecasts to construct binary classifiers would be something like $S_{01}(p, i) = 1\lbrace i \neq \text{argmax } p \rbrace$, which is a kind of 0-1 score, equal to 0 if the forecast $p$ assigns the highest probability to the category that occurs ($i$), and 1 otherwise. It is straightforward to show that $S_{01}$ is proper, since it is minimised in expectation whenever $p$ has the same argmax as the true distribution $q$ (it is therefore not strictly proper).

Interestingly, this is closely related to the penalised scores you mention. The penalised Brier score, for example, can alternatively be written as

$$S_{PBS}(p, i) = S_{BS}(p, i) + \frac{c-1}{c} S_{01}(p, i).$$

As the sum of a strictly proper and a proper scoring rule, $S_{PBS}$ is therefore itself strictly proper. An analogous argument holds for the log score. Extensions of these penalised scores to other scoring rules are then straightforward.

We may anyway later add the functionality to calculate the 0-1 score mentioned above, in which case the penalised scores become easy to calculate since both of the component scores would be in scoringrules already.

Note also that the proof of propriety of the penalised scores in Theorem 5 of the paper is slightly incorrect. It is argued that $S_{BS}(q, q) = S_{PBS}(q, q)$, but this is not true. In particular, for any two probability vectors $p, q$,

$$S_{PBS}(p, q) = S_{BS}(p, q) + \frac{c - 1}{c}P(Y \neq \text{argmax } p),$$

where $Y \sim q$. This latter term is not equal to zero when $p = q$. The proof above is an alternative way of showing strict propriety.

[Ruhallah93]

Thank you very much for your thoughtful and constructive feedback.

I appreciate your clarification regarding the scope of your current development efforts towards version 1.0 of the package and your openness to potentially including additional scoring rules in the future. I agree that the penalised scores could naturally align with such an extension, especially if support for the 0-1 score is introduced, as this would implement the penalised scores more straightforwardly.

Regarding the theoretical discussion, I fully acknowledge your perspective that the concept of a "correct" probabilistic prediction is open to interpretation and may indeed fall more within the realm of philosophy than formal statistics. My intention was not to assert an objective truth in that regard, but rather to propose an alternative viewpoint that could provide a different, and perhaps fairer, framework for evaluation, especially in cases where traditional scoring rules may yield unintuitive rankings of predictive performance.

As you rightly note, in binary classification, the issue of "superiority" is less pronounced, since the correct prediction typically results in lower error. However, in multi-class settings, it is indeed possible for a forecast assigning the highest probability to the correct class to incur a higher score (i.e., greater penalty) than one assigning it to an incorrect class. This phenomenon motivated my exploration of penalised scoring rules as a way to bridge that gap in evaluation logic.

The two penalised scoring functions I proposed are intended merely as a first step toward developing a broader family of such scores. I believe there is ample room for refinement and innovation in this space, and I hope future work can build upon this initial foundation.

Concerning your point about the proof of propriety in Theorem 5, I agree that the equivalence between $S_{PBS}(q, q)$ and $S_{BS}(q, q)$ as stated requires clarification. Your formulation,

$$ S_{PBS}(p, q) = S_{BS}(p, q) + \frac{c - 1}{c} \mathbb{P}(Y \neq \arg\max p), \quad Y \sim q, $$

makes clear that this expression is nonzero even when $p = q$, due to the probabilistic nature of $Y$. However, from the perspective adopted in the paper—treating predictions with $p = q$ as "correct" in a decision-theoretic sense—the penalty term is effectively neutralized. Therefore, while the proof may require a more rigorous justification, the conclusion of strict propriety remains valid and could alternatively be established using the argument you suggested, namely by viewing the penalised score as the sum of a strictly proper and a proper scoring rule.

Once again, I appreciate your engagement and the helpful insights. I believe that the dialogue around these issues helps clarify the scope, applicability, and interpretation of probabilistic scoring rules, and I look forward to seeing how the ideas evolve in future versions of the package and the broader literature.