BEGIN:VCALENDAR VERSION:2.0 PRODID:-//Talks.cam//talks.cam.ac.uk// X-WR-CALNAME:Talks.cam BEGIN:VEVENT SUMMARY:Weighted ROC and Murphy Curves in Cost Space and Applications - Ni kolai Kolev (Sao Paulo State University) DTSTART:20250605T091000Z DTEND:20250605T093000Z UID:TALK230854@talks.cam.ac.uk DESCRIPTION:This work\, jointly with Yuri Verges\, is motivated by the dif ferent methods developed by Herná\;ndes-Orallo et al. (2011\, 2013)\ , \; Dimitriadis et al. (2021)\, Shao et al. (2023)\, Dimitriadis et a l. (2024)\, which complement each other. \;\n \;\nDimitriadis et a l. (2021) have introduced the \; CORP approach based on \; nonpara metric isotonic regression by using the traditional pool-adjacent-violator s (PAV) algorithm for calibration of probabilistic forecasts. \; The C ORP approach generates \;reliability curves\, being the graph of the P AV-(re)calibrated forecast probability. \;\n \;\nDimitriadis et al . (2024) proposed a triplet of diagnostic tools\, each with different capa bilities for performance evaluation of binary classifiers: reliability cur ves produced by CORP with the idea to diagnose calibration\, receiver oper ating characteristic (ROC) curves which evaluate discrimination ability an d Murphy curves for overall assessment \; of predictive performance. I n their Theorem 3 the authors \; show that if X and Z are calibrated p robabilistic forecasts for the binary outcome Y\, then the following state ments are \; equivalent: (i) X is sharper than Z\; (ii) X dominates Z in ROC sense and (iii) X dominates Z in Murphy sense. \;\n \;\nThe  \; area under the ROC curve (to be abbreviated \; AUC) is a popul ar metric of the accuracy of quantitative diagnostic test. However\, the t raditional machine learning models trained with AUC are not well studied f or cost sensitive decision problems. The notable exeption is the work of H erná\;ndes-Orallo et al. (2013) who demonstrate that the \; ROC curves can be transformed into the cost space. This update is equivalent t o computing the area under the convex hull of ROC curves. Thus\, the \ ; AUC can be seen as the performance of the model with uniform cost distri bution\, being an unreasonable assumption for practical needs.\n \;\nE xtending the idea of Herná\;ndes-Orallo et al. (2013)\, Shao et al. (2023) introduced the \; notion of weighted ROC curve in cost space jo ining the robustness of the model to the class distribution and cost distr ibution. In other words\, extending AUC to the non-uniform \; cost-sen sitive learning. The authors \; construct a new environment where the costs are treated like a dataset to share out an arbitrary unknown cost di stribution and launch a weighted version of AUC (to be abbreviated WAUC) w here the cost distribution can be incorporated into its calculation via de cision threshold. \;\n \;\nThus\, Shao et al. (2023) develop the f ollowing \; two-level algorithm \; to bridge WAUC and cost: the in ner-level problem approximates the optimal threshold from sampling costs\, and the \; outer-level problem minimizes the WAUC loss over the optim al threshold distribution. Such an advanced \; approach fits better to the real world cost-sensitive scenario. \;\n \;\nTaking into acco unt the equivalent statements of Theorem 3 in Dimitriadis et al. (2024)\, our goal is to apply the methodology suggested by Shao et al. (2023) in tw o directions: \;\n \;\n1. It is well-known that the Gini concentra tion index (to be denoted by G) is related to AUC as follows: G = 2AUC - 1 . Our proposal is to use the Leimkhuler curve which\, in economics and rel iability context\, plots the cumulative proportion of total productivity a gainst the cumulative proportion of sourses arranged in decreasing order. The area under the Leimkhuler curve\, to be denoted by AUL\, satisfies the same relation\, i.e.\, G = 2AUL - 1\, consult Burrell (1991) and Balakris hnan et al. (2010). \; Moreover\, the Leimkhuler curve has similar sha pe as ROC curve: it begins at (0\,0) and terminates at (1\,1)\, being non- decreasing and concave. \;\n \;\nHence\, the Leimkhuler curve is a cumulative distribution function of a random variable having the Bradford distribution. We will use an analogous procedure as the one suggested by Shao et al. (2023) with respect to the Leimkhuler curve to get the corresp onding cost space and compare their results obtained for WAUC procedure.\n  \;\n2. It is well known that the area below the Murphy curve is equal to the half of the mean Brier score and the area below the Brier curve (s ee Herná\;ndes-Orallo et al. (2011)) is equal to the \; mean Bri er score. Then\, using again the methodology of Shao et al. (2023) we will extend the results of Herná\;ndes-Orallo et al. (2011) when the cos t distribution is different than the uniform one. Finally\, we will show t he corresponding weighted versions in cost space of Murphy curves. \;& nbsp\;\n \;\n \;\nReferences:\n \;\nBalakrishnan\, N.\, Sarabi a\, J.M. and Kolev\, N. (2010). A simple relation between Leimkhuler curve and the mean residual life. \; Joutnal of Informetrics 4\, 602-607.\n  \;\nBurrell\, Q. (1991). The Bradford distribution and the Gini index . Scientometrics 21\, 181-194. \;\n \;\nDimitriadis T.\, Gneiting T. and Jordan\, A. I. (2021). Stable reliability diagrams for probabilisti c classifiers. Proceedings of the National Academy of Sciences 118\, Artic le e2016191118.\n \;\nDimitriadis\, T.\, Gneiting\, T.\, Jordan\, A. I . \; and \; Vogel\, P. (2024). Evaluating probabilistic classifier s: The triptych. International Journal of Forecasting 40\, 1101-1122.\n&nb sp\;\nHerná\;ndez-Orallo J.\, \; Flach P. \; and \; Ferr i C. (2011). Brier curves: A new cost-based visualizationof classifier per formance. In: Proceedings of the 28th International Conference on Machine Learning. \;\n \;\nHerná\;ndez-Orallo J.\, \; Flach P.&n bsp\; and \; Ferri C. (2013). ROC curves in cost space. Machine Learni ng \; 93\, 71-91. \;\n \;\nShao\, H\, Qianqian X.\, Zhiyong Y. \, Peisong W.\, Peifeng G. and Qingming H. \; (2023). Weighted ROC cur ve in cost space: Extending AUC to cost-sensitive learning. \; In: Adv ances of Neural Information Processing Systems (NeurIPS 2023) 36\, 17257-1 7368. \;\n \; LOCATION:Seminar Room 1\, Newton Institute END:VEVENT END:VCALENDAR