Conformal Prediction: Theory and Application

Once that Conformal Prediction is applied to a model, it is of interest to evaluate the results. One thing, very important to look at is the adaptivity of the model: that is, we want the procedure to return larger prediction sets when the input is harder and smaller sets when the input is easier.

To this end, the first thing we can do is to plot the set sizes through some histograms: in this way, we can understand whether the model is correctly adapting the prediction set dimension to each input or not.
However, it is better to have a formal measure for evaluating adaptivity: to do so, we will introduce two metrics that will help us judge the quality of a Conformal Prediction application. However, before we introduce them, we need to formalize the concept of adaptivity; we do so by asking for the conditional coverage property [8]:
[…]

That is a stronger property than the marginal coverage property in (1). The difference between marginal and conditional coverage is subtle but very important; to explain this in the easiest way possible, we will use a very clear example from [3]:
"imagine there are two groups A and B with frequencies 90% and 10%; the prediction sets always cover Y among people in group A and never cover Y when the person comes from group B. Then the prediction sets have 90% coverage, but not conditional coverage, in fact, conditional coverage would imply that the prediction sets cover Y at least 90% of the time in each group".

Now we can finally introduce the two metrics to evaluate conditional coverage (adaptivity)
[…]