When you are asking it a question (like which of these two texts is the best), the output is also just picked by minimising that loss function. There's no guarantee that answering "Text B is better" aligns with text B minimising the loss function.
(And they aren't really minimising loss functions during inference. They sample from a distribution. During training they minimise the loss function of the distribution.)
Point taken, I was possibly overly curt in my response, leading to it being ambiguous, both re when the loss function is being minimised and that it is a stochastic process.
So to OPs question: I guess LLMs do have an "idea" of what is best (conditioned on minimising a loss function during training), however they may not always output that (because stochasticity), which maybe represents a degree of uncertainty in that "idea"?
My point is that when you asked them to 'judge' two texts, they are just going to output the tokens that have the highest probability (or a sample from that distribution).
But depending on their training data, these tokens might say 'correct horse battery staple', and not necessarily 'text A is better' or 'text B is better'.
Even if text A would have been more likely to be produced by the LLM.
When you are asking it a question (like which of these two texts is the best), the output is also just picked by minimising that loss function. There's no guarantee that answering "Text B is better" aligns with text B minimising the loss function.
(And they aren't really minimising loss functions during inference. They sample from a distribution. During training they minimise the loss function of the distribution.)