Allow assistant to give a confidence level along with its answer

[minarchist]

This is a feature recommendation. Discussion here:
https://bdtechtalks.com/2022/09/05/llm-uncertainty-verbalized-probability/
Academic paper here:
https://arxiv.org/abs/2205.14334

[gante]

Hey there 👋

Assessing the model confidence for text generation is a very interesting topic, and its application would certainly be of value for Open-Assistant. As an external observer to the project (that is willing to contribute), I'm dropping my unsolicited 2 cents here 🤗

The paper linked above shows good results in its evaluations, but I'm highly skeptical about it. In a practical sense, the authors raise my concerns in section 4.1: However, while our training and evaluation sets differed significantly in the label distribution, the content and format of questions did not shift much.. In other words, the evaluations stay in the easy realm, with the model being trained and evaluated on simple mathematical prompts. More challenging logical questions and subjective topics would surely be a tough nut to crack with this strategy.

We can even think about the problem on a more abstract level: how can an ML model ever learn to tell us what it doesn't know? Being a numerical black box that is randomly initialized, there is always a chance to return high confidence on the most nonsense answers. From what I've read so far, the most established way to answer this question resides in ensembles, MC Dropout being the next best thing ([1][2]). The former requires multiple models trained from scratch, the latter requires models with dropout, and both require multiple forward passes to be able to compute the confidence, which is very expensive for large models. On the upside, we gain the ability to distinguish epistemic from aleatoric uncertainty or, in other words, "is the confidence low because the topic is ambiguous or because the model doesn't know about it?".

All this to say -- I believe the suggestion at the top of this thread is worthwhile to pursue, due to its low inference costs and interesting research potential, but I also believe its practical usefulness will be limited :)

[mrcabbage972]

Another paper that might be relevant: Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation. Unlike the paper mentioned above, this approach is unsupervised so distribution shift is one thing not to worry about. Also, the evaluation datasets are more diverse (TriviaQA and CoQA).

[JeniaJitsev]

I think this is indeed a very important feature, and should be highly prioritized.

For instance, chatGPT is not able to put any proper evaluation of uncertainty about right or wrong of its responses, and is also not able to correct itself about how confident it should be even if given explicit hints on quality of provided answers (for one striking simple example, see here: https://twitter.com/JJitsev/status/1604546168452796418)

To make Open Assistant more reliable and trustworthy, the uncertainty / confidence representation and signaling, including ability to correctly revise responses if provided with hints, will be crucial and most probably cannot rely on human feedback data only.

[davidak]

chatGPT is not able to put any proper evaluation of uncertainty about right or wrong of its responses

I had the situation that i pointed out a mistake ChatGPT made and it recognized the mistake and was able to explain it.

... but the opposite is true

https://chaos.social/@davidak/110033524401977738

It is interesting that it still made the mistake while it should have known that it was wrong before.