Nano-jev: Calibrated Cross-Encoder Micro-Decisions for Retrieval-Augmented Generation with Confidence Cascading
Article Information
Abstract
Retrieval-Augmented Generation (RAG) pipelines repeatedly make small judgments, such as whether a passage is relevant, whether the retrieved context is sufficient and whether an answer is grounded. These are usually delegated to the generating large language model (LLM), at notable latency and cost, or to score thresholds without a probabilistic interpretation. We study nano-jev, a 33.4\,M-parameter cross-encoder that treats relevance (3-way), sufficiency and groundedness (binary) as typed, closed-set classification problems and returns temperature-calibrated probabilities. Each (question, option, context) triple is scored by a single-logit head, which makes predictions invariant to option order. On held-out halves of the HotpotQA, SQuAD~2.0 and MultiNLI validation sets, nano-jev reaches mean accuracies of 0.812 (relevance), 0.843 (sufficiency) and 0.841 (groundedness) over five seeds, comparable to a 1-pass classifier trained on the same data, which is 2.5$\times$ faster. Dedicated checkers such as MiniCheck remain stronger on groundedness (0.866). nano-jev runs 55-93$\times$ faster than a prompted 4\,B-parameter LLM. On unseen datasets (MuSiQue, VitaminC), accuracy drops by 15-20 points and ECE rises from about 0.03 to about 0.17, so target-domain re-calibration is advisable. A confidence cascade that escalates uncertain decisions to the LLM improves in-distribution groundedness (0.868 vs. 0.841, $p=0.02$) with about a quarter of decisions escalated, but trades accuracy for latency out of distribution. In a small end-to-end test, nano-jev relevance filtering improves HotpotQA answer F1 by 2.1 points at about 39ms per question. Code and weights are released as an open-source package.
Graphical Abstract
Keywords
Data Availability Statement
Funding
Conflicts of Interest
AI Use Statement
Ethical Approval and Consent to Participate
References
- Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., ... & Kiela, D. (2020). Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems, 33, 9459-9474.
[Google Scholar] - Kadavath, S., Conerly, T., Askell, A., Henighan, T., Drain, D., Perez, E., ... & Kaplan, J. (2022). Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221.
[CrossRef] [Google Scholar] - Kuhn, L., Gal, Y., & Farquhar, S. (2023). Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. arXiv preprint arXiv:2302.09664.
[CrossRef] [Google Scholar] - Zha, Y., Yang, Y., Li, R., & Hu, Z. (2023, July). AlignScore: Evaluating factual consistency with a unified alignment function. In Proceedings of the 61st annual meeting of the association for computational linguistics (volume 1: Long papers) (pp. 11328-11348).
[CrossRef] [Google Scholar] - Tang, L., Laban, P., & Durrett, G. (2024). MiniCheck: Efficient fact-checking of LLMs on grounding documents. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (pp. 8818-8847).
[CrossRef] [Google Scholar] - Saad-Falcon, J., Khattab, O., Potts, C., & Zaharia, M. (2024). ARES: An automated evaluation framework for retrieval-augmented generation systems. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics (pp. 338-354).
[CrossRef] [Google Scholar] - Es, S., James, J., Espinosa-Anke, L., & Schockaert, S. (2024). RAGAs: Automated evaluation of retrieval augmented generation. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations (pp. 150-158).
[CrossRef] [Google Scholar] - Guu, K., Lee, K., Tung, Z., Pasupat, P., & Chang, M. (2020, November). Retrieval augmented language model pre-training. In International conference on machine learning (pp. 3929-3938). PMLR. https://proceedings.mlr.press/v119/guu20a.html
[Google Scholar] - Asai, A., Wu, Z., Wang, Y., Sil, A., & Hajishirzi, H. (2024, May). Self-rag: Learning to retrieve, generate, and critique through self-reflection. In International conference on learning representations (Vol. 2024, pp. 9112-9141).
[Google Scholar] - Yan, S. Q., Gu, J. C., Zhu, Y., & Ling, Z. H. (2024). Corrective retrieval augmented generation. arXiv preprint arXiv:2401.15884.
[CrossRef] [Google Scholar] - Laban, P., Schnabel, T., Bennett, P. N., & Hearst, M. A. (2022). SummaC: Re-visiting NLI-based models for inconsistency detection in summarization. Transactions of the Association for Computational Linguistics, 10, 163-177.
[CrossRef] [Google Scholar] - Honovich, O., Aharoni, R., Herzig, J., Taitelbaum, H., Kukliansy, D., Cohen, V., ... & Matias, Y. (2022, July). TRUE: Re-evaluating factual consistency evaluation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (pp. 3905-3920).
[CrossRef] [Google Scholar] - Niu, C., Wu, Y., Zhu, J., Xu, S., Shum, K., Zhong, R., ... & Zhang, T. (2024). RAGTruth: A hallucination corpus for developing trustworthy retrieval-augmented language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (pp. 10862-10878).
[CrossRef] [Google Scholar] - Nogueira, R., Yang, W., Cho, K., & Lin, J. (2019). Multi-stage document ranking with BERT. arXiv preprint arXiv:1910.14424.
[CrossRef] [Google Scholar] - Reimers, N., & Gurevych, I. (2019). Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (pp. 3982-3992).
[CrossRef] [Google Scholar] - Chen, J., Xiao, S., Zhang, P., Luo, K., Lian, D., & Liu, Z. (2024). M3-Embedding: Multi-linguality, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. In Findings of the Association for Computational Linguistics: ACL 2024 (pp. 2318-2335).
[CrossRef] [Google Scholar] - Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (pp. 4171-4186).
[CrossRef] [Google Scholar] - Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., ... & Stoyanov, V. (2019). RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692.
[CrossRef] [Google Scholar] - He, P., Gao, J., & Chen, W. (2021). Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing. arXiv preprint arXiv:2111.09543.
[CrossRef] [Google Scholar] - Yin, W., Hay, J., & Roth, D. (2019). Benchmarking zero-shot text classification: Datasets, evaluation and entailment approach. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (pp. 3914-3923).
[CrossRef] [Google Scholar] - Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017). On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning (pp. 1321-1330). https://proceedings.mlr.press/v70/guo17a.html
[Google Scholar] - Platt, J. (1999). Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers, 10(3), 61-74. https://www.researchgate.net/publication/2594015
[Google Scholar] - Chen, L., Zaharia, M., & Zou, J. (2023). Frugalgpt: How to use large language models while reducing cost and improving performance. arXiv preprint arXiv:2305.05176.
[CrossRef] [Google Scholar] - Ding, D., Mallick, A., Wang, C., Sim, R., Mukherjee, S., Rühle, V., ... & Awadallah, A. H. (2024, May). Hybrid llm: Cost-efficient and quality-aware query routing. In International Conference on Learning Representations (Vol. 2024, pp. 41348-41366).
[Google Scholar] - Leviathan, Y., Kalman, M., & Matias, Y. (2023, July). Fast inference from transformers via speculative decoding. In International conference on machine learning (pp. 19274-19286). PMLR. https://proceedings.mlr.press/v202/leviathan23a.html
[Google Scholar] - Almeida, D. (2026, September 15). Introducing System One models & Jev. TypeSafe AI Blog. Retrieved from https://typesafe.ai/blog/introducing-system-one-models-and-jev
[Google Scholar] - Wang, W., Wei, F., Dong, L., Bao, H., Yang, N., & Zhou, M. (2020). MiniLM: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. In Advances in Neural Information Processing Systems (Vol. 33, pp. 5776-5788).
[Google Scholar] - Liu, D. C., & Nocedal, J. (1989). On the limited memory BFGS method for large scale optimization. Mathematical Programming, 45(1-3), 503-528.
[CrossRef] [Google Scholar] - Naeini, M. P., Cooper, G., & Hauskrecht, M. (2015). Obtaining well calibrated probabilities using Bayesian binning. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 29, No. 1, pp. 2901-2907).
[CrossRef] [Google Scholar] - Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W., Salakhutdinov, R., & Manning, C. D. (2018). HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 conference on empirical methods in natural language processing (pp. 2369-2380).
[CrossRef] [Google Scholar] - Rajpurkar, P., Jia, R., & Liang, P. (2018). Know what you don't know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (pp. 784-789).
[CrossRef] [Google Scholar] - Williams, A., Nangia, N., & Bowman, S. R. (2018, June). A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long papers) (pp. 1112-1122).
[CrossRef] [Google Scholar] - Trivedi, H., Balasubramanian, N., Khot, T., & Sabharwal, A. (2022). MuSiQue: Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10, 539-554.
[CrossRef] [Google Scholar] - Schuster, T., Fisch, A., & Barzilay, R. (2021). Get your vitamin C! Robust fact verification with contrastive evidence. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (pp. 624-643).
[CrossRef] [Google Scholar] - Zhang, X., Zhao, J., & LeCun, Y. (2015). Character-level convolutional networks for text classification. In Advances in Neural Information Processing Systems (Vol. 28, pp. 649-657).
[Google Scholar] - Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A. Y., & Potts, C. (2013). Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing (pp. 1631-1642). https://aclanthology.org/D13-1170/
[Google Scholar] - Thorne, J., Vlachos, A., Christodoulopoulos, C., & Mittal, A. (2018, June). FEVER: a large-scale dataset for fact extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) (pp. 809-819).
[CrossRef] [Google Scholar] - Nie, Y., Williams, A., Dinan, E., Bansal, M., Weston, J., & Kiela, D. (2020). Adversarial NLI: A new benchmark for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (pp. 4885-4901).
[CrossRef] [Google Scholar] - Qwen Team. (2025). Qwen3 technical report. arXiv preprint arXiv:2505.09388.
[CrossRef] [Google Scholar] - Loshchilov, I., & Hutter, F. (2019). Decoupled weight decay regularization. arXiv preprint arXiv:2505.09388.
[CrossRef] [Google Scholar] - Efron, B., & Tibshirani, R. J. (1994). An Introduction to the Bootstrap. Chapman & Hall/CRC.
[CrossRef] [Google Scholar] - McNemar, Q. (1947). Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika, 12(2), 153-157.
[CrossRef] [Google Scholar] - Ovadia, Y., Fertig, E., Ren, J., Nado, Z., Sculley, D., Nowozin, S., ... & Snoek, J. (2019). Can you trust your model's uncertainty? Evaluating predictive uncertainty under dataset shift. In Advances in Neural Information Processing Systems (Vol. 32, pp. 13991-14002).
[Google Scholar] - Park, S., Bastani, O., Weimer, J., & Lee, I. (2020, June). Calibrated prediction with covariate shift via unsupervised domain adaptation. In International Conference on Artificial Intelligence and Statistics (pp. 3219-3229). PMLR. https://proceedings.mlr.press/v108/park20b.html
[Google Scholar]
Cite This Article
TY - JOUR AU - Divakar, Subham AU - Priyadarshini, Rojalina PY - 2026 DA - 2026/11/16 TI - Nano-jev: Calibrated Cross-Encoder Micro-Decisions for Retrieval-Augmented Generation with Confidence Cascading JO - ICCK Transactions on Intelligent Systematics T2 - ICCK Transactions on Intelligent Systematics JF - ICCK Transactions on Intelligent Systematics VL - 3 IS - 4 SP - 245 EP - DO - 10.62762/TIS.2026.197904 UR - https://www.icck.org/article/abs/tis.2026.197904 KW - retrieval-augmented generation KW - cross-encoder KW - decision model KW - calibration KW - confidence cascade KW - natural language inference KW - question answering AB - Retrieval-Augmented Generation (RAG) pipelines repeatedly make small judgments, such as whether a passage is relevant, whether the retrieved context is sufficient and whether an answer is grounded. These are usually delegated to the generating large language model (LLM), at notable latency and cost, or to score thresholds without a probabilistic interpretation. We study nano-jev, a 33.4\,M-parameter cross-encoder that treats relevance (3-way), sufficiency and groundedness (binary) as typed, closed-set classification problems and returns temperature-calibrated probabilities. Each (question, option, context) triple is scored by a single-logit head, which makes predictions invariant to option order. On held-out halves of the HotpotQA, SQuAD~2.0 and MultiNLI validation sets, nano-jev reaches mean accuracies of 0.812 (relevance), 0.843 (sufficiency) and 0.841 (groundedness) over five seeds, comparable to a 1-pass classifier trained on the same data, which is 2.5$\times$ faster. Dedicated checkers such as MiniCheck remain stronger on groundedness (0.866). nano-jev runs 55-93$\times$ faster than a prompted 4\,B-parameter LLM. On unseen datasets (MuSiQue, VitaminC), accuracy drops by 15-20 points and ECE rises from about 0.03 to about 0.17, so target-domain re-calibration is advisable. A confidence cascade that escalates uncertain decisions to the LLM improves in-distribution groundedness (0.868 vs. 0.841, $p=0.02$) with about a quarter of decisions escalated, but trades accuracy for latency out of distribution. In a small end-to-end test, nano-jev relevance filtering improves HotpotQA answer F1 by 2.1 points at about 39ms per question. Code and weights are released as an open-source package. SN - 3068-5079 PB - Institute of Central Computation and Knowledge LA - English ER -
@article{Divakar2026Nanojev,
author = {Subham Divakar and Rojalina Priyadarshini},
title = {Nano-jev: Calibrated Cross-Encoder Micro-Decisions for Retrieval-Augmented Generation with Confidence Cascading},
journal = {ICCK Transactions on Intelligent Systematics},
year = {2026},
volume = {3},
number = {4},
pages = {245-},
doi = {10.62762/TIS.2026.197904},
url = {https://www.icck.org/article/abs/tis.2026.197904},
abstract = {Retrieval-Augmented Generation (RAG) pipelines repeatedly make small judgments, such as whether a passage is relevant, whether the retrieved context is sufficient and whether an answer is grounded. These are usually delegated to the generating large language model (LLM), at notable latency and cost, or to score thresholds without a probabilistic interpretation. We study nano-jev, a 33.4\,M-parameter cross-encoder that treats relevance (3-way), sufficiency and groundedness (binary) as typed, closed-set classification problems and returns temperature-calibrated probabilities. Each (question, option, context) triple is scored by a single-logit head, which makes predictions invariant to option order. On held-out halves of the HotpotQA, SQuAD~2.0 and MultiNLI validation sets, nano-jev reaches mean accuracies of 0.812 (relevance), 0.843 (sufficiency) and 0.841 (groundedness) over five seeds, comparable to a 1-pass classifier trained on the same data, which is 2.5\$\times\$ faster. Dedicated checkers such as MiniCheck remain stronger on groundedness (0.866). nano-jev runs 55-93\$\times\$ faster than a prompted 4\,B-parameter LLM. On unseen datasets (MuSiQue, VitaminC), accuracy drops by 15-20 points and ECE rises from about 0.03 to about 0.17, so target-domain re-calibration is advisable. A confidence cascade that escalates uncertain decisions to the LLM improves in-distribution groundedness (0.868 vs. 0.841, \$p=0.02\$) with about a quarter of decisions escalated, but trades accuracy for latency out of distribution. In a small end-to-end test, nano-jev relevance filtering improves HotpotQA answer F1 by 2.1 points at about 39ms per question. Code and weights are released as an open-source package.},
keywords = {retrieval-augmented generation, cross-encoder, decision model, calibration, confidence cascade, natural language inference, question answering},
issn = {3068-5079},
publisher = {Institute of Central Computation and Knowledge}
}
Article Metrics
Publisher's Note
ICCK stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and Permissions
Portico