Can artificial intelligence explain justice? evaluating faithfulness and legal reasoning in AI-generated summaries of Indonesian court decisions
DOI:
https://doi.org/10.67490/ijll.v1i2.918Keywords:
artificial intelligence, court decisionsAbstract
Background: As Indonesian court decisions become increasingly accessible through public digital repositories, artificial intelligence promises to translate dense judicial language into readable summaries, yet this promise raises a central problem: whether accessibility can be achieved without weakening legal meaning. Objective: This study evaluates how AI-generated summaries of Indonesian court decisions preserve source faithfulness, legal reasoning, and public-facing explainability. Method: Using a qualitative-dominant evaluative design, this study compares ten source-linked AI summaries with ten Indonesian court decisions through structured coding of factual alignment, legal-basis accuracy, reasoning preservation, disposition fidelity, source traceability, and public-comprehension risk. Results: The findings show that AI summaries are strongest in preserving visible textual elements, especially case identity, general legal issue, and final disposition. They are less stable when summarising legal bases, procedural thresholds, interpretive qualifications, ratio decidendi, and uncertainty. Implication: This pattern indicates that AI can support legal accessibility but may also flatten judicial reasoning into outcome-centred explanation when source constraints are not preserved. Novelty: This study contributes a multidimensional framework for evaluating legal AI summaries by linking faithfulness, reasoning fidelity, and explainability as inseparable conditions for accountable public legal communication.
Downloads
References
[1] M. Dahl, V. Magesh, M. Suzgun, and D. E. Ho, “Large legal fictions: Profiling legal hallucinations in large language models,” Journal of Legal Analysis, 2024, doi: 10.1093/jla/laae003.
[2] V. Magesh, F. Surani, M. Dahl, M. Suzgun, C. D. Manning, and D. E. Ho, “Hallucination-free? Assessing the reliability of leading AI legal research tools,” arXiv, 2024, doi: 10.48550/arXiv.2405.20362.
[3] F. Ariai and G. Demartini, “Natural language processing for the legal domain: A survey of tasks, datasets, models, and challenges,” ACM Computing Surveys, vol. 58, pp. 1–37, 2024, doi: 10.1145/3777009.
[4] J. Cui, X. Shen, F. Nie, Z. Wang, and J. Wang, “A survey on legal judgment prediction: Datasets, metrics, models and challenges,” IEEE Access, vol. 11, pp. 102050–102071, 2022, doi: 10.1109/ACCESS.2023.3317083.
[5] E. Ash, A. Kesari, S. Naidu, L. Song, and D. Stammbach, “Translating legalese: Enhancing public understanding of court opinions with legal summarizers,” Proceedings of the 2024 Symposium on Computer Science and Law, 2023, doi: 10.1145/3614407.3643700.
[6] E. J. Bauer, D. Stammbach, N. Gu, and E. Ash, “Legal extractive summarization of U.S. court opinions,” arXiv, 2023, doi: 10.48550/arXiv.2305.08428.
[7] P. Bhattacharya, S. Poddar, K. Rudra, K. Ghosh, and S. Ghosh, “Incorporating domain knowledge for extractive summarization of legal case documents,” Proceedings of the Eighteenth International Conference on Artificial Intelligence and Law, 2021, doi: 10.1145/3462757.3466092.
[8] A. Deroy, K. Ghosh, and S. Ghosh, “Applicability of large language models and generative models for legal case judgement summarization,” Artificial Intelligence and Law, vol. 33, pp. 1007–1050, 2024, doi: 10.1007/s10506-024-09411-z.
[9] S. Ali, T. Abuhmed, S. El-Sappagh, K. Muhammad, J. Alonso-Moral, R. Confalonieri, et al., “Explainable artificial intelligence (XAI): What we know and what is left to attain trustworthy artificial intelligence,” Information Fusion, vol. 99, p. 101805, 2023, doi: 10.1016/j.inffus.2023.101805.
[10] J. Collenette, K. Atkinson, and T. J. M. Bench-Capon, “Explainable AI tools for legal reasoning about cases: A study on the European Court of Human Rights,” Artificial Intelligence, vol. 317, p. 103861, 2023, doi: 10.1016/j.artint.2023.103861.
[11] N. Guha, J. Nyarko, D. E. Ho, C. Ré, A. Chilton, A. Narayana, et al., “LegalBench: A collaboratively built benchmark for measuring legal reasoning in large language models,” arXiv, 2023, doi: 10.48550/arXiv.2308.11462.
[12] X. Wang, X. Zhang, V. Hoo, Z. Shao, and X. Zhang, “LegalReasoner: A multi-stage framework for legal judgment prediction via large language models and knowledge integration,” IEEE Access, vol. 12, pp. 166843–166854, 2024, doi: 10.1109/ACCESS.2024.3496666.
[13] E. Q. Nuranti, E. Yulianti, and H. S. Husin, “Predicting the category and the length of punishment in Indonesian courts based on previous court decision documents,” Computers, vol. 11, no. 6, p. 88, 2022, doi: 10.3390/computers11060088.
[14] G. Wicaksono, S. F. A. Asqalani, Y. Azhar, N. Hidayah, and A. Andreawana, “Automatic summarization of court decision documents over narcotic cases using BERT,” JOIV: International Journal on Informatics Visualization, vol. 7, no. 2, 2023, doi: 10.30630/joiv.7.2.1811.
[15] A. Shukla, P. Bhattacharya, S. Poddar, R. Mukherjee, K. Ghosh, P. Goyal, and S. Ghosh, “Legal case document summarization: Extractive and abstractive methods and their evaluation,” arXiv, 2022, doi: 10.48550/arXiv.2210.07544.
[16] Y. Huang, L. Sun, C. Han, and J. Guo, “A high-precision two-stage legal judgment summarization,” Mathematics, vol. 11, no. 6, p. 1320, 2023, doi: 10.3390/math11061320.
[17] W. Gao, S. Yu, Y. Qin, C. Yang, R. Huang, Y. Chen, and C. Lin, “LSDK-LegalSum: Improving legal judgment summarization using logical structure and domain knowledge,” Journal of King Saud University Computer and Information Sciences, vol. 37, 2025, doi: 10.1007/s44443-025-00022-5.
[18] Y. Song, Y. Qin, R. Huang, Y. Chen, and C. Lin, “Legal text summarization via judicial syllogism with large language models,” Journal of King Saud University Computer and Information Sciences, vol. 37, 2025, doi: 10.1007/s44443-025-00113-3.
[19] M. Koniaris, D. Galanis, E. Giannini, and P. Tsanakas, “Evaluation of automatic legal text summarization techniques for Greek case law,” Information, vol. 14, no. 4, p. 250, 2023, doi: 10.3390/info14040250.
[20] G. Moro, N. Piscaglia, L. Ragazzi, and P. Italiani, “Multi-language transfer learning for low-resource legal case summarization,” Artificial Intelligence and Law, 1–29, 2023, doi: 10.1007/s10506-023-09373-8.
[21] S. Liu, J. Cao, Y. Li, R. Yang, and Z. Wen, “Low-resource court judgment summarization for common law systems,” arXiv, 2024, doi: 10.48550/arXiv.2403.04454.
[22] Z. Fei, X. Shen, D. Zhu, F. Zhou, Z. Han, S. Zhang, et al., “LawBench: Benchmarking legal knowledge of large language models,” arXiv, 2023, doi: 10.48550/arXiv.2309.16289.
[23] W. Hwang, K. Cho, H.-Y. Lee, and M. Seo, “A multi-task benchmark for Korean legal language understanding and judgement prediction,” arXiv, 2022, doi: 10.48550/arXiv.2206.05224.
[24] T. Turan and E. Küçüksille, “Summarization, prediction, and analysis of Turkish Constitutional Court decisions with explainable artificial intelligence and a hybrid natural language processing method,” IEEE Access, vol. 13, pp. 59766–59779, 2025, doi: 10.1109/ACCESS.2025.3556725.
[25] R. Kondo, R. Matsuoka, T. Yoshida, K. Yamasawa, and R. Hisano, “Capturing legal reasoning paths from facts to law in court judgments using knowledge graphs,” Proceedings of the 13th Knowledge Capture Conference 2025, 2025, doi: 10.1145/3731443.3771354.
[26] S. Nigam, A. Sharma, D. Khanna, N. Shallum, K. Ghosh, and A. Bhattacharya, “Legal judgment reimagined: PredEx and the rise of intelligent AI interpretation in Indian courts,” arXiv, 2024, doi: 10.48550/arXiv.2406.04136.
[27] S. Nigam, B. D. Patnaik, S. Mishra, N. Shallum, K. Ghosh, and A. Bhattacharya, “TathyaNyaya and FactLegalLlama: Advancing factual judgment prediction and explanation in the Indian legal context,” arXiv, 2025, doi: 10.48550/arXiv.2504.04737.
[28] M. Ibrahim, A. T. Handoyo, and M. S. Anggreainy, “Hybrid deep learning for legal text analysis: Predicting punishment durations in Indonesian court rulings,” arXiv, 2024, doi: 10.48550/arXiv.2410.20104.
[29] E. Q. Nuranti, N. S. Intizhami, E. Yulianti, A. Latief, O. Iyad, and A. Ghozy, “A systematical procedure to extracting legal entities from Indonesian judicial decisions,” MethodsX, vol. 15, 2025, doi: 10.1016/j.mex.2025.103610.
[30] J. Gu, X. Jiang, Z. Shi, H. Tan, X. Zhai, C. Xu, et al., “A survey on LLM-as-a-judge,” arXiv, 2024, doi: 10.48550/arXiv.2411.15594.
[31] H. Li, Q. Dong, J. Chen, H. Su, Y. Zhou, Q. Ai, Z. Ye, and Y. Liu, “LLMs-as-judges: A comprehensive survey on LLM-based evaluation methods,” arXiv, abs/2412.05579, 2024.
[32] D. S. U. De La Osa and N. Remolina, “Artificial intelligence at the bench: Legal and ethical challenges of informing—or misinforming—judicial decision-making through generative AI,” Data & Policy, vol. 6, 2024, doi: 10.1017/dap.2024.53.
[33] M. F. Ramadhan, “Plain language in legal drafting: Challenges and opportunities in Indonesian regulatory reform,” Indonesian Journal of Linguistics and Legal Discourse, vol. 1, no. 1, pp. 22–30, 2026.
[34] W. B. M. Setiyawan, “Forensic linguistic evidence in Indonesian courtrooms: A study of language use in criminal verdicts,” Indonesian Journal of Linguistics and Legal Discourse, vol. 1, no. 1, pp. 1–12, 2026.
[35] Sadani, “Linguistic construction of justice: A pragmatic analysis of judges’ reasoning in Indonesian constitutional court decisions,” Indonesian Journal of Linguistics and Legal Discourse, vol. 1, no. 1, pp. 31–40, 2026.
[36] D. Setiaji, “Legal language and power: A critical discourse analysis of anti-corruption laws in Indonesia,” Indonesian Journal of Linguistics and Legal Discourse, vol. 1, no. 1, pp. 13–21, 2026.
[37] M. Ansori, “Speech act strategies in police interrogation transcripts: Evidence from bahasa Indonesia investigative records,” Indonesian Journal of Linguistics and Legal Discourse, vol. 1, no. 1, pp. 51–60, 2026.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Ahmad Zairudin (Author)
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Creative Commons Attribution 4.0 International License