Authority and Responsibility Allocation in Human–AI Collaborative Decision-Making: Governance Mechanisms for Enterprises
Article Information
Abstract
Artificial intelligence is moving from analytical support toward active participation in enterprise decisions, creating an organization-design problem: firms must decide which rights may be delegated to AI and how responsibility should follow the actors who can prevent, challenge, or remedy failure. Existing work explains automation, augmentation, delegation, human oversight, and responsible AI governance, but does not reveal how specific transfers of decision authority create responsibility gaps inside a focal enterprise decision. This conceptual paper develops a contingency governance framework through a transparent theory-synthesis procedure. A purposive corpus of 44 peer-reviewed studies, standards, and regulatory sources was assembled through anchor studies, targeted keyword searches, and citation chaining. First-order authority and responsibility terms were coded, compared, and abstracted until two successive search iterations produced no new categories. The resulting framework distinguishes seven decision rights—information access, recommendation, selection, approval, veto, execution, and escalation—and five responsibility domains—system design, decision process, outcome stewardship, oversight, and remediation. Its central mechanism is rights-control-responsibility alignment: delegating a right shifts effective control and evidence access, while governance fails when the responsible actor lacks the competence, authority, or information to intervene. Decision exposure and AI autonomy determine four governance archetypes, while AI reliability conditions the permissible scope of selection and execution rights. Eight empirically testable propositions specify mechanisms, moderators, competing explanations, and falsification conditions. Two worked applications show how the architecture produces more precise governance than a generic human-in-the-loop requirement. The paper contributes a decision-level theory of enterprise AI governance and provides managers with an auditable method for allocating rights, responsibilities, evidence, and lifecycle controls.
Graphical Abstract
Keywords
Data Availability Statement
Funding
Conflicts of Interest
AI Use Statement
Ethical Approval and Consent to Participate
References
- Raisch, S., & Krakowski, S. (2021). Artificial intelligence and management: The automation-augmentation paradox. Academy of Management Review, 46(1), 192-210.
[CrossRef] [Google Scholar] - Berente, N., Gu, B., Recker, J., & Santhanam, R. (2021). Managing artificial intelligence. MIS Quarterly, 45(3), 1433-1450.
[CrossRef] [Google Scholar] - Feuerriegel, S., Hartmann, J., Janiesch, C., & Zschech, P. (2024). Generative AI. Business & Information Systems Engineering, 66(1), 111-126.
[CrossRef] [Google Scholar] - Enholm, I. M., Papagiannidis, E., Mikalef, P., & Krogstie, J. (2022). Artificial intelligence and business value: A literature review. Information systems frontiers, 24(5), 1709-1734.
[CrossRef] [Google Scholar] - Vaccaro, M., Almaatouq, A., & Malone, T. (2024). When combinations of humans and AI are useful: A systematic review and meta-analysis. Nature Human Behaviour, 8(12), 2293-2303.
[CrossRef] [Google Scholar] - Puranam, P. (2021). Human–AI collaborative decision-making as an organization design problem. Journal of Organization Design, 10(2), 75-80.
[CrossRef] [Google Scholar] - Baird, A., & Maruping, L. M. (2021). The next generation of research on IS use: A theoretical framework of delegation to and from agentic IS artifacts. MIS Quarterly, 45(1), 315-341.
[CrossRef] [Google Scholar] - Papagiannidis, E., Mikalef, P., & Conboy, K. (2025). Responsible artificial intelligence governance: A review and research framework. The Journal of Strategic Information Systems, 34(2), 101885.
[CrossRef] [Google Scholar] - Marabelli, M., Newell, S., & Handunge, V. (2021). The lifecycle of algorithmic decision-making systems: Organizational choices and ethical challenges. The Journal of Strategic Information Systems, 30(3), 101683.
[CrossRef] [Google Scholar] - Green, B. (2022). The flaws of policies requiring human oversight of government algorithms. Computer Law & Security Review, 45, 105681.
[CrossRef] [Google Scholar] - Te'eni, D., Yahav, I., & Schwartz, D. (2026). What it takes to control AI by design: human learning. Ai & Society, 41(1), 237-250.
[CrossRef] [Google Scholar] - Tabassi, E. (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0) (NIST AI 100-1). National Institute of Standards and Technology.
[CrossRef] [Google Scholar] - International Organization for Standardization. (2023). ISO/IEC 42001:2023 Information technology - Artificial intelligence - Management system. ISO. https://www.iso.org/standard/42001
[Google Scholar] - Jaakkola, E. (2020). Designing conceptual articles: four approaches. AMS review, 10(1), 18-26.
[CrossRef] [Google Scholar] - Konsynski, B. R., Kathuria, A., & Karhade, P. P. (2024). Cognitive reapportionment and the art of letting go: A theoretical framework for the allocation of decision rights. Journal of Management Information Systems, 41(2), 328-340.
[CrossRef] [Google Scholar] - Santoni de Sio, F., & Mecacci, G. (2021). Four responsibility gaps with artificial intelligence: Why they matter and how to address them. Philosophy & Technology, 34(4), 1057-1084.
[CrossRef] [Google Scholar] - Stark, D., & Vanden Broeck, P. (2024). Principles of algorithmic management. Organization Theory, 5(2), 26317877241257213.
[CrossRef] [Google Scholar] - Trunk, A., Birkel, H., & Hartmann, E. (2020). On the current state of combining human and artificial intelligence for strategic organizational decision making. Business Research, 13(3), 875-919.
[CrossRef] [Google Scholar] - Mäntymäki, M., Minkkinen, M., Birkstedt, T., & Viljanen, M. (2022). Defining organizational AI governance. AI and Ethics, 2(4), 603-609.
[CrossRef] [Google Scholar] - Papagiannidis, E., Enholm, I. M., Dremel, C., Mikalef, P., & Krogstie, J. (2023). Toward AI governance: Identifying best practices and potential barriers and outcomes. Information Systems Frontiers, 25(1), 123-141.
[CrossRef] [Google Scholar] - Birkstedt, T., Minkkinen, M., Tandon, A., & Mäntymäki, M. (2023). AI governance: themes, knowledge gaps and future agendas. Internet Research, 33(7), 133-167.
[CrossRef] [Google Scholar] - Batool, A., Zowghi, D., & Bano, M. (2025). AI governance: a systematic literature review. AI and Ethics, 5(3), 3265-3279.
[CrossRef] [Google Scholar] - Sadek, M., Kallina, E., Bohné, T., Mougenot, C., Calvo, R. A., & Cave, S. (2025). Challenges of responsible AI in practice: scoping review and recommended actions. AI & society, 40(1), 199-215.
[CrossRef] [Google Scholar] - Minkkinen, M., Laine, J., & Mäntymäki, M. (2022). Continuous auditing of artificial intelligence: a conceptualization and assessment of tools and frameworks. Digital Society, 1(3), 21.
[CrossRef] [Google Scholar] - Mökander, J., Morley, J., Taddeo, M., & Floridi, L. (2021). Ethics-based auditing of automated decision-making systems: Nature, scope, and limitations. Science and Engineering Ethics, 27(4), 44.
[CrossRef] [Google Scholar] - Schiff, D. S., Kelley, S., & Camacho Ibáñez, J. (2024). The emergence of artificial intelligence ethics auditing. Big Data & Society, 11(4), 20539517241299732.
[CrossRef] [Google Scholar] - Hemmer, P., Schemmer, M., Kühl, N., Vössing, M., & Satzger, G. (2025). Complementarity in human-AI collaboration: Concept, sources, and evidence. European Journal of Information Systems, 34(6), 979-1002.
[CrossRef] [Google Scholar] - Freisinger, E., & Schneider, S. (2025). Decoding decision delegation to artificial intelligence: A mixed-methods study on the preferences of decision-makers and decision-affected in surrogate decision contexts. European Management Journal, 43(6), 958-969.
[CrossRef] [Google Scholar] - European Parliament and Council of the European Union. (2024). Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence and amending Regulations (Artificial Intelligence Act). Official Journal of the European Union, L 2024/1689. https://eur-lex.europa.eu/eli/reg/2024/1689/oj/eng
[Google Scholar] - Schemmer, M., Kuehl, N., Benz, C., Bartos, A., & Satzger, G. (2023, March). Appropriate reliance on AI advice: Conceptualization and the effect of explanations. In Proceedings of the 28th International Conference on Intelligent User Interfaces (pp. 410-422).
[CrossRef] [Google Scholar] - Buçinca, Z., Malaya, M. B., & Gajos, K. Z. (2021). To trust or to think: cognitive forcing functions can reduce overreliance on AI in AI-assisted decision-making. Proceedings of the ACM on Human-computer Interaction, 5(CSCW1), 1-21.
[CrossRef] [Google Scholar] - Novelli, C., Taddeo, M., & Floridi, L. (2024). Accountability in artificial intelligence: what it is and how it works. AI & Society, 39(4), 1871-1882.
[CrossRef] [Google Scholar] - Minkkinen, M., Zimmer, M. P., & Mäntymäki, M. (2023). Co-shaping an ecosystem for responsible AI: Five types of expectation work in response to a technological frame. Information Systems Frontiers, 25(1), 103-121.
[CrossRef] [Google Scholar] - Laux, J., Wachter, S., & Mittelstadt, B. (2024). Trustworthy artificial intelligence and the European Union AI act: On the conflation of trustworthiness and acceptability of risk. Regulation & Governance, 18(1), 3-32.
[CrossRef] [Google Scholar] - Fügener, A., Grahl, J., Gupta, A., & Ketter, W. (2022). Cognitive challenges in human–artificial intelligence collaboration: Investigating the path toward productive delegation. Information systems research, 33(2), 678-696.
[CrossRef] [Google Scholar] - Glikson, E., & Woolley, A. W. (2020). Human trust in artificial intelligence: Review of empirical research. Academy of management annals, 14(2), 627-660.
[CrossRef] [Google Scholar] - Alon-Barkat, S., & Busuioc, M. (2023). Human–AI interactions in public sector decision making:"automation bias" and "selective adherence" to algorithmic advice. Journal of Public Administration Research and Theory, 33(1), 153-169.
[CrossRef] [Google Scholar] - Langer, M., Oster, D., Speith, T., Hermanns, H., Kästner, L., Schmidt, E., ... & Baum, K. (2021). What do we want from Explainable Artificial Intelligence (XAI)?–A stakeholder perspective on XAI and a conceptual model guiding interdisciplinary XAI research. Artificial intelligence, 296, 103473.
[CrossRef] [Google Scholar] - Ehsan, U., Liao, Q. V., Muller, M., Riedl, M. O., & Weisz, J. D. (2021). Expanding explainability: Towards social transparency in AI systems. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (pp. 1-19). ACM.
[CrossRef] [Google Scholar] - Choudhary, V., Marchetti, A., Shrestha, Y. R., & Puranam, P. (2025). Human-AI ensembles: When can they work? Journal of Management, 51(2), 536-569.
[CrossRef] [Google Scholar] - Heyder, T., Passlack, N., & Posegga, O. (2023). Ethical management of human-AI interaction: Theory development review. The Journal of Strategic Information Systems, 32(3), 101772.
[CrossRef] [Google Scholar] - Ayling, J., & Chapman, A. (2022). Putting AI ethics to work: are the tools fit for purpose?. AI and Ethics, 2(3), 405-429.
[CrossRef] [Google Scholar] - Ibáñez, J. C., & Olmeda, M. V. (2022). Operationalising AI ethics: how are companies bridging the gap between practice and principles? An exploratory study. Ai & Society, 37(4), 1663-1687.
[CrossRef] [Google Scholar] - Morley, J., Kinsey, L., Elhalal, A., Garcia, F., Ziosi, M., & Floridi, L. (2023). Operationalising AI ethics: Barriers, enablers and next steps. AI & Society, 38(1), 411-423.
[CrossRef] [Google Scholar]
Cite This Article
TY - JOUR AU - Sun, Fang PY - 2026 DA - 2026/08/09 TI - Authority and Responsibility Allocation in Human–AI Collaborative Decision-Making: Governance Mechanisms for Enterprises JO - ICCK Transactions on Systems Safety and Reliability T2 - ICCK Transactions on Systems Safety and Reliability JF - ICCK Transactions on Systems Safety and Reliability VL - 2 IS - 3 SP - 192 EP - 206 DO - 10.62762/TSSR.2026.930520 UR - https://www.icck.org/article/abs/TSSR.2026.930520 KW - human-AI collaboration KW - enterprise decision-making KW - decision rights KW - responsibility allocation KW - AI governance KW - meaningful human control AB - Artificial intelligence is moving from analytical support toward active participation in enterprise decisions, creating an organization-design problem: firms must decide which rights may be delegated to AI and how responsibility should follow the actors who can prevent, challenge, or remedy failure. Existing work explains automation, augmentation, delegation, human oversight, and responsible AI governance, but does not reveal how specific transfers of decision authority create responsibility gaps inside a focal enterprise decision. This conceptual paper develops a contingency governance framework through a transparent theory-synthesis procedure. A purposive corpus of 44 peer-reviewed studies, standards, and regulatory sources was assembled through anchor studies, targeted keyword searches, and citation chaining. First-order authority and responsibility terms were coded, compared, and abstracted until two successive search iterations produced no new categories. The resulting framework distinguishes seven decision rights—information access, recommendation, selection, approval, veto, execution, and escalation—and five responsibility domains—system design, decision process, outcome stewardship, oversight, and remediation. Its central mechanism is rights-control-responsibility alignment: delegating a right shifts effective control and evidence access, while governance fails when the responsible actor lacks the competence, authority, or information to intervene. Decision exposure and AI autonomy determine four governance archetypes, while AI reliability conditions the permissible scope of selection and execution rights. Eight empirically testable propositions specify mechanisms, moderators, competing explanations, and falsification conditions. Two worked applications show how the architecture produces more precise governance than a generic human-in-the-loop requirement. The paper contributes a decision-level theory of enterprise AI governance and provides managers with an auditable method for allocating rights, responsibilities, evidence, and lifecycle controls. SN - 3069-1087 PB - Institute of Central Computation and Knowledge LA - English ER -
@article{Sun2026Authority,
author = {Fang Sun},
title = {Authority and Responsibility Allocation in Human–AI Collaborative Decision-Making: Governance Mechanisms for Enterprises},
journal = {ICCK Transactions on Systems Safety and Reliability},
year = {2026},
volume = {2},
number = {3},
pages = {192-206},
doi = {10.62762/TSSR.2026.930520},
url = {https://www.icck.org/article/abs/TSSR.2026.930520},
abstract = {Artificial intelligence is moving from analytical support toward active participation in enterprise decisions, creating an organization-design problem: firms must decide which rights may be delegated to AI and how responsibility should follow the actors who can prevent, challenge, or remedy failure. Existing work explains automation, augmentation, delegation, human oversight, and responsible AI governance, but does not reveal how specific transfers of decision authority create responsibility gaps inside a focal enterprise decision. This conceptual paper develops a contingency governance framework through a transparent theory-synthesis procedure. A purposive corpus of 44 peer-reviewed studies, standards, and regulatory sources was assembled through anchor studies, targeted keyword searches, and citation chaining. First-order authority and responsibility terms were coded, compared, and abstracted until two successive search iterations produced no new categories. The resulting framework distinguishes seven decision rights—information access, recommendation, selection, approval, veto, execution, and escalation—and five responsibility domains—system design, decision process, outcome stewardship, oversight, and remediation. Its central mechanism is rights-control-responsibility alignment: delegating a right shifts effective control and evidence access, while governance fails when the responsible actor lacks the competence, authority, or information to intervene. Decision exposure and AI autonomy determine four governance archetypes, while AI reliability conditions the permissible scope of selection and execution rights. Eight empirically testable propositions specify mechanisms, moderators, competing explanations, and falsification conditions. Two worked applications show how the architecture produces more precise governance than a generic human-in-the-loop requirement. The paper contributes a decision-level theory of enterprise AI governance and provides managers with an auditable method for allocating rights, responsibilities, evidence, and lifecycle controls.},
keywords = {human-AI collaboration, enterprise decision-making, decision rights, responsibility allocation, AI governance, meaningful human control},
issn = {3069-1087},
publisher = {Institute of Central Computation and Knowledge}
}
Article Metrics
Publisher's Note
ICCK stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and Permissions
Portico