ICCK Transactions on Applied Intelligence and Cybernetics Home About Editors Current Issue
Comparison Based on SLR and Empirical Data: Factors Affecting Sustainability Aspects in Software Crowdsourcing
Research Article  ·  Published: 16 April 2026
Issue cover
ICCK Transactions on Applied Intelligence and Cybernetics
Volume 1, Issue 1, 2026: 36-52
Research Article Open Access

Comparison Based on SLR and Empirical Data: Factors Affecting Sustainability Aspects in Software Crowdsourcing

by
Waqas Haider Waqas Haider 1 * ORCID
Adnan Khan Adnan Khan 1 ORCID
Samreen Ihsan Samreen Ihsan 2 ORCID
Ihsan Adil Ihsan Adil 2
1 Software Engineering Research Group (SERG-UOM), Department of Computer Science and IT, University of Malakand, Chakdara 18800, KPK, Pakistan
2 School of Software Technology, Dalian University of Technology, Dalian 116000, China
* Corresponding Author: Waqas Haider, [email protected]
Volume 1, Issue 1
Volume 1, Issue 1 2026
Received: 10 January 2026 Accepted: 11 March 2026 Published: 16 April 2026 CrossMark
Download PDF 3.25 MB Cite Metrics
DOI: 10.62762/TAIC.2026.980583
ARK: ark:/57805/taic.2026.980583

Article Information

Published in ICCK Transactions on Applied Intelligence and Cybernetics
Volume/Issue Volume 1, Issue 1, 2026
Pages 36-52

Abstract

Crowdsourced software development (CSD) has gained increasing popularity in software engineering. However, empirical evidence on its sustainability remains limited. Although previous studies have suggested several sustainability-related factors, their applicability in real industrial CSD environments lacks sufficient validation. Through a systematic literature review (SLR), we identified eleven factors affecting sustainability in CSD. This study empirically validates these factors via a questionnaire survey with practitioners from various crowdsourced software development platforms. The results confirm that all eleven factors identified in the literature are also recognized by practitioners, with no additional factors reported. A comparison of factor rankings between the SLR and the survey yields a moderate positive Spearman rank correlation of 0.4909, indicating general agreement between academic and industry perspectives, albeit with some differences in prioritization. These findings provide empirical support for sustainability in CSD and offer practical insights for researchers and practitioners to better design and manage sustainable crowdsourced software development projects.

Graphical Abstract

Comparison Based on SLR and Empirical Data: Factors Affecting Sustainability Aspects in Software Crowdsourcing

Keywords

crowdsourcing crowdsourced software development (CSD) sustainability Empirical study systematic literature review (SLR)

Data Availability Statement

Data will be made available on request.

Funding

This work was supported without any funding.

Conflicts of Interest

The authors declare no conflicts of interest.

AI Use Statement

The authors declare that no generative AI was used in the preparation of this manuscript.

Ethical Approval and Consent to Participate

This work was conducted in accordance with the ethical principles of the Declaration of Helsinki. Ethical approval was not required for this research as it involved a voluntary online questionnaire survey with adult professionals, posed no more than minimal risk, and collected primarily non-sensitive professional opinions. Participation in the survey was entirely voluntary. All participants were informed about the purpose of the study, the approximate time required to complete the questionnaire, and that their responses would be treated confidentially and used solely for academic research purposes. By proceeding to complete and submit the questionnaire, participants implicitly provided informed consent to participate in the study. Participants could withdraw from the survey at any time without any consequences. No personally identifiable information was used in the analysis or reporting of results.

References

  1. Guillemin, F., Robitza, W., Wunderer, S., & Hoßfeld, T. (2022). Definitions of crowdsourced network and QoE measurements. ACM SIGMultimedia Records, 12(2), 1-1.
    [CrossRef] [Google Scholar]
  2. Howe, J. (2006). The rise of crowdsourcing. Wired, 14(6), 1-4.
    [Google Scholar]
  3. Estellés-Arolas, E., & González-Ladrón-de-Guevara, F. (2012). Towards an integrated crowdsourcing definition. Journal of Information science, 38(2), 189-200.
    [CrossRef] [Google Scholar]
  4. Hammon, L., & Hippner, H. (2012). Crowdsourcing. Business & Information systems engineering, 4(3), 163-166.
    [CrossRef] [Google Scholar]
  5. Christainas, G., Kehagias, D., Salamanis, A., Spanidis, P., Kyrkoy, M., & Tzovaras, D. (2022, August). A Crowdsourcing Framework for Reporting Available Parking Spots in Urban Areas. In Conference on Sustainable Urban Mobility (pp. 323-334). Cham: Springer Nature Switzerland.
    [CrossRef] [Google Scholar]
  6. Munir, Y., Umer, Q., Faheem, M., Akram, S., & Jaffar, A. (2025). Developer Recommendation and Team Formation in Collaborative Crowdsourcing Platforms. IEEE Access, 13, 63170-63185.
    [CrossRef] [Google Scholar]
  7. Tong, F., Yu, F., Xu, L., & Zhao, D. (2025, December). Subsystem Integration and Stability Evaluation Methods for Industrial Control Software Crowdsourcing Testing Platforms. In 2025 IEEE International Conference on Blockchain Technology and Information Security (ICBCTIS) (pp. 1-7). IEEE.
    [CrossRef] [Google Scholar]
  8. Klymenko, A., Meisenbacher, S., Favaro, L., & Matthes, F. (2025). Supporting the integration of privacy-enhancing technologies into the software development life cycle. SN Computer Science, 6(6), 592.
    [CrossRef] [Google Scholar]
  9. Khojastehpour, M., Sahebi, S., & Samimi, A. (2022). Public acceptance of a crowdsourcing platform for traffic enforcement. Case studies on transport policy, 10(4), 2012-2024.
    [CrossRef] [Google Scholar]
  10. Bazaluk, O., Rahman, M. A., Zayed, N. M., Faisal-E-Alam, M., Nitsenko, V., & Kucher, L. (2024). Crowdsourcing review: The crowd workers’ perspective. Journal of Industrial and Business Economics, 51(3), 647-666.
    [CrossRef] [Google Scholar]
  11. Elsawy, M., & Youssef, M. (2023). Economic sustainability: Meeting needs without compromising future generations. International Journal of Economics and Finance, 15(10), 23-31.
    [CrossRef] [Google Scholar]
  12. Zhang, Y., & Dong, C. (2024). Sustainable development of digital cultural heritage: a hybrid analysis of crowdsourcing projects using fsQCA and system dynamics. Sustainability, 16(17), 7577.
    [CrossRef] [Google Scholar]
  13. Hariram, N., Mekha, K., Suganthan, V., & Sudhakar, K. (2023). Sustainalism: An integrated socio-economic-environmental model to address sustainable development and sustainability. Sustainability, 15(13), 10682.
    [CrossRef] [Google Scholar]
  14. Moises de Souza, A. C., Soares Cruzes, D., Jaccheri, L., & Krogstie, J. (2023, December). Social sustainability approaches for software development: A systematic literature review. In International Conference on Product-Focused Software Process Improvement (pp. 478-494). Cham: Springer Nature Switzerland.
    [CrossRef] [Google Scholar]
  15. Zada, I., Shahzad, S., Ali, S., & Mehmood, R. M. (2023). OntoSuSD: Software engineering approaches integration ontology for sustainable software development. Software: Practice and Experience, 53(2), 283-317.
    [CrossRef] [Google Scholar]
  16. Khalifeh, A., Farrell, P., Alrousan, M., Alwardat, S., & Faisal, M. (2020). Incorporating sustainability into software projects: a conceptual framework. International Journal of Managing Projects in Business, 13(6), 1339-1361.
    [CrossRef] [Google Scholar]
  17. Haider, W., Ilyas, M., Khalid, S., & Ali, S. (2024). Factors influencing sustainability aspects in crowdsourced software development: A systematic literature review. Journal of Software: Evolution and Process, 36(6), e2630.
    [CrossRef] [Google Scholar]
  18. König, C., Lang, D. J., & Schaefer, I. (2025). Sustainable software engineering: Concepts, challenges, and vision. ACM Transactions on Software Engineering and Methodology, 34(5), 1-28.
    [CrossRef] [Google Scholar]
  19. Oyedeji, S., Adisa, M. O., Penzenstadler, B., & Wolf, A. (2019). Validation study of a framework for sustainable software system design and development. sustainable development, 3(4), 5.
    [Google Scholar]
  20. Khan, H. H., Malik, M. N., Alotaibi, Y., Alsufyani, A., & Alghamdi, S. (2021). Crowdsourced Requirements Engineering Challenges and Solutions: A Software Industry Perspective. Computer Systems Science & Engineering, 39(2).
    [CrossRef] [Google Scholar]
  21. Zhen, Y., Khan, A., Nazir, S., Huiqi, Z., Alharbi, A., & Khan, S. (2021). Crowdsourcing usage, task assignment methods, and crowdsourcing platforms: A systematic literature review. Journal of Software: Evolution and Process, 33(8), e2368.
    [CrossRef] [Google Scholar]
  22. Füller, J., Hutter, K., & Kröger, N. (2021). Crowdsourcing as a service–from pilot projects to sustainable innovation routines. International Journal of Project Management, 39(2), 183-195.
    [CrossRef] [Google Scholar]
  23. Chen, Y., Pandey, M., Song, J. Y., Lasecki, W. S., & Oney, S. (2020, April). Improving crowd-supported gui testing with structural guidance. In Proceedings of the 2020 CHI Conference on human factors in computing systems (pp. 1-13).
    [CrossRef] [Google Scholar]
  24. Alyahya, S. (2020). Crowdsourced software testing: A systematic literature review. Information and Software Technology, 127, 106363.
    [CrossRef] [Google Scholar]
  25. Ahmad, A., Khan, R. A., Khan, S. U., Alwageed, H. S., Al-Atawi, A. A., & Lee, Y. (2023). Green cloud computing adoption challenges and practices: A client’s perspective-based empirical investigation. Cognition, Technology & Work, 25(4), 427-446.
    [CrossRef] [Google Scholar]
  26. Malik, M. N., & Khan, H. H. (2018). Investigating software standards: A lens of sustainability for software crowdsourcing. IEEE Access, 6, 5139-5150.
    [CrossRef] [Google Scholar]
  27. Ahmad, A., Khan, S. U., Khan, H. U., Khan, G. M., & Ilyas, M. (2021). Challenges and Practices Identification via a Systematic Literature Review in the Adoption of Green Cloud Computing: Client’s Side Approach. IEEE Access, 9, 81828-81840.
    [CrossRef] [Google Scholar]
  28. Li, X., Zhang, Y., Osborne, C., Zhou, M., Jin, Z., & Liu, H. (2025). Systematic literature review of commercial participation in open source software. ACM Transactions on Software Engineering and Methodology, 34(2), 1-31.
    [CrossRef] [Google Scholar]
  29. Shrestha, N. (2021). Factor analysis as a tool for survey analysis. American journal of Applied Mathematics and statistics, 9(1), 4-11.
    [CrossRef] [Google Scholar]
  30. Nyirongo, B., Jiang, Y., Niu, N., & Liu, H. (2025). An Empirical Study of Software Refactorings in Real-World Open-Source Java Projects. IEEE Transactions on Software Engineering, 51(11), 3013-3037.
    [CrossRef] [Google Scholar]
  31. Baral, M. M., & Verma, A. (2021). Cloud computing adoption for healthcare: An empirical study using SEM approach. FIIB Business Review, 10(3), 255-275.
    [CrossRef] [Google Scholar]
  32. Kitchenham, B., & Charters, S. (2007). Guidelines for performing systematic literature reviews in software engineering (EBSE Technical Report EBSE-2007-01). Keele University & Durham University.
    [Google Scholar]
  33. Li, T., Higgins, J. P., & Deeks, J. J. (2019). Collecting data. Cochrane handbook for systematic reviews of interventions, 109-141.
    [CrossRef] [Google Scholar]
  34. Lunny, C., Ramasubbu, C., Puil, L., Liu, T., Gerrish, S., Salzwedel, D. M., ... & Wright, J. M. (2021). Over half of clinical practice guidelines use non-systematic methods to inform recommendations: a methods study. PLoS One, 16(4), e0250356.
    [CrossRef] [Google Scholar]
  35. Alem, D. D. (2020). An overview of data analysis and interpretations in research. International Journal of Academic Research in Education and Review, 8(1), 1-27.
    [CrossRef] [Google Scholar]
  36. Baltes, S., & Ralph, P. (2022). Sampling in software engineering research: A critical review and guidelines. Empirical Software Engineering, 27(4), 94.
    [CrossRef] [Google Scholar]
  37. Khan, R. A., Khan, S. U., Khan, H. U., & Ilyas, M. (2022). Systematic literature review on security risks and its practices in secure software development. IEEE Access, 10, 5456-5481.
    [CrossRef] [Google Scholar]
  38. Nirmani, S., Shahin, M., Khalajzadeh, H., & Liu, X. (2025). A systematic literature review on task recommendation systems for crowdsourced software engineering. Information and Software Technology, 184, 107753.
    [CrossRef] [Google Scholar]
  39. Imran, A., & Kosar, T. (2019). Software sustainability: a systematic literature review and comprehensive analysis. arXiv preprint arXiv:1910.06109.
    [Google Scholar]
  40. Salam, M., & Khan, S. U. (2018). Challenges in the development of green and sustainable software for software multisourcing vendors: Findings from a systematic literature review and industrial survey. Journal of Software: Evolution and Process, 30(8), e1939.
    [CrossRef] [Google Scholar]
  41. Pereira, R. A. A. (2018). Energyware engineering: techniques and tools for green software development (Doctoral dissertation, Universidade do Minho (Portugal)).
    [Google Scholar]
  42. Jakobsen, A. M. S. Ø., & Kadenic, M. D. (2025). Too Little Fit, Too Much Demand: A Task--Technology Perspective on Sustainability in Engineering Management. IEEE Transactions on Engineering Management, 73, 987-1008.
    [CrossRef] [Google Scholar]
  43. Mishra, A., & Mishra, D. (2021). Sustainable software engineering: Curriculum development based on ACM/IEEE guidelines. In Software Sustainability (pp. 269-285). Cham: Springer International Publishing.
    [CrossRef] [Google Scholar]
  44. Moreira, A., Lago, P., Heldal, R., Betz, S., Brooks, I., Capilla, R., ... & Venters, C. C. (2025). A roadmap for integrating sustainability into software engineering education. ACM Transactions on Software Engineering and Methodology, 34(5), 1-27.
    [CrossRef] [Google Scholar]
  45. Marantos, C., Papadopoulos, L., Lamprakos, C. P., Salapas, K., & Soudris, D. (2022). Bringing energy efficiency closer to application developers: An extensible software analysis framework. IEEE Transactions on Sustainable Computing, 8(2), 180-193.
    [CrossRef] [Google Scholar]
  46. Mao, K., Capra, L., Harman, M., & Jia, Y. (2017). A survey of the use of crowdsourcing in software engineering. Journal of Systems and Software, 126, 57-84.
    [CrossRef] [Google Scholar]
  47. Alsayyari, M., & Alyahya, S. (2018, March). Supporting coordination in crowdsourced software testing services. In 2018 IEEE Symposium on Service-Oriented System Engineering (SOSE) (pp. 69-75). IEEE.
    [CrossRef] [Google Scholar]
  48. Jayanti, E. (2012). Open sourced organizational learning: implications and challenges of crowdsourcing for human resource development (HRD) practitioners. Human Resource Development International, 15(3), 375-384.
    [CrossRef] [Google Scholar]
  49. Sarı, A., Tosun, A., & Alptekin, G. I. (2019). A systematic literature review on crowdsourcing in software engineering. Journal of Systems and Software, 153, 200-219.
    [CrossRef] [Google Scholar]
  50. de Campos, V. Q., David, J. M. N., & Braga, R. (2021, May). Coordination in crowdsourced software development: A systematic mapping study. In 2021 IEEE 24th International Conference on Computer Supported Cooperative Work in Design (CSCWD) (pp. 305-310). IEEE.
    [CrossRef] [Google Scholar]
  51. Jiang, H., & Matsubara, S. (2014, December). Efficient task decomposition in crowdsourcing. In International conference on principles and practice of multi-agent systems (pp. 65-73). Cham: Springer International Publishing.
    [CrossRef] [Google Scholar]
  52. Uchoa, A. P., & Schneider, D. (2026). Driving the Emergence of Self-managed Enterprises of Crowdworkers: A Survey and Review of Motivating Factors. In International Conference on Enterprise Information Systems (pp. 63-88). Springer, Cham.
    [CrossRef] [Google Scholar]
  53. LaToza, T. D., & Van Der Hoek, A. (2015). Crowdsourcing in software engineering: Models, motivations, and challenges. IEEE software, 33(1), 74-80.
    [CrossRef] [Google Scholar]
  54. Maddalena, E., Ibáñez, L.-D., Reeves, N., & Simperl, E. (2023). Qrowdsmith: Enhancing paid microtask crowdsourcing with gamification and furtherance incentives. ACM Transactions on Intelligent Systems and Technology, 14(5), 1-26.
    [CrossRef] [Google Scholar]
  55. Dratler Jr, J., & McJohn, S. M. (2025). Intellectual property law: Commercial, creative and industrial property. Law Journal Press.
    [Google Scholar]
  56. Kumar, P. (2024). Intellectual property rights (IPR): nurturing creativity, fostering innovation. ldealism Journal of Advanced Research in Progressive Spectrums (IJARPS) eISSN--2583-6986, 3(02), 32-38.
    [Google Scholar]
  57. Atadoga, A., Umoga, U. J., Lottu, O. A., & Sodiy, E. O. (2024). Tools, techniques, and trends in sustainable software engineering: A critical review of current practices and future directions. World Journal of Advanced Engineering Technology and Sciences, 11(1), 231-239.
    [CrossRef] [Google Scholar]

Cite This Article

APA Style
Haider, W., Khan, A., Ihsan, S., & Adil, I. (2026). Comparison Based on SLR and Empirical Data: Factors Affecting Sustainability Aspects in Software Crowdsourcing. ICCK Transactions on Applied Intelligence and Cybernetics, 1(1), 36–52. https://doi.org/10.62762/TAIC.2026.980583
Export Citation
RIS Format
Compatible with EndNote, Zotero, Mendeley, and other reference managers
TY  - JOUR
AU  - Haider, Waqas
AU  - Khan, Adnan
AU  - Ihsan, Samreen
AU  - Adil, Ihsan
PY  - 2026
DA  - 2026/04/16
TI  - Comparison Based on SLR and Empirical Data: Factors Affecting Sustainability Aspects in Software Crowdsourcing
JO  - ICCK Transactions on Applied Intelligence and Cybernetics
T2  - ICCK Transactions on Applied Intelligence and Cybernetics
JF  - ICCK Transactions on Applied Intelligence and Cybernetics
VL  - 1
IS  - 1
SP  - 36
EP  - 52
DO  - 10.62762/TAIC.2026.980583
UR  - https://www.icck.org/article/abs/TAIC.2026.980583
KW  - crowdsourcing
KW  - crowdsourced software development (CSD)
KW  - sustainability
KW  - Empirical study
KW  - systematic literature review (SLR)
AB  - Crowdsourced software development (CSD) has gained increasing popularity in software engineering. However, empirical evidence on its sustainability remains limited. Although previous studies have suggested several sustainability-related factors, their applicability in real industrial CSD environments lacks sufficient validation. Through a systematic literature review (SLR), we identified eleven factors affecting sustainability in CSD. This study empirically validates these factors via a questionnaire survey with practitioners from various crowdsourced software development platforms. The results confirm that all eleven factors identified in the literature are also recognized by practitioners, with no additional factors reported. A comparison of factor rankings between the SLR and the survey yields a moderate positive Spearman rank correlation of 0.4909, indicating general agreement between academic and industry perspectives, albeit with some differences in prioritization. These findings provide empirical support for sustainability in CSD and offer practical insights for researchers and practitioners to better design and manage sustainable crowdsourced software development projects.
SN  - 3143-0309
PB  - Institute of Central Computation and Knowledge
LA  - English
ER  -
BibTeX Format
Compatible with LaTeX, BibTeX, and other reference managers
@article{Haider2026Comparison,
  author = {Waqas Haider and Adnan Khan and Samreen Ihsan and Ihsan Adil},
  title = {Comparison Based on SLR and Empirical Data: Factors Affecting Sustainability Aspects in Software Crowdsourcing},
  journal = {ICCK Transactions on Applied Intelligence and Cybernetics},
  year = {2026},
  volume = {1},
  number = {1},
  pages = {36-52},
  doi = {10.62762/TAIC.2026.980583},
  url = {https://www.icck.org/article/abs/TAIC.2026.980583},
  abstract = {Crowdsourced software development (CSD) has gained increasing popularity in software engineering. However, empirical evidence on its sustainability remains limited. Although previous studies have suggested several sustainability-related factors, their applicability in real industrial CSD environments lacks sufficient validation. Through a systematic literature review (SLR), we identified eleven factors affecting sustainability in CSD. This study empirically validates these factors via a questionnaire survey with practitioners from various crowdsourced software development platforms. The results confirm that all eleven factors identified in the literature are also recognized by practitioners, with no additional factors reported. A comparison of factor rankings between the SLR and the survey yields a moderate positive Spearman rank correlation of 0.4909, indicating general agreement between academic and industry perspectives, albeit with some differences in prioritization. These findings provide empirical support for sustainability in CSD and offer practical insights for researchers and practitioners to better design and manage sustainable crowdsourced software development projects.},
  keywords = {crowdsourcing, crowdsourced software development (CSD), sustainability, Empirical study, systematic literature review (SLR)},
  issn = {3143-0309},
  publisher = {Institute of Central Computation and Knowledge}
}

Article Metrics

Citations
Crossref
0
Scopus
0
Views
584
PDF Downloads
156

Publisher's Note

ICCK stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and Permissions

CC BY Copyright © 2026 by the Author(s). Published by Institute of Central Computation and Knowledge. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.
ICCK Transactions on Applied Intelligence and Cybernetics
ICCK Transactions on Applied Intelligence and Cybernetics
ISSN: 3143-0309 (Online)
Portico
Preserved at
Portico

Article QR Code

Article QR Code
Scan to read this article
Submit Manuscript Edit a Special Issue

Popular Articles

Share This Article