Research Article
Open Access
Optimizing Cloud-Native Lakehouse Architectures for Real-Time Semiconductor Analytics: Balancing Performance, Cost, and Energy Efficiency
1
University of California, Berkeley, CA 94720, United States
* Corresponding Author:
Min Yin,
[email protected]
Volume 2, Issue 3
2026
Received: 10 November 2025
Revised: 26 December 2025
Accepted: 12 March 2026
Published: 05 August 2026
Article Information
Published in
ICCK Transactions on Advanced Computing and Systems
Volume/Issue
Volume 2, Issue 3, 2026
Pages
255-271
Abstract
This paper presents a cloud-native Lakehouse architecture designed for real-time semiconductor analytics, with a focus on optimizing storage tiering, data lineage, and cost-energy co-optimization. As semiconductor data analytics require processing massive amounts of real-time data, traditional data warehouses are often insufficient in addressing the need for low-latency, high-concurrency queries. The proposed framework leverages cloud-native technologies, such as AWS, Azure, and distributed databases like Apache Doris, to design a dynamic multi-tier storage system that segregates data based on access frequency and volatility, incorporating columnar compression techniques for efficient storage and faster query performance. Moreover, the paper introduces an innovative query routing mechanism and data lineage tracking framework to ensure data transparency and verifiability, critical for industrial applications. A novel cost-energy optimization model is proposed, which balances query performance with resource efficiency, aiming to minimize both operational costs and energy consumption while meeting business KPIs and SLAs. Experimental results demonstrate significant improvements in storage efficiency, query performance, and cost-energy trade-offs, particularly in semiconductor industry use cases. This research offers practical solutions to overcome the limitations of existing architectures and provides valuable insights for the future development of cloud-native platforms for real-time industrial analytics. Further investigation is needed to explore the scalability and adaptability of the proposed model in different data-intensive domains.
Graphical Abstract
Keywords
cloud-native Lakehouse
semiconductor analytics
storage tiering
columnar compression
query routing
cost-energy optimization
Data Availability Statement
Data will be made available on request.
Funding
This work was supported without any funding.
Conflicts of Interest
The authors declare no conflicts of interest.
AI Use Statement
The authors declare that no generative AI was used in the preparation of this manuscript.
Ethical Approval and Consent to Participate
Not applicable.
References
- Espadinha-Cruz, P., Godina, R., & Rodrigues, E. M. (2021). A review of data mining applications in semiconductor manufacturing. Processes, 9(2), 305.
[CrossRef] [Google Scholar] - Dean, J., & Barroso, L. A. (2013). The tail at scale. Communications of the ACM, 56(2), 74-80.
[CrossRef] [Google Scholar] - Moyne, J., & Iskandar, J. (2017). Big data analytics for smart manufacturing: Case studies in semiconductor manufacturing. Processes, 5(3), 39.
[CrossRef] [Google Scholar] - Armbrust, M., Das, T., Sun, L., Yavuz, B., Zhu, S., Murthy, M., ... & Zaharia, M. (2020). Delta Lake: High-performance ACID table storage over cloud object stores. Proceedings of the VLDB Endowment, 13(12), 3411-3424.
[CrossRef] [Google Scholar] - Armbrust, M., Ghodsi, A., Xin, R., & Zaharia, M. (2021). Lakehouse: a new generation of open platforms that unify data warehousing and advanced analytics. In Proceedings of the 11th Annual Conference on Innovative Data Systems Research (CIDR 2021), Amsterdam, The Netherlands. https://vldb.org/cidrdb/papers/2021/cidr2021_paper17.pdf
[Google Scholar] - Dageville, B., Cruanes, T., Zukowski, M., Antonov, V., Avanes, A., Bock, J., ... & Unterbrunner, P. (2016, June). The snowflake elastic data warehouse. In Proceedings of the 2016 International Conference on Management of Data (pp. 215-226).
[CrossRef] [Google Scholar] - Mazumdar, D., Hughes, J., & Onofre, J. B. (2023). The data lakehouse: Data warehousing and more. arXiv preprint arXiv:2310.08697.
[CrossRef] [Google Scholar] - Vuppalapati, M., Miron, J., Agarwal, R., Truong, D., Motivala, A., & Cruanes, T. (2020). Building an elastic query engine on disaggregated storage. In 17th USENIX Symposium on Networked Systems Design and Implementation (NSDI 20) (pp. 449-462). https://www.usenix.org/conference/nsdi20/presentation/vuppalapati
[Google Scholar] - Marcus, R., Negi, P., Mao, H., Zhang, C., Alizadeh, M., Kraska, T., ... & Tatbul, N. (2019). Neo: A learned query optimizer. Proceedings of the VLDB Endowment, 12(11), 1705-1718.
[CrossRef] [Google Scholar] - Leis, V., & Kuschewski, M. (2021). Towards cost-optimal query processing in the cloud. Proceedings of the VLDB Endowment, 14(9), 1606-1612.
[CrossRef] [Google Scholar] - Marcus, R., & Papaemmanouil, O. (2019). Plan-structured deep neural network models for query performance prediction. arXiv preprint arXiv:1902.00132. https://arxiv.org/abs/1902.00132
[Google Scholar] - Zhou, X., Sun, J., Li, G., & Feng, J. (2020). Query performance prediction for concurrent queries using graph embedding. Proceedings of the VLDB Endowment, 13(9), 1416-1428.
[CrossRef] [Google Scholar] - Herodotou, H., & Kakoulli, E. (2019). Automating distributed tiered storage management in cluster computing. arXiv preprint arXiv:1907.02394. https://arxiv.org/abs/1907.02394
[Google Scholar] - Susto, G. A., Schirru, A., Pampuri, S., McLoone, S., & Beghi, A. (2015). Machine learning for predictive maintenance: A multiple classifier approach. IEEE Transactions on Industrial Informatics, 11(3), 812-820.
[CrossRef] [Google Scholar] - Gog, I., Schwarzkopf, M., Crooks, N., Grosvenor, M. P., Clement, A., & Hand, S. (2015, April). Musketeer: All for one, one for all in data processing systems. In Proceedings of the Tenth European Conference on Computer Systems (pp. 1-16).
[CrossRef] [Google Scholar] - Herschel, M., Diestelk\"{amper, R., & Ben Lahmar, H. (2017). A survey on provenance: What for? What form? What from? The VLDB Journal, 26(6), 881-906.
[CrossRef] [Google Scholar] - Interlandi, M., Shah, K., Tetali, S. D., Gulzar, M. A., Yoo, S., Kim, M., ... & Condie, T. (2015). Titian: Data provenance support in Spark. Proceedings of the VLDB Endowment, 9(3), 216-227.
[CrossRef] [Google Scholar] - Radovanovi\'{c, A., Koningstein, R., Schneider, I., Chen, B., Duarte, A., Roy, B., ... & Cirne, W. (2023). Carbon-aware computing for datacenters. IEEE Transactions on Power Systems, 38(2), 1270-1280.
[CrossRef] [Google Scholar] - Pedreira, P., Erling, O., Basmanova, M., Wilfong, K., Sakka, L., Pai, K., ... & Chattopadhyay, B. (2022). Velox: Meta's Unified Execution Engine. Proc. VLDB Endow., 15(12), 3372-3384.
[CrossRef] [Google Scholar] - Van Renen, A., & Leis, V. (2023). Cloud analytics benchmark. Proceedings of the VLDB Endowment, 16(6), 1413-1425.
[CrossRef] [Google Scholar] - McCann, M., & Johnston, A. (2008). SECOM dataset. UCI Machine Learning Repository.
[CrossRef] [Google Scholar] - Zaharia, M., Xin, R. S., Wendell, P., Das, T., Armbrust, M., Dave, A., ... & Stoica, I. (2016). Apache Spark: A unified engine for big data processing. Communications of the ACM, 59(11), 56-65.
[CrossRef] [Google Scholar] - Ousterhout, K., Rasti, R., Ratnasamy, S., Shenker, S., & Chun, B. G. (2015). Making sense of performance in data analytics frameworks. In 12th USENIX Symposium on Networked Systems Design and Implementation (NSDI 15) (pp. 293-307). https://www.usenix.org/conference/nsdi15/technical-sessions/presentation/ousterhout
[Google Scholar] - Chen, Y., Alspaugh, S., & Katz, R. (2012). Interactive analytical processing in big data systems: A cross-industry study of MapReduce workloads. Proceedings of the VLDB Endowment, 5(12), 1802-1813.
[CrossRef] [Google Scholar] - Karpathiotakis, M., De Oliveira Branco, M. S., Alagiannis, I., & Ailamaki, A. (2014). Adaptive query processing on RAW data. Proceedings of the VLDB Endowment, 7(13), 1364-1375.
[CrossRef] [Google Scholar] - Hsieh, J. W., Kuo, T. W., & Chang, L. P. (2006). Efficient identification of hot data for flash memory storage systems. ACM Transactions on Storage (TOS), 2(1), 22-40.
[CrossRef] [Google Scholar] - Han, Y., Wu, Z., Wu, P., Zhu, R., Yang, J., Tan, L. W., ... & Cui, B. (2021). Cardinality estimation in DBMS: A comprehensive benchmark evaluation. arXiv preprint arXiv:2109.05877.
[CrossRef] [Google Scholar] - Psallidas, F., & Wu, E. (2018). Smoke: Fine-grained lineage at interactive speed. arXiv preprint arXiv:1801.07237. https://arxiv.org/abs/1801.07237
[Google Scholar] - Tsirogiannis, D., Harizopoulos, S., & Shah, M. A. (2010). Analyzing the energy efficiency of a database server. In Proceedings of the 2010 ACM SIGMOD International Conference on Management of Data (pp. 231-242).
[CrossRef] [Google Scholar] - Ananthanarayanan, G., Douglas, C., Ramakrishnan, R., Rao, S., & Stoica, I. (2012, October). True elasticity in multi-tenant data-intensive compute clusters. In Proceedings of the Third ACM Symposium on Cloud Computing (pp. 1-7).
[CrossRef] [Google Scholar] - Collberg, C., & Proebsting, T. A. (2016). Repeatability in computer systems research. Communications of the ACM, 59(3), 62-69.
[CrossRef] [Google Scholar] - Camacho-Rodr\'{iguez, J., Agrawal, A., Gruenheid, A., Gosalia, A., Petculescu, C., Aguilar-Saborit, J., ... & Ramakrishnan, R. (2024). Lst-bench: Benchmarking log-structured tables in the cloud. Proceedings of the ACM on Management of Data, 2(1), 1-26.
[CrossRef] [Google Scholar]
Cite This Article
APA Style
Yin, M., & Frank, L. F. (2026). Optimizing Cloud-Native Lakehouse Architectures for Real-Time Semiconductor Analytics: Balancing Performance, Cost, and Energy Efficiency. ICCK Transactions on Advanced Computing and Systems, 2(3), 255-271. https://doi.org/10.62762/TACS.2025.879079
Export Citation
RIS Format
Compatible with EndNote, Zotero, Mendeley, and other reference managers
TY - JOUR AU - Yin, Min AU - Frank, Ledee-FI PY - 2026 DA - 2026/08/05 TI - Optimizing Cloud-Native Lakehouse Architectures for Real-Time Semiconductor Analytics: Balancing Performance, Cost, and Energy Efficiency JO - ICCK Transactions on Advanced Computing and Systems T2 - ICCK Transactions on Advanced Computing and Systems JF - ICCK Transactions on Advanced Computing and Systems VL - 2 IS - 3 SP - 255 EP - 271 DO - 10.62762/TACS.2025.879079 UR - https://www.icck.org/article/abs/TACS.2025.879079 KW - cloud-native Lakehouse KW - semiconductor analytics KW - storage tiering KW - columnar compression KW - query routing KW - cost-energy optimization AB - This paper presents a cloud-native Lakehouse architecture designed for real-time semiconductor analytics, with a focus on optimizing storage tiering, data lineage, and cost-energy co-optimization. As semiconductor data analytics require processing massive amounts of real-time data, traditional data warehouses are often insufficient in addressing the need for low-latency, high-concurrency queries. The proposed framework leverages cloud-native technologies, such as AWS, Azure, and distributed databases like Apache Doris, to design a dynamic multi-tier storage system that segregates data based on access frequency and volatility, incorporating columnar compression techniques for efficient storage and faster query performance. Moreover, the paper introduces an innovative query routing mechanism and data lineage tracking framework to ensure data transparency and verifiability, critical for industrial applications. A novel cost-energy optimization model is proposed, which balances query performance with resource efficiency, aiming to minimize both operational costs and energy consumption while meeting business KPIs and SLAs. Experimental results demonstrate significant improvements in storage efficiency, query performance, and cost-energy trade-offs, particularly in semiconductor industry use cases. This research offers practical solutions to overcome the limitations of existing architectures and provides valuable insights for the future development of cloud-native platforms for real-time industrial analytics. Further investigation is needed to explore the scalability and adaptability of the proposed model in different data-intensive domains. SN - 3068-7969 PB - Institute of Central Computation and Knowledge LA - English ER -
BibTeX Format
Compatible with LaTeX, BibTeX, and other reference managers
@article{Yin2026Optimizing,
author = {Min Yin and Ledee-FI Frank},
title = {Optimizing Cloud-Native Lakehouse Architectures for Real-Time Semiconductor Analytics: Balancing Performance, Cost, and Energy Efficiency},
journal = {ICCK Transactions on Advanced Computing and Systems},
year = {2026},
volume = {2},
number = {3},
pages = {255-271},
doi = {10.62762/TACS.2025.879079},
url = {https://www.icck.org/article/abs/TACS.2025.879079},
abstract = {This paper presents a cloud-native Lakehouse architecture designed for real-time semiconductor analytics, with a focus on optimizing storage tiering, data lineage, and cost-energy co-optimization. As semiconductor data analytics require processing massive amounts of real-time data, traditional data warehouses are often insufficient in addressing the need for low-latency, high-concurrency queries. The proposed framework leverages cloud-native technologies, such as AWS, Azure, and distributed databases like Apache Doris, to design a dynamic multi-tier storage system that segregates data based on access frequency and volatility, incorporating columnar compression techniques for efficient storage and faster query performance. Moreover, the paper introduces an innovative query routing mechanism and data lineage tracking framework to ensure data transparency and verifiability, critical for industrial applications. A novel cost-energy optimization model is proposed, which balances query performance with resource efficiency, aiming to minimize both operational costs and energy consumption while meeting business KPIs and SLAs. Experimental results demonstrate significant improvements in storage efficiency, query performance, and cost-energy trade-offs, particularly in semiconductor industry use cases. This research offers practical solutions to overcome the limitations of existing architectures and provides valuable insights for the future development of cloud-native platforms for real-time industrial analytics. Further investigation is needed to explore the scalability and adaptability of the proposed model in different data-intensive domains.},
keywords = {cloud-native Lakehouse, semiconductor analytics, storage tiering, columnar compression, query routing, cost-energy optimization},
issn = {3068-7969},
publisher = {Institute of Central Computation and Knowledge}
}
Article Metrics
Publisher's Note
ICCK stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and Permissions
Copyright © 2026 by the Author(s). Published by Institute of Central Computation and Knowledge. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.
Preserved at
Portico
Portico
Scan to read this article