The evolution of data engineering and its impact on global digital transformation

Data Engineering is a relatively new term, which has resulted from the necessary protection of the engineering profession and the natural evolution of Digital Technology, in this way Data Engineers have to work closely with Data Science. The use of Data Science has been available for a short period of time, and plenty of times, the Data Scientists are mistaken and considered Data Engineers, therefore these tasks are separated and come back to repeat its original meanings and function; therefore, it is necessary to take a look through history to better understand these two new professions.The data-driven organizations are at the top of their respective markets, considering the point of view of digital transformation that gives organizations innovate through Digital Technology. As Digital Transformation is a new way of thinking how organizations utilize Digital Technology, Data Engineering has to work to define the foundations of assured data, which is essential for the algorithms necessary for these organizational strategies and tactics. Digital Technologies evolve at a high rate, in the production, storage, and transport of data; we are talking about the Fourth Industrial Revolution, where the most important assets are information and the processes that generate knowing-how are more efficient and lean, but having their foundations based on accurate data. How do organizations use their organizational strategies focused on digital transformation? Having the backlogs of their data transformation processes developed by a certified information technology professional, the Data Engineer (Chen et al., 2012; Chen & Lien, 2014; Einsiedler et al., 2015).

Data Engineers are professionals who are responsible for building the management chains of Data in Business Intelligence solutions, from Data Extraction, Transformation, and Loading, to the Data Visualization tools, they design and optimize Data Pipelines and Datamarts, Data Warehouses and Data Lakes, Data Robotics scripts, Cloud Computing, developing automation scripts, using Application Programming Interfaces and connecting data from different databases, in Data Platforms. But continuing our approach, Data Engineering is within Business Intelligence and never outside as Data Science (Panwar et al., 2016; Kumar et al., 2024).