Developing robust data engineering pipelines for complex retail analytics and manufacturing intelligence

What is Data Engineering? Why is data engineering crucial for analytics in retail and manufacturing? Researchers in the fields of marketing and operations management have emphasized the importance of making analytical insights readily available for front-line users. Largely ignored in this literature is the necessity of creating a data engineering platform that will make the availability of data possible. Such a data engineering effort not only facilitates the smooth functioning of analytics in organizations but also leads to democratization of analytics by making it available for use by front line management and, even by customers. In this paper, we define, delineate, and describe the extensive and complex processes and activities involved in constructing the foundations of a robust data engineering platform.