Data Engineering for Risk and Fraud Intelligence

Data engineering involves designing and building scalable infrastructure, architecture, and workflows for ingesting, processing, and storing very large volumes of data. It is a prerequisite for data science, machine learning, and big data analytics. Data Engineering for Risk and Fraud Intelligence applies these general principles to the specifics of risk and fraud intelligence. Investigations into specific data sourcing techniques, storage options, and risk analysis methods help generate high-quality data to feed risk models. Many of the risks that organizations face are related to the risk of fraud and financial crime. Fraud consumes an inordinate amount of an organization’s resources and exposes the firm to considerable losses and regulatory penalties. These are some of the primary objectives of risk and fraud intelligence: detecting money laundering; mitigating fraud losses; understanding the identity of the persons you are investigating; understanding the activity undertaken by those persons; and determining the risk associated with those persons.

Risk and fraud intelligence require a vast amount of different data types—structured and unstructured. Data can be utilized straight from a database, or can be extracted by scraping websites, using external data feeds, or tapping into social media URLs. These data streams can take the form of either batch or real-time, depending on the forensic and analytics requirements. Storage options range from online, transactional systems to data lakes or federated data warehouses. Increasingly, storage is moving to the Cloud for considerations regarding security, scale, location, and regulatory compliance. Risk and fraud models can be generated through batch or stream processing paradigms. Data pipelines can be built using either an Extract, Transform, Load (ETL) or Extract, Load, Transform (ELT) approach. The following sections explore these aspects in more detail.