Building data engineering pipelines for payment processing and risk analysis

As long as financial transactions are recorded and exchanged, it has always been unavoidable that irresponsible actors, or risk factors, engage in various acts of fraud to gain wealth unethically. Such unauthorized transactions include stealing of payment method information, creating unauthorized accounts for receiving gain from illegal activities such as money laundering, creating unauthorized merchant accounts to trick customers into fraudulent transactions, and phishing to swindle customers. Payment processing companies should detect such behaviors to protect legitimate users and save their companies from losing a considerable amount of money (Ghemawat et al., 2003; Kimball & Ross, 2013; Akidau et al., 2015). Growing up in an internet-enriched environment, millennials and Generation Z are the largest active user segment. They are highly sensitive to payment transaction efficiencies and prone to try new payment methods compared to Generation X and Baby Boomers. These new trends require payment service providers to design novel transaction processing systems to accommodate different payment methods, account users, and merchants, while keeping real-time fraud detection at tolerable costs. Acquiring, storing, and processing the intensive data streams generated by users during payment transactions are core challenges for payment processing services. Streaming data engineering is the cornerstone for building such payment processing and risk analysis pipelines. Data such as transactions, partner systems, and payment gateways are collected to a data lake from internal and external sources. Thereafter, the collected data are processed using batch or streaming pipelines to provide real-time transactional and risk insights for the payment operations organizations and partners. Those insights are then ingested into dashboards for tracking transaction activity and risk detection performance.