Optimizing Threshold-based Financial Crime Detection: Smoothing Rank Scale Gradient Descent for Discrete Loss Function-Based Threshold Adjustment

<title>Abstract</title> <p>Financial institutions employ threshold-based detection for anti-money laundering (AML) and counter-terrorist financing (CTF) to meet regulatory interpretability requirements, creating a discrete optimization challenge for balancing detection accuracy with operational costs. Regulatory agencies define suspicious transaction patterns requiring thresholds on transaction frequency and amounts. However, static thresholds are susceptible to reverse engineering, necessitating dynamic adjustment to balance recall, Human Resource Savings (HRS), and precision. The resulting discrete loss function exhibits dramatically varying gradients, complicating optimization. Traditional methods require exhaustive enumeration, while simulated annealing suffers from slow convergence and instability. We propose Smoothing Rank Scale Gradient Descent (SRSGD), transforming the discrete loss function into a continuous one through sigmoid-based approximation. Uneven data distribution creates rugged loss landscapes with dramatically varying gradients, making learning rate adjustment difficult and explaining simulated annealing's systematic failures. SRSGD addresses this through two essential components: (1) rank-scale transformation that handles extreme gradient variations caused by uneven data density, enabling effective learning rate tuning, and (2) quantile-based initial threshold selection providing effective starting points. Optimized thresholds are determined in rank space then converted to actual values. SRSGD achieves 4000 times computational speedup over discrete gradient methods while delivering superior HRS and maintaining regulatory-compliant recall. This framework has been successfully integrated into AML systems at financial institutions, enhancing financial crime detection while adhering to Financial Supervisory Commission standards.</p>