Abstract
<title>Abstract</title> <p>The proliferation of Large Language Models (LLMs) has driven a paradigm shift toward on-device inference, motivated by privacy preservation, latency reduction, and offline capability. Simultaneously, Retrieval-Augmented Generation (RAG) has emerged as the dominant pattern for grounding language model outputs in external knowledge. However, deploying RAG pipelines on resource-constrained edge devices—smartphones, IoT gateways, and embedded systems equipped with Neural Processing Units (NPUs)—introduces a complex multi-dimensional trade-off among accuracy, latency, memory footprint, energy consumption, and carbon emissions. While each pillar has been surveyed individually, the three-way intersection of on-device AI inference optimization, retrieval-augmented generation, and Green AI / sustainability has not previously been drawn together into a unified system-design perspective, and this survey undertakes that cross-domain synthesis. We systematically examine (i) model compression techniques—quantization, pruning, knowledge distillation—with emphasis on NPU-aware optimization; (ii) on-device RAG system architectures including lightweight retrieval, context compression, and modular frameworks; and (iii) energy measurement, carbon footprint estimation, and sustainable inference strategies for edge deployment. As our core contribution, we propose ALEMC, an evaluation framework extending conventional metrics with explicit Energy and Carbon dimensions, and synthesize design principles for building sustainable on-device RAG systems. We review 142 papers spanning 2017–2026, identify open challenges in hardware–software co-design, standardized energy benchmarking, and lifecycle-aware model selection, and outline future research directions at this rapidly evolving intersection.</p>