Abstract
<title>Abstract</title> <p> Autonomous multi-agent systems built on large language models exchange goals, constraints, evidence commitments, and tool-mediated actions through natural-language handoffs. Across hops, semantic meaning may drift, constraints may weaken, unsupported claims may propagate, and organizations may lose visibility into how decisions were produced. This paper develops a semantic-tracing framework that treats meaning preservation as a first-class measurand and operationalizes it with three LangChain ecosystem components that are genuinely used in the local experimental artifact: LangChain for LLM invocation, prompt templates, tool interfaces, structured-output parsing, and retries; LangGraph for stateful multi-agent orchestration (nodes, edges, workflow state, routing, and human-review control); and LangSmith for private tracing, span capture, metadata, and evaluation-linked observability. The system separates framework orchestration, LLM reasoning, deterministic safeguards, trace capture, semantic evaluation, and human decision. The framework is anchored in SHADOWAI-RISK, a live academic research prototype for enterprise AI governance and risk intelligence. In addition to the live prototype, this study implements a separate local experimental extension with (i) a preserved deterministic baseline pathway and (ii) a genuine LLM multi-agent condition (Inventory → Evidence → Governance → Human Review Gate) implemented with LangChain (ChatOpenAI, ChatPromptTemplate, JsonOutputParser, tool wrappers, retries), LangGraph (StateGraph nodes/edges/state), optional private LangSmith instrumentation, and a rule-based semantic evaluation layer. The full experimental matrix is complete at 450/450 (complete:true; deterministic 32, llm_no_eval 160, llm_langsmith_semantic_eval 160, variability 50, ablations 48), with 0 terminal failures. Primary eval versus no-eval contrasts (n=160 each) show Holm-corrected Mann–Whitney effects including lower TCS with evaluation (0.624 vs. 0.912, Holm-adjusted p < 0.001, Cliff δ≈ −0.33 medium), significantly lower WRS_equal and WRS_expert (small effects), non-significant SPS/AHF/IDR/TUA/GCR/HPR/HEP, higher latency with evaluation (∼126.2 s vs. ∼101.7 s, Cliff δ≈0.67 large), non-significant token usage, and $0 local cost; decision-family correctness did not differ (59/160 vs. 57/160; Fisher/χ <sup>2</sup> n.s.). Human calibration, inter-rater agreement, surveys, and production integration remain pending. </p>