Measuring and Repairing Document Conversion Errors in the Michigan Child Welfare Policy Manual

<p>Child welfare practice is governed by documents written for people to read. A state policy manual specifies how quickly a caseworker must see a child after the child returns home, and it does so in a PDF laid out for the eye. Any system that answers questions from such a manual must first convert it into text, and that conversion step is rarely measured. This study measures it. Five test documents were constructed to replicate the layout of the Michigan Children’s Protective Services Manual, with identifiers replacing its words. The identifiers run consecutively, so that a loss reads as a gap in a counted run and a person can see it without tooling, and each one ends in a check code, so that a converter driven by a language model cannot supply a convincing replacement for whatever it dropped. The same artifact is therefore verifiable by a person reading it and by a script counting it, and the tool under test cannot repair the evidence before either of them looks at it.Four converters were evaluated on the test documents, and the two generations of the commercial service were evaluated on the manual itself, for a total of 31 documents and 328 pages. No converter was adequate for every property required by a retrieval corpus. The older generation invented 43 section headings that appear nowhere in the source, and the corpus assembled from it labels a column in a contact-standards table with the wrong requirement, so it lists three contacts with each child at intensive risk, where the manual states one. The current generation’s cheapest tier deletes the opening line of every paragraph beginning Exception:, which is how this manual marks the cases a rule does not cover. None of these failures garbles the text. Each produces fluent prose that reads as policy and states a rule the manual does not contain. The study then reports a repair procedure that derives document structure from the PDF’s geometry, and fifteen checks that halt conversion if any fail. Every check passes across all 31 documents, measured against the same documents from which the repair rules were derived. A fourth failure property emerged only after presence, level, and order had all reached their best values: 380 paragraph boundaries across 200 of the 328 pages were marked in the source by vertical spacing that the converter does not read.</p>