FEMS Project #2 — A Real 230-Page Corpus + Chunk Quality Gates + the Chinese-Character Mixing Trap in a Local LLM
Built a corpus of roughly 230 pages / 35,000 rows from the Korea BEMS Association guides, Korea Energy Agency materials, and the UCI Steel dataset. After paragraph-based chunking (target 800 chars) + bge-m3 + Chroma indexing, a quality gate (ratio of complete Hangul/ASCII characters) excluded 7 chunks from table-of-contents pages. Then I hit a trap — qwen2.5:7b mixed in Chinese characters on the second question and suffered generation collapse (spitting out unrelated Chinese city coordinates as GeoJSON). Partially fixed with temperature / system prompt → ultimately switched to exaone3.5:7.8b for clean handling.
