Tag: rag
All the articles with the tag "rag".
-
RAG Embedding Comparison — Measuring recall@k on My Blog Data (OpenAI vs bge-m3)
After establishing the ["feeling-based benchmarking → numeric benchmarking" principle](/en/posts/quant-study-00-pandas) in my quant retrospective, I actually quantified an embedding model comparison this time. I indexed 441 chunks from my blog posts with OpenAI text-embedding-3-small and bge-m3 respectively, then measured recall@3 with a test set of 20 question-answer source pairs. Overall: OpenAI 80% vs bge-m3 90%. bge-m3 hit 100% on hard-difficulty questions — the decisive factor was connecting to the source text by meaning even when words didn't overlap. On easy questions, the misses turned out to be caused by typos (cladue, underscores) — a twist showing the grading criteria itself was wrong.
-
Eval Study #2 — The Pitfalls of Similarity-Based Evaluation, 5 Principles of Test Set Design, and a Misunderstanding About the Term 'Regression Test'
My second session studying Eval. I ran similarity-based evaluation (embedding cosine similarity) myself and got an unexpected result — both OpenAI's text-embedding-3-small and bge-m3 gave the highest score to the 'wrong answer.' This is because similarity captures topical/expressive closeness, not content correctness. Just a difference in markdown formatting can swing similarity scores significantly. In practice, combining similarity with LLM-as-Judge is the standard approach. Also covers 5 principles of test set design, plus a terminology correction: it's not a 'regression test,' it's improvement validation (A/B).
-
Eval Study #1 — Ending 'Vibe Benchmarking' · Accuracy-based + First LLM-as-Judge Implementation
Eval Study #1 — Ending 'Vibe Benchmarking' · Accuracy-based + First LLM-as-Judge Implementation
-
FEMS Project #3 — Streamlit Comparison Dashboard + 3-Backend Question Evaluation (Claude 9 / exaone 7 / gpt 7)
Put a Streamlit dashboard on top of FEMS RAG and threw the same question at 3 backends (exaone3.5:7.8b / claude-opus-4-8 / gpt-4o) simultaneously to compare. Q1 (air compressor anomaly in May) — exaone and gpt said 'no data', while Claude inferred 'weekend-hours anomaly' from summary stats alone without raw data + disclosed its limitations. Q2 (savings measures from a manager's perspective) — Claude 9/10 (incomplete due to token truncation), exaone 7/10 (broken index), gpt 7/10 (concise but hallucinated 'capacitor'). Results of 5-axis scoring.