Hackathon-winning retrieval with small open models.
A four-hour build for the BCU AI Hackathon 2026. Answer 100 multiple-choice questions using only open models of 8 billion parameters or fewer, grounded in Wikipedia, with confidence scores and repeatable runs.
- Sector
- Knowledge retrieval
- Method
- RAG with small open models
- When
- Jun 2026
- Context
- BCU AI Hackathon 2026: winning team
- Role
- Built the team's retrieval pipeline
Key figures
- 100% on a hand-checked 20-question set
- 8B largest model allowed
- 2 full runs, byte-identical
How it works
- Query : Entity-focused search terms from the question
- Retrieve : Wikipedia first, then the web; cached for repeat runs
- Assemble : The top passages within a length budget, reranked
- Answer : Qwen3-8B at temperature 0, with and without evidence
- Score : Confidence from parsing, evidence, support and agreement
- Verify : Low-confidence answers checked by hand against sources
Architecture and results
- A model runs here
- Deterministic code
- An outside source
- A stored artefact
- A boundary the design refuses
-
Find the evidence, answer twice, then score
Each question becomes an entity-focused search. Wikipedia passages, with the web as a fallback, are cached so repeat runs match, reranked within a length budget, and given to Qwen3-8B, which answers at temperature 0 with and without the evidence. Parsing, support and agreement set the confidence, and low-confidence answers were checked by hand.
Open full size: Find the evidence, answer twice, then score -
Retrieval on a hand-checked sample
On 20 questions whose answers were verified by hand, the model alone scored 90%, and 100% with Wikipedia evidence. Twenty questions show that retrieval helps; they do not measure accuracy on all 100.
The question
At the BCU AI Hackathon 2026, teams had four hours to answer 100 multiple-choice questions using only open models of 8 billion parameters or fewer. The questions were phrased “according to Wikipedia”, so grounding the answers in Wikipedia was the biggest lever. I built the team’s retrieval pipeline.
What it does
Each question becomes an entity-focused search. Wikipedia passages, with web results as a fallback, are cached, assembled within a length budget and reranked. Qwen3-8B answers at temperature 0, with and without the evidence, on one 8 GB laptop GPU. A confidence score combines whether the answer parsed, whether the evidence supports it, and whether the two answers agree.
Results
On a hand-verified set of 20 questions, the model alone scored 90%, and 100% with Wikipedia evidence. Two full runs of the pipeline were byte-identical. Low-confidence and disputed answers were checked by hand against Wikipedia, with a source for each. One question turned out to be unanswerable, because all five options named the same village, so it was answered “Unknown”, as the rules allowed.
Limits
- Twenty questions is a small check: 100% on them shows that retrieval helps, not the accuracy on all 100.
- The submitted answers combine the pipeline, the team’s research and sourced manual checks, so they are not the pipeline’s output alone.