getting a chatbot to stay on topic

in summer 2024 i did a co-op with RIT dubai career services, building a chatbot on llama 3.0. one requirement shaped the whole project: it should answer questions about careers, CVs, and interviews, and refuse everything else. a university assistant that improvises on random topics is a liability, not a feature.

the refusing turned out to be harder than the answering. a system prompt alone doesn't hold up, students are creative. what held up was layers: restrict what content the model can see to career-services material, check the incoming question before it reaches the model, and make the refusal messages polite enough that getting declined doesn't feel like an error.

the other half of the job was everything around the model. the university provided cloud GPU compute, and i set up and ran inference on it. i also built a management interface with next.js and fastapi so staff could monitor conversations and step in when needed. handing a black box to a non-technical team doesn't end well, and they were the ones who had to live with it after my co-op ended.

looking back, the model was a small part of the system. most of the work, and most of what made it usable, was the guardrails and the admin tooling around it.