I use an OpenAI model through an API alias, and the same prompt sometimes seems to produce a noticeably different answer weeks later. I’m not sure whether that’s a model update, normal variation, or a change elsewhere in my app.

For anyone tracking this in production, what’s a practical way to catch meaningful behavior changes without treating every wording difference as a regression? I’m considering a small set of fixed regression prompts, but I’m unsure how often to run them or what to compare when model snapshot changes are involved.