I’m considering fine-tuning an OpenAI model to make routine customer replies sound more like our team. We have a few hundred examples, but they cover a fairly narrow set of situations.

Has anyone had good results with house-style fine-tuning using a modest dataset? I’m trying to work out how to tell whether the model has learned the tone itself or is just memorizing familiar examples. What did you use to evaluate it on requests that weren’t in the training set?