I have a document search feature using OpenAI embeddings, and I want to try a different embedding model without taking search offline. The index contains several hundred thousand text chunks, so rebuilding everything would be a noticeable job.

What I’m unsure about is compatibility. If the old and new models produce vectors with the same dimensions, does that make them suitable for the same similarity search, or could they still represent text in incompatible ways?

The migration I’m considering

• Keep the existing index serving production searches.
• Generate new embeddings for a copy of the same chunks.
• Run a set of real search queries against both indexes before switching traffic.

Would this be a sensible approach to embedding model migration? I’d also like to understand whether a gradual rollout can mix old and new document vectors, or whether each query needs to stay entirely within one model’s index.

For anyone who has done this, what did you measure besides whether the expected document appeared first? I’m particularly interested in semantic search relevance testing for short, ambiguous queries.