Product Evals (for AI Applications) in Three Simple Steps

Eugene Yan wrote a GREAT blogpost on how to build product evals (for AI applications) in three simple steps: https://eugeneyan.com/writing/product-evals/

These three steps are:
1. Label data
2. Align LLM as judge evaluator
3. Use this on all changes

LangSmith is great for building evals as it supports all three of these steps through:
1. Annotation Queues
2. Align Evals flow
3. Experiments

See more docs on how to do this in LangSmith here: https://docs.langchain.com/langsmith/improve-judge-evaluator-feedback