Evidently is an open-source framework for evaluating, testing and monitoring ML and LLM-powered systems, from experiments through to production. It handles tabular and text data, RAG applications and AI agents, and includes more than 100 built-in metrics, from data drift detection to LLM-based judges. Reports summarise evaluations, and test suites add pass/fail conditions, for example for regression testing in CI/CD. On top of the library, the Evidently Platform adds a user interface, tracing, synthetic data, dashboards and alerting, available as Evidently Cloud.
The library is licensed under Apache 2.0 and runs locally: after installing with pip or conda, you start the monitoring UI on your own machine or server, so data stays within your own environment. The project documentation says that enterprise users can choose between a cloud and a self-hosted option. Evidently Cloud is a managed service operated by the company Evidently AI, so using it means a third-party provider processes your data. For European organisations that want to retain control, the self-hosted open-source route is the most direct option.