--- slug: "agent-observability-and-evals-tools" title: "Agent observability and eval tools" description: "Langfuse, LangSmith, Arize Phoenix, Opik, DeepEval, Promptfoo and Inspect: what each traces or tests, whether you can self-host it, license and install." published: "2026-10-15" updated: "2026-10-15" signals_fetched_at: "2026-10-15T05:02:49.930Z" tools: ["arize-phoenix", "opik", "langfuse", "langsmith", "promptfoo", "deepeval", "inspect-ai"] page: "https://agenttooldrop.com/guides/agent-observability-and-evals-tools" --- # Agent observability and eval tools Updated 2026-10-15. Page: https://agenttooldrop.com/guides/agent-observability-and-evals-tools Tools from our catalog for seeing what an agent did and checking whether it was right: platforms that trace each LLM call and tool step and run evals on those traces, test frameworks that run evals in CI, and a framework for benchmark-style task suites. The table shows what each covers, whether you can self-host it, and the license. Several of these pair an open-source part with a hosted product. The self-hosting and license columns say where that line is, as each project documents it. ## At a glance | Tool | Covers | Self-hosting | License | Pricing | Stars | Last push | | --- | --- | --- | --- | --- | --- | --- | | [Arize Phoenix](https://agenttooldrop.com/tools/arize-phoenix) | OpenTelemetry traces, evals, datasets, experiments, prompt playground | Locally, in a container or on Kubernetes | Elastic License 2.0 | Free | 11,840 | 2026-10-15 | | [Opik](https://agenttooldrop.com/tools/opik) | Traces, datasets, experiments, LLM-as-a-judge metrics, monitoring | Docker Compose or Helm; or Comet's cloud | Apache-2.0 | Open source | 22,615 | 2026-10-15 | | [Langfuse](https://agenttooldrop.com/tools/langfuse) | Traces, prompt management, datasets, evals | Docker Compose, a VM or Kubernetes; or Langfuse Cloud | MIT | Open source | 35,838 | 2026-10-15 | | [LangSmith](https://agenttooldrop.com/tools/langsmith) | Tracing, evals and monitoring on a hosted platform | Enterprise plan only | MIT | Free tier + paid | 1,079 | 2026-10-15 | | [Promptfoo](https://agenttooldrop.com/tools/promptfoo) | Evals, side-by-side provider comparison, red teaming, PR scanning | Runs locally | MIT | Open source | 26,000 | 2026-10-15 | | [DeepEval](https://agenttooldrop.com/tools/deepeval) | Pytest-style tests with LLM-as-a-judge and other metrics | Runs locally; shared reports need a Confident AI account | Apache-2.0 | Open source | 18,875 | 2026-10-15 | | [Inspect](https://agenttooldrop.com/tools/inspect-ai) | Task suites from datasets, solvers and scorers; 200+ pre-built evals | Runs locally; sandboxes in Docker, Kubernetes, Modal and others | MIT | Open source | 3,005 | 2026-10-15 | ## Tracing, with evals on the same data ### Arize Phoenix Traces LLM and agent apps over OpenTelemetry and runs evals, datasets and experiments in a local UI Pick it if you want an observability and eval UI on your own machine, instrumented with OpenTelemetry. The Elastic License 2.0 is not OSI open source. License: Elastic License 2.0 (source-available). Pricing: Free. GitHub: 11,840 stars, last push 2026-10-15. Skip it if: You need an OSI open source license or want to offer it as a hosted service; the Elastic License 2.0 forbids that. For a managed platform Arize points to its separate Arize AX product. Install with pip: ```sh pip install arize-phoenix phoenix serve ``` Install source: https://github.com/Arize-ai/phoenix#readme. Entry: https://agenttooldrop.com/tools/arize-phoenix.md ### Opik Traces agent runs and LLM calls, scores them with LLM-as-a-judge metrics and monitors production Pick it if you want tracing, evaluation and production monitoring under one Apache-2.0 license you can self-host. License: Apache-2.0. Pricing: Open source. GitHub: 22,615 stars, last push 2026-10-15. Skip it if: You only need lightweight offline tests; self-hosting runs a multi-container stack with databases and caches, and the alternative is a Comet account. Install the Python SDK: ```sh pip install opik opik configure ``` Install source: https://github.com/comet-ml/opik#readme. Entry: https://agenttooldrop.com/tools/opik.md ### Langfuse Traces LLM calls and agent steps, then runs evals, datasets and prompt versioning on top of them Pick it if you need to see what your app did on each request and iterate on prompts against the same traces, self-hosted or in the cloud. License: MIT; ee/ directories under a separate enterprise license. Pricing: Open source. GitHub: 35,838 stars, last push 2026-10-15. Skip it if: You do not want to run a server stack yourself and cannot use a hosted account; self-hosting means running the Docker Compose or Kubernetes deployment. Code under the ee/ directories is not MIT licensed. Install the Python SDK: ```sh pip install langfuse ``` Install source: https://langfuse.com/docs/observability/sdk/overview. Entry: https://agenttooldrop.com/tools/langfuse.md ### LangSmith Traces, evaluates and monitors LLM apps and agents on LangChain's hosted platform via Python and JS SDKs Pick it if you want managed tracing and evals with no servers to run, especially if your app already uses LangChain. License: MIT (client SDKs); the LangSmith platform is a commercial hosted service. Pricing: Free tier + paid. GitHub: 1,079 stars, last push 2026-10-15. Skip it if: You want to self-host without an enterprise contract; self-hosted and hybrid deployment are Enterprise plan options. Usage beyond the included traces is billed pay-as-you-go. Install the Python SDK: ```sh pip install -U langsmith export LANGSMITH_TRACING=true export LANGSMITH_API_KEY=ls_... ``` Install source: https://github.com/langchain-ai/langsmith-sdk#readme. Entry: https://agenttooldrop.com/tools/langsmith.md ## Evals as tests in CI ### Promptfoo Runs evals and red-team scans against prompts, models and agents from a CLI, with a web results viewer Pick it if you want repeatable local eval runs and red-team scans that can gate changes in CI. License: MIT. Pricing: Open source. GitHub: 26,000 stars, last push 2026-10-15. Skip it if: You need tracing and monitoring of live production traffic; promptfoo is built around evals, red teaming and code scanning rather than observability. Install with npm: ```sh npm install -g promptfoo promptfoo init --example getting-started ``` Install source: https://github.com/promptfoo/promptfoo#readme. Entry: https://agenttooldrop.com/tools/promptfoo.md ### DeepEval Unit tests LLM apps and agents in a Pytest-style workflow with LLM-as-a-judge and other metrics Pick it if you want LLM quality checks written as tests that run locally or in any CI environment. License: Apache-2.0. Pricing: Open source. GitHub: 18,875 stars, last push 2026-10-15. Skip it if: You are not working in Python (it requires Python 3.9+). Shared reports, iteration comparisons and production monitoring live in the separate Confident AI platform, which needs an account. Install with pip: ```sh pip install -U deepeval ``` Install source: https://github.com/confident-ai/deepeval#readme. Entry: https://agenttooldrop.com/tools/deepeval.md ## Benchmark-style evaluation ### Inspect Defines and runs model and agent evaluations in Python from datasets, solvers and scorers Pick it if you want to measure a model or agent on a task suite, including tool use and sandboxed code, and compare providers. License: MIT. Pricing: Open source. GitHub: 3,005 stars, last push 2026-10-15. Skip it if: You need tracing of a live production app; Inspect is an evaluation framework, and evals are written in Python. Install with pip: ```sh pip install inspect-ai ``` Install source: https://inspect.aisi.org.uk/. Entry: https://agenttooldrop.com/tools/inspect-ai.md ## Newer options worth a look New and rising tools from the catalog that this guide does not compare yet, by new and rising score. Younger and less proven than the picks above. - [Reef](https://agenttooldrop.com/tools/reef.md): Turns feedback on an agent's live requests into versioned updates to its model weights or its harness. Skip it if: You want a stable agent today: harness work needs a model endpoint, representative tasks and an evaluator, weight training needs a supported GPU stack, and it is at v0.1.3 with the repo created on 2026-08-31. The harness install pipes a script into bash and the pi adapter runs commands with no sandbox, so keep the install root outside the project the agent works in. - [SkillSpector](https://agenttooldrop.com/tools/skillspector.md): Scans agent skills for prompt injection, data exfiltration and supply-chain risks before you install them. Skip it if: You only use skills you wrote yourself. The optional LLM analysis sends skill content to the configured provider with your API key; --no-llm runs static checks only. The README notes the stdio MCP transport still has an initialize hang reported in issue #199. ## How we chose Picked from the 95 tools in our catalog, each verified against its own repository and docs; 7 made this guide. Within each group, new and rising tools come first, sorted by the new and rising score (star velocity, recent pushes and releases, repo age and an editorial weight), then the standards, the well-known tools, by name. It is not a benchmark. The number beside a tool is its rank in the full new and rising ledger, not within this guide; standards are not ranked. Install commands are copied from each project's docs; every tool page links to the page they came from. GitHub signals (stars, last push, latest release) were fetched on 2026-10-15. Frameworks with tracing built in, such as the OpenAI Agents SDK, are covered in AI agent frameworks compared. The MCP Inspector tests MCP servers rather than agents and is left out.