AgentOps

What it does

AgentOps is a developer platform for tracing, debugging and deploying AI agents and LLM apps. It records LLM calls, tool use and multi-agent interactions as sessions in a dashboard. Tools include time travel debugging that rewinds and replays agent runs, log and error trails, token counts and cost tracking. Integrations cover OpenAI, CrewAI, AutoGen and other frameworks, and it is available for Python and TypeScript. Installation starts with pip install agentops. The free tier covers up to 5,000 events, Pro starts at $40 per month, and Enterprise is custom. Documentation and dashboard signup are on the official site.

The platform is described as an agent agnostic SDK with native integrations for agent frameworks and 400+ LLMs. A session drilldown and waterfall view show LLM calls, action events, tool calls and errors, including the prompt and completion for a given call. A session overview gives a meta-analysis of past sessions. Fine-tuning is listed as an option for specialized LLMs on saved completions. The Pro plan adds unlimited event limit, unlimited log retention, session and event export, dedicated Slack and email support, and role-based permissioning. Enterprise adds SLA, Slack Connect, custom SSO, on-premise deployment, custom data retention, self-hosting on AWS, GCP or Azure, and SOC-2, HIPAA and NIST AI RMF coverage. The app is open source and the docs include self-hosting guides.