Research & Standards

At a Glance

Everything the collective has shipped, in one place:

Installation and details for each tool are on the Tools page.

Research

Our research is organized around a core question: What does the agent ecosystem need to mature into reliable, long-lived infrastructure for software development?

Current Research Areas

Documentation and information access How agents consume documentation, where they fail, and what site owners can do about it. Standards work to codify best practices. This area produced the Agent-Friendly Documentation Spec and the afdocs audit tool. Active testing is measuring what happens between “agent fetches URL” and “model sees content” across platforms including Claude, Cursor, Gemini, and OpenAI, documenting truncation limits, HTML processing, and content negotiation behavior. A documentation ecosystem report is in progress, evaluating 75+ documentation sites across 9 categories with a three-track evaluation strategy combining automated scoring, real-agent testing, and cross-model comparison.

Tool quality and interoperability Evaluating MCP servers, Agent Skills, and other tool integrations for reliability, spec compliance, and developer experience. This area produced the Agent Skill Report and the skill-validator analysis tool (v1.5.6, stable). The Agent Skill Implementation benchmark empirically tests how platforms actually load, manage, and present Agent Skills, with 40 checks across 10 categories (38 of them fully automated by a purpose-built benchmark runner), 33 benchmark skills, and published platform reports for Antigravity CLI, Claude Code, and Codex CLI. Supporting this work is a purpose-built instrumentation toolchain: agentsummons invokes agent harnesses headlessly, agentminutes parses their session logs into a unified event schema, and skillxp stages Agent Skills on real harnesses and reports what actually reached the model, with transcript evidence instead of model self-reporting.

Ecosystem health signals Developing metrics and methodologies for tracking the overall health of the agent ecosystem over time.

Agent behavior patterns How agents navigate code, use context, handle errors, and interact with external resources in real development workflows.

Standards and advocacy Driving adoption of specifications that make the ecosystem work better for everyone: agents, developers, and platform builders alike.

Standards

Our standards work follows the research: when the evidence shows a pattern that site owners or tool authors can act on, we codify it into an open, testable specification.

Agent-Friendly Documentation Spec

agentdocsspec.com | v0.5.1 (draft)

A specification defining 23 checks across 7 categories that evaluate how well a documentation site serves agent consumers:

• Content Discoverability (7 checks): A discovery index that exists, is valid, fits in a single fetch, and links to markdown, plus embedded directives pointing agents to llms.txt
• Markdown Availability (2): .md URL support and content negotiation via Accept headers
• Page Size (4): Rendering strategy (SPA/CSR detection), markdown and HTML size, content start position
• Content Structure (3): Tabbed content serialization, section header quality, code fence validity
• URL Stability (2): Soft 404 detection and redirect behavior
• Observability (3): llms.txt coverage, markdown/HTML content parity, cache header hygiene
• Authentication (2): Auth gate detection and alternative access paths for gated content

Each check has defined pass/warn/fail criteria, an automation level, and a severity.

Built from empirical observation of agent behavior across hundreds of documentation sites. The spec gives documentation teams concrete, testable criteria for agent-friendliness instead of vague advice. Compliance is measurable with afdocs, and companies are already competing on their scores.

Future Standards Work

As the collective identifies new areas where standardization would benefit the ecosystem, additional specifications will be developed here. Current areas under investigation include:

• Agent tool quality scoring methodology
• MCP server compliance evaluation criteria
• Agent context consumption patterns and limits

Published Reports

• Agent Skill Report: Qualitative analysis of 673+ public Agent Skills, including findings on spec compliance issues across the ecosystem.
• Agent Skill Implementation: Benchmark testing how agent platforms load, manage, and present Agent Skills. 40 checks across 10 categories covering loading, validation, execution, and access control, 38 of them automated by a purpose-built benchmark runner that uses skillxp to stage 33 canary-phrase benchmark skills on real harnesses. Includes platform reports for Antigravity CLI, Claude Code, and Codex CLI, plus authoring guidance based on how the platforms diverge from each other and from the Agent Skills spec.
• Docs Ecosystem Report: In progress. Evaluating 75+ documentation sites across 9 categories (AI providers, DaaS platforms, developer frameworks, enterprise, and more) with automated scoring, real-agent testing, and cross-model comparison.
• MCP Ecosystem Report: In progress. Investigating when MCP servers earn their complexity: abstraction value, dynamic capabilities, CLI alternatives, ecosystem portability, security isolation, performance overhead, developer experience, real-world usage patterns, registry/discovery gaps, and lifecycle trust signals.

Published Articles

aeshift.com publishes commentary on the agent ecosystem, generated through our automated pipeline and reviewed before publication. Topics range from agent sandbox escapes to behavioral evaluation methodology. Additional articles on dacharycarey.com cover:

• Agent-friendly documentation: what it means and why it matters
• The distinction between LLM training data and agent consumption
• Practical implementation guides for documentation teams and software development organizations

Automated Research Infrastructure

Our daily intelligence pipeline is a key differentiator. It runs autonomously across four stages on self-hosted infrastructure:

1. news-gather (06:00 UTC): Scans RSS feeds, arXiv preprints, and GitHub releases for agent ecosystem developments. Deduplicates against MongoDB Atlas and uses Claude Sonnet to tag and route items to downstream pipelines.
2. research-sourcing (06:20 UTC): Evaluates items for research potential, identifies connections via Voyage AI vector search, and tracks theme evolution across lifecycle stages (emerging, active, mature, fading, dormant).
3. shift-sourcing (06:40 UTC): Selects top items for commentary, searches existing coverage, generates parallel drafts with Claude Opus and GPT-5.2, then fact-checks and edits for style. Opens a PR on the aeshift repo with publication-ready drafts.
4. dailies dashboard (07:20 UTC): Synthesizes pipeline outputs, API usage, research themes, and task status into an operational dashboard for the team.

This infrastructure means we catch developments as they happen, not weeks later. The pipeline continuously tracks research themes to inform priorities and publishes commentary on aeshift.com.

Research Roadmap

Our current work in documentation and tool quality is the foundation. As the research matures, we plan to expand along three directions:

• Broader ecosystem reports. The agent ecosystem extends well beyond documentation and skills. We’re laying groundwork for reports covering code generation quality, agent platform comparisons, tooling developer experience, and agent performance benchmarks.
• Additional specifications. The Agent-Friendly Documentation Spec addresses one surface where agents interact with the web. We see similar opportunities for agent-friendly standards in other areas: web content more broadly, tool definition formats, and output quality guidelines.
• Registry and scoring infrastructure. We’re working toward infrastructure that provides quality signals, interoperability metadata, and trust indicators across agent tools, servers, and integrations.