Work03
Agentic Fleet
A fleet of domain-specialised AI agents over Microsoft 365. The company's first internal AI architecture, built with an evaluation method designed to prove whether it works.
- Built with
- Microsoft Copilot Studio, Power Automate, Dataverse, SharePoint, Entra ID
The problem
The project sales division of a German architectural glazing manufacturer runs construction projects across four disconnected systems: opportunities in the CRM, order documents in SAP, execution data in a project portal, and files in per-project document sites. Answering one question, a project's status and what had been invoiced, meant opening all four and knowing which of them holds which fact. That knowledge was tacit, held by long-tenured staff. Existing Copilot licences searched documents without domain understanding, and the company's internal RAG system had no access to live transactional data.
The system
A domain-scoped agent over five deterministic workflows and one document library. Asked for the invoices on a project, it ranks projects in the portal, resolves the opportunity family in the CRM, locates the project's document site, and returns linked invoices. The difficulty is not retrieval but the encoded domain knowledge: which system is authoritative for which fact and silent on the rest; that an empty status means “not yet in execution, lost, or old” and must be reported as an interpretation rather than a fact; that an invoice carries four different figures, and quoting the wrong one produces a confident, plausible, wrong number a salesperson might repeat to a customer.
How it is built
Three layers. A core instruction set carries global behaviour (grounding, source boundaries, output format, pre-send verification) while procedural skills load only when triggered, keeping the always-on context small and each procedure independently testable. Five Power Automate workflows over Dataverse and SharePoint do the retrieval: the agent picks the tool and the parameters, but the query itself is fixed, so results are reproducible. An orchestrator routes to domain specialists, with a REST bridge to the existing internal RAG system so deep product knowledge stays in one place. Every query runs as the asking user, so SharePoint, Dataverse and Entra ID trimming applies automatically, and no document needed a new label. Every response in a structured evaluation set is scored on three independent axes: factual correctness, correct source, and rule compliance, because a right answer from the wrong source is a latent failure. Each failure category maps to a different fix, and every prompt change is measured against the baseline before it ships.
- 1
Question in Teams
A salesperson asks in natural language, in Teams or Outlook.
- 2
Orchestrator
Routes to the domain specialist that owns the question; carries the shared company knowledge layer.
- 3
Project-sales agent
Core instructions plus procedural skills that load only when triggered, keeping the always-on context small.
- 4
Deterministic workflows
Five Power Automate flows over Dataverse and SharePoint. The agent picks the tool and parameters; the query itself is fixed, so results are reproducible.
- 5
Grounded answer
Linked documents, sources named, and every query run as the asking user so existing permissions apply unchanged.