Designing effective agents

Design the agent’s capabilities

Once you’ve defined the type of agent that works best for your scenario and selected an orchestration model, the next stage is defining what the agent is responsible for and what it will need to complete the job you are assigning it. In this section, you will learn about designing these core capabilities of your agent.

When designing the agent’s capabilities, you should consider the agent’s purpose and scope, the brand voice, and the knowledge and tools the agent needs to access. You must also decide how to structure the responses using a balance of deterministic logic, topics, and generative answers, and how to plan for fallback and escalation.

Define the agent’s scope and purpose

Every design decision should be informed by a clear, user-centric understanding of the agent’s purpose and the business problem you are solving. Revisit the work you did in Chapter 1 to document the business problem, impact, and success metrics, and keep these top of mind as you design your agent.

Start by collaborating with users and internal stakeholders to identify the most common questions, tasks, and challenges they face. Support this discovery process with any available data or reports, such as FAQ documents, reports on the most frequently raised tickets, customer complaints, employee onboarding feedback, or website analytics. Consult with customer service representatives, HR or IT staff, or any frontline worker who interacts with the users of your agent to understand their experience and insights.

Earlier in this chapter you learned how orchestration models work with intents. Understanding the underlying human intent of your users is even more important.

Give the agent your brand voice

Your external agent is a representative of your brand, and you should think of it as a virtual customer-facing team member when you design it. Designing your agent so that it reflects the tone and personality of your brand voice can build trust with your customers and align the agent’s actions with the rest of your external-facing communication. Doing so will help the users feel as if they are interacting with your organization rather than a generic AI chatbot.

Consider the tone (formal, friendly, direct, playful), style (concise, empathetic, detailed), vocabulary (specific industry terminology or abbreviations), and the emotional context of the user. The “personality” of an agent for a fashion retailer and an agent for an insurance company can and should feel very different.

Identify the knowledge sources and tools your agent needs

Once the agent’s purpose is clearly defined, you will be able to identify the knowledge and actions it will need to take to fulfill that purpose effectively. As part of this step, you will need to determine the data sources and tools your agent will use to answer questions and complete tasks.

Start by assessing what knowledge your agent will need to find information and generate answers. Will it use internal documents, SharePoint libraries, public websites, structured data, or connections with your enterprise systems? Do you already have data indexed in Azure AI that you want to use? If you are using unstructured knowledge sources (particularly SharePoint document libraries), consider whether they are well organized and curated to be able to provide consistent and accurate answers. Generative AI doesn’t provide a magic solution to inconsistent, conflicting, or duplicate information. You will find your agent performs much better when it is working with knowledge that is backed by a good content management strategy.

You can add a range of knowledge sources to your agent, including public websites and SharePoint, Dataverse, and enterprise data sources.

Next, define the actions your agent will perform, and the tools it will need to do so. Which business processes does it support or automate? Will it only retrieve data from business systems, or will it also write back to databases, initiate workflows, create content, or send notifications? For each task or action, consider which tools, connectors, integration, and authentication will be required.

Copilot Studio provides a range of tools that you can use for your agent, including prompts, agent flows, custom connectors, REST APIs, the computer use tool, and the Model Context Protocol.

In Copilot Studio, you can add a wide range of tools to extend your agent.

Define the logic strategy

Defining the logic strategy for your agent involves defining which tasks or parts of the process should be handled by generative AI and which should be handled by deterministic logic. Copilot Studio allows you to blend both to create effective and reliable agents.

Generative AI has enabled us to create more comprehensive and powerful conversational AI experiences than were possible in the past. Before it became available, building a traditional chatbot meant designing and authoring topics for every intent you wanted the chatbot to handle, leaving a long tail of user experiences where the chatbot just couldn’t help or had to escalate to a human agent. With generative AI, we can design agents that are grounded in trusted knowledge and have tools to perform actions. These agents can handle more powerful and natural conversations without the need to perform a lot of manual authoring.

However, generative AI can sometimes be unpredictable and risky. Your agent scenario most likely includes critical business processes or information where you need reliable, repeatable tasks run in a specific sequence. Striking the right balance between generative and deterministic logic is key in designing an effective agent.

Generative AI is very powerful for tasks like answering questions, summarizing information, or providing helpful explanations, especially when the user input is unpredictable. However, it can be less effective in scenarios where precision, order, or compliance is critical. In all but the most basic retrieval agents, you should not rely entirely on generative answers, especially in high-risk or high-value scenarios.

Deterministic logic plays an essential role in agent design, because it allows you to build in structured, rule-based flows that can work alongside generative AI. Deterministic agent flows are the right choice for processes that follow strict business rules, or that require a series of specific steps to be executed in a defined order, such as compliance checks, approvals, payments, and password resets.

Understanding when to use generative AI and when to use deterministic logic will result in an agent that can handle a wide range of conversations and user intent, while also providing consistent and trustworthy answers where needed.

Use an agent flow in Copilot Studio to define a series of sequential steps in a task or process.

Identify topics

In addition to using deterministic logic for tasks and process automation, Copilot Studio allows you to design deterministic dialog flows, or topics, that you can use to control business-critical parts of the conversation where you don’t want to rely on generative AI to respond. In Copilot Studio, topics are the core building blocks of an agent that defines the conversational dialog using messages, questions, variables, and branching logic.

Topics are how you can create consistent, predictable responses in the agent conversation. For instance, suppose that when you are building your IT Helpdesk Agent, you discover that users frequently ask, “How do I create a Teams channel?” Your organizational policy is that users cannot create channels on their own, and you have a link to a form for requesting a new channel. Rather than relying on generative answers to find this information, you could author a topic that provides the right information every time along with the link to the request form.

A topic is used to control the response to critical parts of the conversation.

Topics can be triggered by the user input, or by being called from another topic or tool in the agent. Topics can be complex, with branching logic and variables, or they can be just a simple intent and response.

To decide which topics you should build for your agent, start with a few of the most frequently asked questions or tasks that have the highest impact on your success metrics. Look for issues that generate the highest volume of calls, queries, or unnecessary manual work in your business scenario. Think about these tasks from the user’s point of view, reviewing the sources you’ve used in defining the agent’s purpose and scope.

Keep your topic structure manageable and modular. Don’t try to cover every interaction or answer with a topic. Copilot Studio has a limit of 1000 topics, but if you are hitting that limit you are likely creating an overly complex design that will be hard to manage and maintain. Use knowledge sources and generative answers to handle the long tail of less frequently asked questions, and consider creating knowledge sources or using FAQ documents specifically for this purpose. Remember, you can also consider a multi-agent handoff design rather than including everything in a single agent.

As you design topics, look for opportunities to create reusable topics that can work like subroutines to simplify the overall conversational design. These can be small, simple topics with inputs and outputs that are called by other topics. Reusable topics can help with tasks like collecting customer information, capturing feedback, and confirming an action. You can pass variables or context from one topic to another.

You can design a point in the conversation where you redirect the dialog to another topic.

To avoid confusion during runtime, make sure your topics are well defined and distinct from each other, avoiding overlap in the intents, trigger phrases, and descriptions. When mapping out your topics, if you identify intents that are likely to overlap, consider using disambiguation topics. A disambiguation topic is used when multiple topics could be triggered by the user input; it prompts the user to clarify their intent. For example, suppose the user asks, “I need help with leave.” You could create a disambiguation topic that asks whether the user wants to check their leave balance, apply for leave, or understand the leave policy. From there, you can redirect the agent to the appropriate topic.

You can combine deterministic topics with generative answers inside the dialog flow to help focus generative AI on a specific knowledge source. For instance, you can handle the intent “return an item” by building a topic that collects the information required and then uses generative answers within that topic to reason over the return policy to provide the appropriate response.

As noted in Chapter 1, Copilot Studio is a low-code SaaS tool that allows you to quickly and easily iterate and republish your agent without needing a full development cycle. You should review the performance of your agent over time, and identify topics that are ambiguous as well as frequent questions from the users that could be better handled by a topic. Use of agents is a process of continuous iteration and review. Involve the business users and customer service representatives who are subject-matter experts to refine the topic language and descriptions, and to identify new or trending intents that should be added as new topics.

Plan for fallback and escalation

No matter how well you design your agent, there will undoubtedly be situations where it doesn’t understand the intent or can’t find an answer, either because the answer isn’t in the knowledge sources or because the user is asking for something out of scope for the agent’s purpose. Planning for fallback and escalation is therefore an important part of the design process.

In Chapter 1, you created a very basic agent that was connected to a public website and was immediately able to answer questions. This capability was enabled by the system’s Conversational boosting topic, which acts as a fallback to generate answers from knowledge.

In most agent designs, you will use both topics and generative answers as described in the previous section. At runtime, the agent will first check whether the user intent matches any of the authored topics. It will then use those topics to provide answers and perform actions where it finds a match.

You can design the fallback behavior to handle cases where your agent doesn’t find a match in the authored topics, using the following capabilities in Copilot Studio:

• You can enable general knowledge or web search to allow your agent to use the large language model or to search all public websites to generate an answer.

• You can use the Conversational boosting system topic for generative answers. This topic will act as the first fallback for unrecognized intent, automatically applying reasoning across all the knowledge sources you have added to your agent and providing a response.

• If generative answers still can’t answer the user query, the agent will trigger the system’s Fallback topic. By default, this topic prompts the user for clarification, and if necessary, responds with a message stating that it can’t help. You can configure the message, tone, and behavior of the fallback topic to match your scenario.

The system’s Fallback topic sends a message to the user or redirects to the Escalate topic when the intent can’t be determined.

• If the user’s request still isn’t resolved, you can configure the fallback to behave or escalate in the following ways:

  • End the conversation with a helpful message

  • Integrate with an external system by calling another generative AI model using a custom connector

  • Hand off to a live agent using a customer engagement application

• You can also use fallback behavior in your topic design by adding generative answers inside the topic and directing the user to a different topic or escalation option if that fails to provide a useful response.

Ultimately, all these capabilities are geared toward setting your agent up for success, solving the business problem you defined, and providing a reliable and user-friendly experience.