If you are building agents with the new Copilot Studio GitHub Copilot harness and wondering how to configure Application Insights, here is the current option. Configure it at environment level. The GitHub Copilot harness does not support agent-level configuration at the time of writing this post.
If you are new to Application Insights, refer to my earlier Copilot Studio Application Insights post. To understand the new harness, refer to Understanding the GitHub Copilot harness.
What This Does
- Configures Application Insights for a Copilot Studio environment.
- Supports agents powered by the standard and GitHub Copilot harnesses.
- Exports agent, tool, and output traces to Application Insights.
Note: Environment-level Copilot Studio telemetry is currently in preview. It is available only for Managed Environments and might still be rolling out.
Lets configure the export from the Power Platform admin center.
Configure the Application Insights Data Export
- Sign in to the Power Platform admin center.
- Select Manage > Data export from the left navigation.
- Open the App Insights tab.
- Select New data export.
- Enter a name for the export package.
- Select Copilot Studio (preview) as the data type.
- Select Next.
- Select the Managed Environment that contains your GitHub Copilot harness agent.
- Select Next.
Note: Only Managed Environments are available for this export.
- Select the Azure subscription.
- Select the resource group.
- Select the Application Insights resource.
- Select Next.
The Application Insights resource must already exist. My earlier Application Insights post covers how to create it.
- Confirm that Export data shows Copilot Studio.
- Verify the selected environment and Application Insights resource.
- Confirm that Export data shows Copilot Studio.
- Verify the selected environment and Application Insights resource.
Now we have the environment-level export connected. Lets run the agent and verify its telemetry.
Run the Agent and Check the Logs
Querying Application Insights was covered in my earlier Application Insights post. That post uses customEvents for agent-level telemetry. For this environment-level export, use the dependencies query below.
- Open the GitHub Copilot harness agent in Copilot Studio.
- Start a test conversation.
- Ask the agent to complete a task that calls one or more tools.
- Open the Application Insights resource in the Azure portal.
- Select Monitoring > Logs.
New export configurations can take up to 24 hours to send telemetry. Run the following KQL query after the data arrives:
dependencies| where timestamp > ago(24h)| where name in ("InvokeAgent", "ExecuteTool", "OutputMessages")| project timestamp, name, success, resultCode, duration, operation_Id, customDimensions| order by timestamp desc
Environment-level telemetry is written mainly to the dependencies table. It includes InvokeAgent, ExecuteTool, and OutputMessages records.
Summary
Agent-level Application Insights is available only for agents powered by the standard harness. GitHub Copilot harness agents use the environment-level export configured from PPAC.
For more details, refer to Microsoft documentation for Copilot Studio telemetry scopes, environment-level telemetry, and Application Insights data export.
You can also refer to my earlier posts about Application Insights integration and the GitHub Copilot harness for the related setup and concepts.
🙂
Leave a Reply