Copilot for Microsoft 365: What we’ve learned after one year – Q&A

Recently, we hosted a webinar with two amazing Microsoft MVPs, Zoe Wilson, and Luke Evans, who presented Copilot for Microsoft 365: What We’ve Learned After One Year. I highly encourage you to watch the recording below. We had many interesting questions, and we didn’t get to all of them during the webinar, so I wanted to create a blog post in which we answered all of them.

To make it easier for everyone to follow, we have combined certain questions together as they covered a very similar topic, as well as split this blog down in multiple sections based on interests. Let’s get started!

Copilot for Microsoft 365: Adoption & ROI Questions

With Copilot for Microsoft 365 being a significant investment (30$ USD / user / month), of course many attendees had questions about measuring the return on investment on their Copilot licenses. Let’s cover some of them!

Q: How can we identify the users that use (or do not use) their Copilot for Microsoft 365 license, and figure out time saved with Copilot?

Microsoft offers multiple reports to help organizations figure out if Copilot is paying off. First of all, Microsoft offers a Copilot for Microsoft 365 Usage report in the admin center, but you do need to have the right role to access it. If you don’t have access to it, here are some screenshots from my demo tenant where I only have one user licensed.

You can see from this screenshot that while I have been active in Copilot, it hasn’t been in every single application, and I didn’t use Copilot in Excel, Word, or Outlook on this account in the report period.

You can also view details on the actual users, but, by default their usernames and display names are concealed. Administrators do have the option of showing the username and display name in those reports, but I would encourage checking your legal and compliance requirements before turning that setting on.

Our second way to measure ROI is by using Viva Insights, but only if you have the right licenses. In order to access all the features you need to have either:

• 50+ Copilot for Microsoft 365 licenses OR
• 10+ Viva Insights premium licenses

Without those licenses, you do have access to some extra information such as usage per app to summarize, or create content for example, but not the full features.

Where the Microsoft Copilot Dashboard is very useful is the Impact page, shown below (picture credit: Tech Community) which shows the key information you need to prove Copilot ROI such as:

• Copilot assisted hours
• Copilot actions taken
• # of Meetings & Meeting hours summarized by Copilot
• # of E-mails created / summarized by Copilot
• # of Documents created / edited / or summarized by Copilot
• & more!

Now as you can see from the screenshot below, 2,227 users saved almost 14,000 hours over the last 28 days. This gives you a good number to calculate ROI in your organization depending of course on user salary or any other metric you want to measure.

Something else that is interesting at the bottom of that image is you can see what your own company users think about Copilot (employee sentiment) and how it measures against Microsoft’s research metrics. If your users do not answer surveys, you might end up like in my demo environment here with not enough data.

Q: Is there a way to get those analytics outside of Microsoft 365 (via PowerShell, Power Automate, or API) in order to build our own reports?

You can export some data from the Microsoft 365 Usage reports to Excel as outlined in this article and you can create custom reports using information such as:

• Username (if you show it)
• Last activity date of that user in Copilot

Per Microsoft, a user is considered active in a given app if they performed an intentional action for an AI-powered capability. For example, if a user selects the Copilot icon in the Word ribbon to open the Copilot chat pane, this does not count towards active usage. However, if the user interacts with the chat pane by submitting a prompt, this action would count towards active usage.

Unfortunately, it seems the Microsoft 365 Usage Analytics app in Power BI does not yet have any Copilot data that you could customize.

Q: Based on your experience, what would you do differently in terms of Copilot for Microsoft 365 deployment or adoption compared to a year ago? What was your main lesson learned?

One key lesson we learned was that focusing on use cases for different departments such as finance, HR, or marketing didn’t work as well as we initially thought. Everyone wanted to know specific use cases, but we quickly realized that this approach slowed things down. Instead, the most effective strategy was to get Copilot licenses directly into the hands of users in these teams and help them understand how to use it. From there, we worked with them to identify and capture business value organically. This pivot allowed us to accelerate the adoption process significantly.

On the technical side, the biggest lesson was around data governance. Initially, we didn’t fully grasp the critical importance of understanding key technical areas related to Copilot for Microsoft 365. Going forward, we would focus much earlier on establishing strong data governance practices as part of the deployment strategy. This would ensure that the technical foundation is solid from the start, avoiding issues later.

Microsoft 365 Copilot readiness questions

Q: What would be your approach if a business asked to turn on Copilot immediately for an organization with over-permissioned identities, over-shared resources, and no data classification, ownership, or lifecycle management in place?

I would suggest starting cautiously by implementing a restricted SharePoint search as a temporary measure. It’s a quick fix, but it allows you to limit what Copilot for Microsoft 365 can access while you work on cleaning up the underlying issues. Begin with an allow-list approach, where you grant Copilot access only to specific, well-understood sites. The SharePoint Admin Center offers reports that can help identify popular sites and sharing behaviors, which can inform your allow list.

For data classification, start small. You don’t need to overcomplicate things. Implement one or two labels, focusing on tagging content without jumping straight into encryption or strict security measures. Over time, you can scale up your classification efforts. If you have access to Microsoft E5 features, consider using trainable classifiers to automate the process, allowing the system to learn and categorize documents for you. Automatic sensitivity labeling and rule-based logic are also options, but they can be more complex.

On the access management side, I recommend using tools like Entra ID Governance Suite, or of course, Syskit Point to review and manage permissions and assess the Copilot readiness of your tenant. This helps bring a more structured approach to identity and access control.

Q: What is required for the successful adoption of Copilot in an organization with 10K Microsoft 365 users, where 25% are expected to start using it? Should we manage adoption internally or bring in external consultants? What skills and resources are needed?

The most important factor in successfully adopting Copilot for Microsoft 365 is not necessarily whether you use internal teams or external consultants, but whether your organization has the right skills in place. You need to evaluate your existing capabilities in managing technology rollouts. Does your team have experience in change management? If not, external consultants might be necessary to help drive the process.

Adoption requires a structured, programmatic approach to change management. Start before users even get their licenses, ensuring clear communication and education. Organize sessions, create resources, and set expectations. A Champions Network is key—identifying super users from different departments who can promote and model Copilot usage. Additionally, tools like prompt libraries and examples of early wins help foster a community of collaboration and knowledge sharing.

Sustained effort is crucial. Simply training users once isn’t enough; people tend to fall back into old habits. Adoption is a lot like training for a marathon; you have to keep reinforcing the message. Regular reminders, updates, and opportunities for users to re-engage with Copilot are necessary to keep momentum. Furthermore, Copilot for Microsoft 365 is constantly evolving, so adoption isn’t just about teaching users the current features, but also encouraging a mindset of experimentation as new capabilities roll out.

Q: When you say that we should archive old content that is sitting on a SharePoint site, what exactly does this mean in practical terms?

At the end of the day, you want Copilot for Microsoft 365 to be trained only on quality data. The more irrelevant data you have in your tenant that users have access to, the more Copilot can learn from outdated information and, therefore, provide lower-quality results.

Archiving your data will depend on many different factors such as your compliance requirements, but your goal needs to be to get it out of the Copilot index and that can be done using tools such as Microsoft 365 Archive (read our in-depth blog post over here: Can you save money with Microsoft 365 Archive?), you could also simply remove all user permissions to the SharePoint site except for one archive user, or use Syskit’s archiving functionality.

At the end of the day, remember that storage management in SharePoint Online is important. Not only can it cost you extra every month, but keeping the data without a proper lifecycle in place can make Copilot for Microsoft 365 give lower-quality results.

Copilot for Microsoft 365 Security & Compliance questions

Q: If user A has delegated access to user B’s mailbox, can Microsoft 365 Copilot in Outlook access that data and give answers about it?

We have confirmed that the answer is no from this Microsoft Support Article that states: Copilot scenarios in Outlook are only available on a user’s primary mailbox. They are not available on a user’s archive mailbox, group mailboxes, or shared and delegate mailboxes that they have access to.

Q: How can I apply my required retention policy for data in Copilot for Microsoft 365?

In order to apply retention polices for Copilot for Microsoft 365 interactions, you need to use the retention policy location named Teams chats and Copilot interactions because they are retained and deleted by using the same mechanisms. Check out this Microsoft page to learn about retention for Copilot for Microsoft 365.

Copilot for Microsoft 365 features questions

Q: What is the difference between Microsoft Copilot that is included with every Entra ID license and Copilot for Microsoft 365 that costs 30$ per user, per month?

Microsoft Copilot (Available today at https://copilot.microsoft.com/ ) but moving soon to https://copilot.cloud.microsoft for enterprise accounts is your assistant on the Web.  It’s trained on publicly available data on the web and uses AI to answer questions and create content. Copilot for Microsoft 365 is a collection of over 20 copilots integrated in your every day apps (Word, Excel, Teams, Outlook, etc.) which in addition to the web, is also trained on your enterprise data.  This ensures that the answers are more accurate as they are trained on the best possible data, your data!

Conclusion

This is it for now, we truly hope you enjoyed the webinar and this Q&A follow-up. For more info, take a look at how Syskit Point can check the Copilot readiness of your Microsoft 365 tenant.