Copilot interview with Tony Redmond: Beautiful results come from curated data

While at the Microsoft 365 Community Conference in Orlando, our very own Frane Borozan sat down with Microsoft MVP Tony Redmond to talk about all things Microsoft Copilot.

Tony Redmond is a popular international speaker, writer, and consultant in the world of Microsoft 365 and is well known for his eBook Office 365 for IT Pros. Frane and Tony discussed the pros and cons of Microsoft Copilot as well as Tony’s session Don’t let Copilot be a Vanity Project. You can watch the entire interview below or read our summarized blog post.

Is Microsoft Copilot hype or not?

It depends on how you define a hype. Copilot still is not generating any huge amount of revenue. The target market for Copilot is 400 million users. That’s the Office 365 install base. How many people have Copilot today? And your answer is pretty few.

So, if you compare the impact this has had versus all of the marketing efforts that Microsoft has put around it, you got to say it’s hype. On the other hand, you could say, well, is the introduction of a really revolutionary technology worthy of putting all this marketing effort around? And it might seem like hype, but it’s really just communication about what it can do.

And I guess I think we’re in the middle of those two states. There’s probably too much marketing hype, but it is an important technology. It could change the way we do things. So, I’ll go in the middle to say half and half.

Does Copilot help us understand more content?

I’m not sure Copilot helps you understand too much content. It does a good job of summarizing. My favorite Copilot feature is, summarizing Teams meetings. I think it’s very good at that.

You can summarize Word documents, and it does okay at that. It is okay at summarizing emails, but I’m not sure whether I learn that much from it because I’d prefer myself. But it’s my personal belief of going and looking at documents and reading through and finding the nuances.

Because I’ve been a writer and author for so many years, I probably do a better job than Copilot. At least, I hope I do, but it could be totally different for other people. Everybody’s got to find their own Copilot magic if you like.

Will it help people make better use of the information that they have? Possibly, provided that they take care of that information. I think there’s an awful lot of what I call “digital rot” inside Microsoft 365 today, old stuff that’s just there hanging around. And the horrible thing is, is that if a search can find that digital rot, then Copilot can find it. And not only can it find it, but it can also consume it. And that’s the downside of Copilot.

Why is governance important for Microsoft Copilot?

We have certain tools that are available to us, like retention policies. Retention policies are a bit of a blunt tool. It says go find all documents that meet these criteria, let’s say age criteria, and remove them. And you could say, well, that will get rid of all my digital debris, but it doesn’t really because it doesn’t put any value on the documents that are removed. Users put value on the documents by putting retention labels on them. But I’m not sure the users are very good at that type of process either.

So the overall conclusion I’ve seen, especially in OneDrive accounts, my gosh, OneDrive has just become the dumping ground of every Microsoft 365 app. And you end up with this five-terabyte collection of stuff that may or may not have been valuable at one point at a time, and then we get to a point where Copilot starts consuming it. And some of the stuff that you may not think is too good turns up in output, and that’s not great. That’s a challenge I think large organizations face with the implementation Copilot.

Copilot, remember, is just a word-computer. It’s just taking in words. And assembling words into a form that makes sense to it and putting it out in response. So sometimes it gets those words that it’s going to use for its responses from very odd places.

What’s the best way to ask Microsoft Copilot a question?

You need a well-written prompt. You need grounding. So you give your own documents to ground the prompt, and you need to pay a lot of attention to what comes back. Because odd stuff does happen. Now, if you pay attention to your prompts, you write them very precisely, you ground the problems with good documents, and you’ll get value out of it.

But if you go up to Copilot, you say, “Oh, this is magic. AI is magic. It will solve all problems.” You just put in a question. You’re quite likely to get the same type of response as if you would stop somebody that you didn’t know on the street and say, “What did you think of X?” Gosh knows what you get back from that person. You might get a great response, but you might get rubbish.

Can you tell us about your Microsoft Copilot session?

I mean, generally, the session was all about asking people to do one thing, when they’re looking at Copilot, that’s engaging the best organ in their body. That’s their brain. Instead of listening to all the hype, instead of listening to all the marketing, start looking at what the technology could do for them in the context of their own company. And every company is different. Every company’s technology environment is different. Every company’s business is different.

So when you take a generalized technology like Copilot and you try and make it work for a company, you’ve got to make sure that you’re going to solve some business goals. You have to make sure that you get some return on investment and all of that. You could accept that “Oh, it’ll do some magic, and we’ll save all this time or whatever.” But what I was really trying to prompt people is to think about the technology, understand the technology, understand your business. Understand how you’re going to use the technology within your business and understand how you’re going to get results. And just that, simple.

What should we worry about the most in terms of Copilot security?

I’ll say two things about oversharing. The first thing is to remember that anything Microsoft Search can find, Copilot can use. The second thing is that if you remember when Delve came out in 2016, everybody got excited about oversharing. When Delve hinted to people that there might be some documents they might like to see, where people had to go and open those documents. And again, because search allowed them to find those documents, they can get it.

The big difference now is that Copilot consumes content. So, if it can find content, it can consume it, and it can output that content in documents, messages, or whatever. So oversharing is a problem. If you’ve been very lax with oversharing and you let anybody see anything in an organization, well then, congratulations, your new AI systems are going to consume all of that information. And it’s very good at it.

I am less concerned about that. I’m concerned about the inaccurate, misleading, and obsolete data that we all have in SharePoint and OneDrive. I think that’s actually a bigger problem.

What about hallucinations in Microsoft Copilot?

Oh no, not hallucinations. Just because hallucinations are caused by the AI doing something wrong. I’m talking about the quality of documents. Let’s say you asked Copilot a question about, me. And if I have old information? Copilot has no sense of whether the information is accurate and just no sense of whether the information is up to date. And there’s no sense of whether information is really relevant to the subject. It knows it’s got a match.

Remember we’re dealing with a word-computer, not a binary computer. So, all it’s doing is trying to get a string of words that is the best possible output that it can put out and the response given to the user. So, if you haven’t been paying attention to all of the information that’s been piling up in OneDrive and SharePoint, you could get some very interesting results back from Copilot.

Do we need to think about cleaning up our tenants?

I think it’s a very good idea for any large organization considering Copilot, to do a spring clean, a big spring clean to get rid of the old crap.