How to keep up with Microsoft 365 tenant configuration changes

Microsoft 365 is a massive ecosystem of various services that work together to provide a seamless experience. While the end-user experience can be characterized as nice and solid, admins may have some challenges accomplishing their tasks. These challenges include navigating across all kinds of admin centers, automation authentication issues, remote PowerShell issues, Skype for Business and Exchange remnants, etc. There are noticeable improvements that have been made over the years to the administration experience, but the undeniable fact remains that it’s maybe a bit more complex than it should be.

How do you track configuration changes in Microsoft 365?

In such a system, the question arises – “How do you keep up with changes in the Microsoft 365 configuration?” Unless you are highly coordinated inside your organization, unwanted changes may creep up on you.

We can track the configuration changes on the tenant:

• Manually, taking screenshots and making notes on each change that was made
• Using PowerShell and writing scripts
• Using the audit logs
• Using a 3^rd party tool

It must be said that no tool can offer full coverage of each configuration setting across all the Microsoft 365 services. Every tool is a best effort of what can be done by using the available PowerShell modules and REST APIs. Until Microsoft brings support on a deeper architectural level, things will remain as is.

In this blog post, we will be looking at Audit logs, the awesome Microsoft365DSC project, and our very own Syskit Point Configuration Inventory feature which would fall into the 3rd party tool category.

Audit logs

Microsoft 365 provides auditing of user and administrator actions. Audit logs are a much broader subject than tracking configuration changes.

Downloaded a file? It’s recorded there. Created a new conditional access policy? It’s recorded there.

Tracking changes to the tenant could technically be done by analyzing the audit log for crucial activity types. And while Microsoft is doing its best to provide complete coverage of admin actions, not every change is guaranteed to be logged.

Microsoft 365 has three primary sources of audit logs:

• The unified audit logs
• Exchange logs
• Azure AD logs

Unified audit logs promise a central place for all auditing. Still, not all information is available from the unified audit logs. It may be needed to check all of them for possible changes to the tenant configuration or, better said, for the supported workloads.

As you can imagine, there is a wealth of information available in the audit logs. Still, it’s not always easy correlating an event to a change or deciphering the data contained in the event. One example is modifying a policy in the Teams admin center. The resulting entry in the audit log will be of a generic TeamsAdminAction activity type. The item modified will be in the Item field, and the modified properties will be in the ModifiedProperties field. Creating a named location in Azure Active Directory will create a log entry in the Azure AD logs with an activity type of Add Policy; creating a conditional access policy will do the same. The difference is in the PolicyType field. Simple enough, but not always, and not for all the changes you can make inside Microsoft 365. For instance, creating a Call Queue within the Teams admin center will result in a generic event without any details except that the Call Queues were updated.

Exchange audit logs are straightforward and tied to the Exchange Online PowerShell. Each change will be logged as a call to the PowerShell cmdlet used to perform the change in the system. This also applies to the Admin center UI. Every change is a call to some PowerShell cmdlet that will be logged.

As mentioned, not all workloads can be found in the unified audit logs. Ie. Intune has its audit logs found in the admin center under Endpoint Manager admin center -> Tenant Administration -> Audit logs.

Tracing changes based on audit log entries can help when you have a specific change that you need to track, you have correlated it to events in the audit log, and the event contains sufficient data. Trying to analyze the whole audit log without knowing what you are looking for might prove problematic due to the sheer amount of data available.

The unified audit log also has support for alerts when specific activity occurs. You can set up alerts in the Office 365 Security & Compliance admin center by navigating to Alerts -> Alert Dashboard -> Other alerts -> Activity Alerts. Once there, you can create an alert policy for any audit log activity.

PowerShell desired state configuration and Microsoft365DSC

Microsoft365DSC is a Microsoft community project that automates deployment, configuration, and monitoring of the Microsoft 365 Tenant via PowerShell Desired State Configuration. Desired State Configuration is a management platform in PowerShell that enables you to manage your IT and development infrastructure with configuration as code.

With Microsoft365DSC, monitoring the tenant for configuration changes is done automatically by PowerShell DSC. This means that you can write a configuration file, and it will be applied to the tenant via PowerShell. If somebody changes the settings defined in the file, they can automatically be reverted once detected on the tenant. Besides automatically reverting the change, it can also log the change in the Windows Event log or send an email when changes are detected.

Another possibility of writing a DSC configuration from scratch is to extract the configuration from the tenant for the workloads or resources you are interested in and use that as a baseline.

To extract a configuration from the tenant, you can use the command:

Export-M365DSCConfiguration -Quiet -Path [ConfigurationSavePath] -Mode Default -GlobalAdminAccount (Get-Credential)

MFA is supported and application identity as well, but not on all resources.

Once extracted you can create a report from the configuration by using New-M365DSCReportFromConfiguration:

New-M365DSCReportFromConfiguration -Type HTML -ConfigurationPath [ConfigurationPathFromExtract]

To compare two extracted configurations, you can use the following PowerShell command:

Compare-M365DSCConfigurations -Source [SourceConfigurationPathFromExtract] -Destination [DestinationConfigurationPathFromExtract]

Or you can export a report to a file by using the New-M365DSCDeltaReport cmdlet.

As you can see, Microsoft365DSC offers a lot. With a little bit of PowerShell knowledge and the cmdlets, Microsoft365DSC can become a great asset in your Microsoft 365 administration toolbelt.

However, we did face some challenges with Microsoft365DSC while working with our customers:

• Trying to run all the workloads in a non-interactive context
• Authentication issues with some tenants, especially non-public Azure environments like GCC High
• Various minor issues with some of the resources
• Performance during the extraction process for some of the resources on larger tenants

Configuration inventory feature

With support from the Microsoft365DSC team, Syskit set out to deliver our Configuration and Inventory feature in Syskit Point – a centralized solution for documenting Office 365 settings and tracking configuration changes.

Configuration Inventory tries to solve the Microsoft365DSC problems previously mentioned. The feature has a more robust authentication mechanism for workloads for which the collection process is harder to automate. It also contains a modified version of the extraction process that should be less error-prone and faster. Configuration Inventory also uses more evolved configuration change tracking than M365DSC, built upon our on-prem tool, SPDockit.

Once installed, you will need to connect to your tenant.

Here you can also automate snapshots of your tenant configuration. A snapshot is a point in time collection of configuration settings for the tenant. Options to be alerted by email when a change is detected are also here. This way, you can stay on top with any change to the configuration.

Once an automated snapshot of the configuration has been made, generated documentation of the configuration and the changes detected in the configuration will be sent to the configured email addresses.

Changes are detected based on the last configuration marked as good.
If no configuration snapshot was marked as good, it would always compare with the previous one. More on this a bit later.

Multiple formats for the documentation are supported:

• Xlsx
• Pdf
• Docx

Once the snapshot completes, you will get an email like the following example:

If you already have a few snapshots made, you will also notice a document with differences in the email (assuming there were indeed changes).

The document will contain an overview of all the changes detected.

From the overview, we can navigate to the individual changes, they will be highlighted among the settings.

The default tenant configuration settings collected in the snapshot may not be to your liking. You can change the selection in the Configuration Inventory options according to your requirements.

Also, you may not want to compare every extracted change. This is also possible to change in the Configuration Inventory Options.

The vital Configuration Inventory function is marking a configuration snapshot as the baseline – the last known proper configuration. Any newly created scheduled configuration snapshot will be compared to the baseline snapshot.

You can mark a configuration snapshot by navigating to the snapshots screen, selecting a snapshot, and clicking on the Mark Configuration as Good in the ribbon.

Besides the automatic change detection using the scheduled snapshot, you can also manually compare any two snapshots using the Compare Wizard on the Start screen. Baseline snapshots will also appear here with a checkmark in the Marked as Good column.

The data is presented hierarchically with filtering.
Configuration Inventory will mark the object and its properties as different if they differ from the last known good configuration.
Once you select an object in the hierarchy you will be able to see more details about the changes made.+

We can see the changes made to the OneDrive Storage Quota and the retention period for orphaned OneDrives. The changes will always be highlighted in the results.

In the example below Configuration Inventory determined that there is a change to when anonymous links expire.

In the following screenshot, it’s visible that two conditional access policies have been deleted indicated by the missing in source state.

When you are done exploring the differences via the UI, you can also export them in a .xlsx document.

Conclusion

As with many things in life, in the end, there is no silver bullet. You will probably need to combine a 3^rd party tool, some scripts, and maybe even some manual work to achieve the desired coverage of your Microsoft 365 tenant configuration. If you’d like to check out Syskit Point’s Configuration Inventory options, start a free trial and see firsthand how it works.