Programatically Retrieve all Office 365 Unified Audit Logs - microsoft-graph-api

Question

I would like to programmatically retrieve and process all logs available from the Office 365 Unified Audit Logs for the purpose of forensic investigation. From the front end, these logs are available through the Office 365 Compliance Admin Center.
I have tried the following options to access these logs from a script, with no success:
Microsoft 365 Management API - This contains the correct data, but is of limited usefulness for forensic investigations due to the short 7 day retention period.
Microsoft Graph - This does not contain all the relevant data - you cannot access the Unified Audit Logs directly through Graph, and the usage reports do not cover all items contained in the Audit Logs (e.g. Exchange actions).
Search-UnifiedAuditLog on Exchange Online PowerShell - Microsoft themselves recommend not to use this programmatically, and I've experienced extremely unreliable results and unmanageable rate-limiting when trying to do so.
So is there something I'm missing here, or is there no way to programmatically retrieve all items from the Unified Audit Logs for the entire retention period? (generally 90 days).

Answer 1

As far as I know the only way to do this is to use the Management API on a regular basis and output the results to some solution for long term storage (Azure Log Analytics Workspace comes to mind, or SIEM like Splunk / Graylog). I.e. write a script that retrieves logs for the last week, and run it at least weekly.
I'll explain how to retrieve logs manually and also show a tool which already exists for this at the bottom of the post.
Manually:
1: Enable Audit logging on the tenant if not already enabled
2: Create an App registration in Azure AD and for getting single tenant audit logs choose "Accounts in this organizational directory only (xyz only - Single tenant)"
3: Create a 'secret key' from within the newly created App Registration. Store it somewhere safe as it's only shown once. From the overview page of the App Registration also store the "Tenant ID" and "Application (Client) ID". You will need all three.
4: From within the new App Registration go to "API permissions" and add 'Application type' permissions for: 'ActivityFeed.Read' and 'ActivityFeed.ReadDlp'.
5: For the following steps you will need to start calling the Office API's, for which you will need a bearer token in the header. To obtain this send the following POST request:
URL: https://login.microsoftonline.com/***tenant_ID***/oauth2/token
Headers: "{'Content-Type': 'application/x-www-form-urlencoded'}"
Data: "grant_type=client_credentials&client_id=Application_ID&client_secret=Secret_Key&resource=https://manage.office.com"
You will receive a JSON response which contains 'access_token'. For all the upcoming API calls, use the following header:
"{'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'bearer access_token'}"
6: Subscribe to the audit log feeds you would like to retrieve. The following exist: 'Audit.General', 'Audit.AzureActiveDirectory', 'Audit.Exchange', 'Audit.SharePoint', 'DLP.All'. The POST for Exchange for example would look like: "https://manage.office.com/api/v1.0/tenant_ID/activity/feed/subscriptions/start?contentType=Audit.Exchange"
7: You are now ready to start retrieving actual logs. Individual logs live inside content blobs, which live inside pages, which live inside feeds (e.g. the Audit.Exchange feed). Therefore, for each feed you would like to retrieve logs from, you must collect all the content blobs (iterating through the pages of them) and then retrieve the actual content from that blob.
To retrieve a page of content blobs use the following URL (change bolded content to your situation): "https://manage.office.com/api/v1.0/tenant_ID/activity/feed/subscriptions/content?contentType=Audit.Exhange&startTime=2022-04-13T09:42:52&endTime=2022-04-14T08:42:52"
This will give you a JSON response with content blobs inside. In the response header check "NextPageUri"; if it contains a URL, call that URL for the next page of content.
Now that you have content blobs, use them to retrieve the actual logs. Each content blob is a JSON dict, which contains a "contentUri" field. Call that URL to retrieve a JSON response with the actual logs inside.
You can do this in most programming/scripting languages, but for larger amounts of logs you will want to retrieve logs in parallel, or it will take a long time.
With a tool
In case you want to use an existing tool, this one is free, works on Linux and Windows, and supports multiple outputs.