For the first episode of 2022, Paul Robichaux and I didn’t think we’d still be talking about Microsoft Exchange. But, here we are – as with the new year, new problems in Exchange On-Premises greeted Exchange admins.
Tony Redmond posted more details about the issue on Practical 365 earlier in the week, but the bug, referred to widely as Y2K22, made not only the wider computing press but mainstream news media too. In short, the bug was within the malware scanning agent and it’s date checking code, which failed and then caused the transport service on Exchange Servers that use the anti-malware agent to stop processing email.
Could Exchange admins have foreseen this?
When many organizations ran large Exchange On-Premises deployments, keeping email running was second only to the core supporting infrastructure, like AD, Networking and Virtual Infrastructure; without email most organizations are (even today) hobbled in their operations – and in the event of other systems failing will fallback to using email to make sure invoices arrive at the right place or people perform tasks.
Therefore, it wasn’t uncommon to see in IT departments prominent monitoring displays – often near the Exchange admins – showing current mail queues and other performance information, and naturally folks would perform daily maintenance and checks that went beyond simple monitoring.
These days many organizations have moved all, or most, mailboxes to Exchange Online and trust Microsoft to take care of email for them; but will still have a need to relay application emails through on-premises out to Exchange Online. Not only that – a sizable minority of organizations still need to host mailboxes on-premises for applications that don’t yet support Exchange Online.
Those organizations could be forgiven for not monitoring Exchange On-Premises mail queues quite so closely as they perhaps used to, but even if they were keeping tabs on things, this couldn’t have been foreseen or prevented.
Exchange Online and Exchange On-Premises aren’t the same anymore
As Tony reminds us, and we discuss on the show, Exchange Online and the On-Premises versions aren’t the same anymore. For a small amount of time in the earlier days of Exchange Online it could be argued that Microsoft were continually testing Exchange in the service, so bugs in code were found and fixed in the next on-premises release.
These days, Exchange Online is more than Exchange Online; known sometimes as “the substrate”, parts of Exchange Online act as the underlying layer for many Microsoft 365 services, taking advantage of the reliable, performant database and compliance features it offers. If you use Outlook.com – the days of it being Hotmail, hosted on FreeBSD are long gone, as is the platform they used on Windows; these days Outlook.com also runs on Exchange Online, and the version of Outlook on the web is common across both – with Exchange 2019 being left behind with a version similar to one phased out from Exchange Online several years ago.
This isn’t to say that Exchange Online didn’t or couldn’t have suffered from the same issues; they could have reacted quickly and solved it, or perhaps they don’t use the built-in anti-malware agents, as of course, Exchange Online Protection runs within Exchange Online.
However, it does underscore that the divergent codebase of the two products mean that Exchange On-Premises isn’t getting the same love and attention as it’s cloud twin (if you can even call it that), and to use another analogy, feels like it has been placed in the retirement home a little too early and is only getting emergency medical treatment when absolutely necessary.
Customers have little choice but to continue to run Exchange Server
If Exchange Server was only being used by a small number of customers, who refused to migrate to the cloud, then Microsoft effectively retiring the on-premises version could be seen as reasonable. That’s not the case though.
As I point out in the podcast, customers paying for the enterprise versions of Office 365 E1, Exchange Online and above are still paying for Exchange On-Premises CALs (Client Access Licenses) on an ongoing basis; and Exchange Server licensing (for Hybrid usage) is included too. The Hybrid license is often referred to as free, but – it’s not free – you have to pay for Office 365, Exchange Online or Microsoft 365 licensing to get it. Therefore, it isn’t just customers licensing Exchange Server via Enterprise Agreements with or without Software Assurance have bought it.
Not only have customers bought it, to remain supported by Microsoft if you run Azure AD Connect, you must run an Exchange server on-premises.
It’s now close to a year since HAFNIUM, and at the time, Microsoft had hinted that the message was received – and they need to solve how customers can remove the last Exchange Server, after a “sorry, there’s no news on this” at the virtual Ignite 2020.
As Paul and I discuss on the podcast – options are limited if you are trying to plan for an eventual removal, with the only guidance being to run Exchange Server 2016, as it may receive an update to support whatever solution becomes available. Since then there’s no Microsoft official guidance on what you should transition your internal mail relays to, which will – for many customers – but the one reason they’ll still need some sort of on-premises email server even after the Hybrid attribute management requirement is solved.
We discuss what Microsoft needs to do next
What we need now is to understand what Microsoft are planning, so customers have a solid roadmap they can depend upon for Exchange On-Premises and related Exchange Online features that are keeping Exchange around for customers who don’t want it and those that do – including:
- What will the solution to remove the last Hybrid Server be, and when will it be delivered?
- What will the solution for mail relay from on-premises application servers, multi-function copiers and other “dumb” devices that will never support modern auth direct to Microsoft 365, and just need to send out email to internal users without punching outbound holes in the firewall? Will Edge Servers be the go-to solution?
- When will features, like shared email domains across multiple tenants, be released, and what will the caveats be for the initial and subsequent releases?
- What new, innovative features are coming to subsequent CUs for Exchange 2019, and when will a new version of Exchange – “Subscription Edition” if it follows the SharePoint model – be released?
- What are Microsoft’s plans to make sure Exchange is more reliable for those that want to or have to run Exchange – as even if the last Exchange Server problem is solved – Exchange On-Premises isn’t going away, and there aren’t many other good options to replace it with a competitor product
So – Exchange was the focus this week, and of course – all of this is said because firstly – it needs to be said – but also because we care about Exchange, Microsoft and want them to be successful.
Normal service will be resumed on the next podcast in two weeks time, so join us then.
