Tips and Techniques to Avoid Common Issues

In a previous post, I discussed how to leverage the Microsoft Tunnel solution to control access to your Microsoft 365 environment. In this article, I discuss deploying the Microsoft Tunnel VPN gateway in an enterprise environment.

Basic Deployment

The basic steps to deploy the Microsoft Tunnel VPN gateway are:

• Prepare a Linux server (yes, the tunnel is only supported on Linux).
• Install Docker or Podman depending on your Linux Distro. This is because the Microsoft Tunnel VPN gateway is a docker image that differs from other components Microsoft provides, and must be installed on a Windows host server.

Certificates

Before deployment, you must obtain a TLS certificate that is trusted by Microsoft 365, meaning that the certificate must be issued by a public CA. This certificate is used by the Microsoft Tunnel VPN gateway to form the encryption tunnel to protect your data. Once you have the certificate ready, you need to copy either the PFX or PEM set (crt file and key file) to /etc/mstunnel/private/ of the host server where you run the Microsoft Tunnel VPN gateway docker image.

After that, you must download two scripts from https://aka.ms/microsofttunnelready (mstunnel-readiness) and https://aka.ms/microsofttunneldownload (mstunnel-setup). I recommend uploading to /tmp/.  Remember to update permissions so you can run the scripts. Although this step is not required, I strongly recommend running the mstunnel-readiness script as this checks if your Linux host is ready to run the Microsoft Tunnel VPN gateway. If all checks pass, you are ready to run mstunnel-setup to setup the docker image on your host and perform the other configuration steps.

That’s it, right? Yes, for a clean setup of Redhat (8.4 or below) or Ubuntu server used for testing. Unfortunately, this was not the case in a recent enterprise client project where I helped to deploy the Microsoft Tunnel VPN gateway. Enterprise environments tend to add complexity to a simple deployment task as they have many extra components. Let’s dive into the issues I met in the deployment and how I resolved the problems.

Redhat 8.5 or above Specific – Iptables

My client uses Redhat 8.5 as their standard distro, and we found that the Microsoft Tunnel VPN Gateway will start on this distro but then reports some errors. After checking with Microsoft Support, we discovered this issue is specific to Redhat 8.5 or above, which by default doesn’t load the iptables. Microsoft Tunnel VPN Gateway depends on iptables for it to work properly. My team then needed to configure the host to use the iptables instead of nftables and firewalld. The following is a command to disable nftables and firewalld services as they cannot coexist with iptables.

systemctl disable nftables firewalld
systemctl stop nftables firewalld

After the command completes, we can install iptables and load them when the system starts.

dnf install iptables-services
systemctl start iptables
systemctl enable iptables

Following the reconfiguration to use iptables, Microsoft Tunnel VPN Gateway will work properly.

Using Web Proxy with Microsoft Tunnel

In my recent deployment, my client requires the use of an HTTP proxy for any outbound HTTP/HTTPS traffic. I deployed the proxy a few months ago. At that time, Microsoft didn’t provide much information on how to do it. Now, Microsoft has updated their documentation to make things easier. In short, there are two ways to configure the HTTP proxy support on the docker.  Configuring the HTTP proxy support may differ depending on the Linux Distro you are using.

It is important to note that the Microsoft Tunnel VPN Gateway does not support proxy with authentications. Make sure you have the network or proxy administrator configure the proxy to bypass authentications for all traffic coming from the VM running the docker image.

Configure Outbound Proxy at Linux Host

In my experience, the easiest way to configure the environment variable at the host level is to tell all applications, including Docker/Podman, to send all outbound traffic via HTTP Proxy. You can either update the proxy setting at Docker/Podman level or host level. I always go with the simplest way, which is to update the host using the method outlined below.

1. Modify /etc/mstunnel/env.sh (Redhat) or /etc/environment (other distros) to add the following two lines to the end of the file
   1. HTTP_PROXY=http://<proxy>:<port>
   1. HTTPS_PROXY=http://<proxy>:<port>
2. Restart the Microsoft Tunnel VPN Gateway (mst-cli server restart) or restart the server to make the change effective.

For other methods to configure Docker support, refer to Microsofts Documentation.

SSL Inspections on HTTP Proxy

There are two kinds of internet traffic from the Microsoft Tunnel VPN Gateway host:

• Download the docker image for the Microsoft Tunnel VPN gateway from Microsoft Container Registry
• Docker image to communicate with the Microsoft services in order to provide the Microsoft Tunnel services

In our experience, we find setting the proxy environment variable at the host level allows the host and docker image to access the required services but then the tunnel still reports connection issues at the Intune admin portal. After checking with Microsoft Support, we found the proxy-enabled SSL Inspections were causing some of the authentication requests to fail completely. After disabling SSL Inspection at the proxy level, the Microsoft Tunnel VPN Gateway works perfectly.

Conditional Access

After we sorted all the server-side issues, we focused on mobile devices. We discovered that the connection from mobile endpoints did not work when we generated the log for mobile devices. We find errors like that shown in Figure 1. Note that we used Microsoft Tunnel app at that time. Now, we should use the Microsoft Defender mobile app and the log contents may look different.

The log clearly indicates that conditional access is blocking client access. My client deploys conditional access policies to allow selected services from mobile devices. Obviously, the Microsoft Tunnel Gateway is not part of the allowed services at the time and hence blocks the traffic.

We modified the conditional access rules to include an exception for the Microsoft Tunnel Gateway app by entering the App ID. Instructions for this step can be found here.

Make sure this app is configured to bypass any MFA checking and allowed access from any location, otherwise Microsoft Tunnel won’t work.

Access to Azure Storage Services

In a recent change to the Microsoft Tunnel VPN Gateway image, there is a need to write logs to Azure Storage operated by Microsoft. This change caused some issues because the proxy is not configured to allow all traffic from Microsoft Gateway VPN Gateway.

After checking the log using the command journalctl -t ocserv -t ocserv-access -t mstunnel-agent -t mstunnel_monitor, we found the following log entries (Figure 2).

After we updated the proxy configuration to allow the Microsoft Tunnel VPN Gateway host to access Azure Storage, the services resumed as normal. There are some other sites/services that must be enabled (see this page).

Conclusion

Deploying Microsoft Tunnel is supposed to be a simple and easy task if you prepare the dependencies. In most situations, this is the case, but perhaps not for enterprise environments. Check that all the needed prerequisites are ready before proceeding to the actual deployment. One more thing to note is that Microsoft constantly updates components. This can cause additional dependencies (like the introduction of Azure Storage requirements a few months back). Keep your eyes open for these kinds of changes and react accordingly.