What Happens To Modern Apps After Joining AD Domains?

So, you’re all excited about the new Windows start screen, you’ve familiarized yourself with the interface, and you’re ready to think about deploying Windows 8 across the enterprise. You install your first test system, join it to the domain, and just to see what the weather is like outside, you click the Weather App and maybe allow access to location services, only to be greeted by:

or…

That’s odd. It worked before you joined it to the domain.

Since you’re testing with a laptop and it’s getting late, you take the system home to troubleshoot the problem further, only to find there are no problems. Rather than question your mental capacity, you instead chalk everything up to some temporary unknown glitch, but when you get back to work the next morning, you find that your Modern Apps are broken again.

Such was the nature of our first experiences at rolling out Windows 8. Since, we did not have a big push to roll it out early, and before long there was talk of Windows 8.1, I thought maybe this was a quirk of of Windows 8, and maybe Windows 8.1 would resolve the issue. However, when Windows 8.1 previews were released we quickly saw that, not only did it not fix the problem, but we began to observe odd behavior in IE 11 on Windows 8.1. For example, MSN showed up without most of its styles applied, and some content missing. The same thing was true for many other web sites.

I came to the conclusion that this was not going to go away by itself, that we had to get to the root cause of this issue, and that the cause might be something fairly unique about our environment.

Isolating the Problem

After a few weeks of off and on troubleshooting of the issue, I determined that it was related to the presence of an Akamai device, (content delivery service utilized by Microsoft and many other companies), that we happen to host on our network. We have little control over the Akamai equipment, but it serves an important function in that of dampening peak demands over our ISP connections when there are heavy demands for new software downloads, and other types of heavy internet content demand. Our networks have thousands of systems connected at any one time, and events like this can saturate an internet connection.

As it turns out, I found that while a Windows 8.1 system was domain joined and on our network, any traffic that was targeted, from IE or Modern Apps, toward this Akamai device, was simply getting blocked by Windows itself before it ever hit the network. Here are some of the techniques used to isolate the problem:

Initially, packet captures did not show much.
- This is because the particular traffic that was getting blocked, was blocked before the capturing software ever saw it, but I did notice there was extra traffic, when the problem was not there, as compared with a capture from a system that was exhibiting the problem.
- Later, as I looked more closely, I noticed captures from non-domain joined systems, were sending traffic to the Akamai system, so using netmon, it was the lack of traffic from wwahost.exe and iexplore.exe at the appropriate times that was important.
I noticed that the same circumstances where modern apps were failing were the circumstances where IE 11 was having problems on Windows 8.1.
F12 Developer Tools built into IE 11 show that certain requests were aborted.
- Clicked the Tool Menu (Alt+X), and select “F12 Developer Tools”
- Selected the Network tool (Ctrl+4), and click the green icon to start capturing network activity for IE.
- In IE, I duplicated the problem by navigating to a site that has the issue, or click the refresh icon if already on that site.
- In the F12 tools, then clicked the red square icon to stop capturing
- Noticed that there are URLs listed that show “(aborted)” in the Result column.
- Using nslookup, I determined that the hostnames for the aborted requests all resolved to an address on our network that was assigned to the Akamai system.
If I “turned off” the firewall in all 3 profiles, Windows still blocks the traffic.
If I stopped the Windows Firewall service, the problem disappears, and outbound traffic to the Akamai was succeeding.

I just could not seem to make anymore progress after that, so it was time for a call to Microsoft support, and after reviewing the situation with them, we eventually arrived at the solution. While attempting to resolve the issue, I initially found little documentation on the issues, other than a few proxy server related hints at the problem. (Since we didn’t really use a traditional proxy server, I dismissed those.) This is my effort to help document these issues more completely, and hopefully save others some of the time that I spent on it.

Modern Apps and Network Isolation

Modern Apps are different than standard desktop applications. In some ways, Modern Apps are sort of like a cross between running normal desktop applications, and simply browsing the web with a web browser. They run under a host process called WWAHost.exe, and as it turns out, Modern Apps have a couple of characteristics that introduce an interesting dilemma.

Users with relatively low levels of awareness can download and install the apps from the Windows Store, without being administrators.
Because these apps come from the Windows Store, which we administrators typically have little control over, by default we have little control over what they do.

Given the above points, what’s to stop somebody from “unknowingly” downloading an app that surveys all of the computers and addresses connected to the local network, and sends this information to “the bad guys”?

Well, one technology that can help with this issue is what Microsoft refers to as Network Isolation. The idea is that Windows will attempt to automatically discover your local network, and treat it differently than the rest of the internet. If an app needs access to the Internet, it must request it. If an app needs access to the local network, (Home/Work network), it must request that also. To accomplish this, developers have to declare the network access requirements, in a manifest, in order to gain access to the various types of resources.

When we are browsing through available apps out on the Windows Store, we can see what type of network access the apps have declared that they need access to. We might see, the app needs to use, “Your home or work networks…”, or it might say it needs to use, “Your Internet connection…” If we agree to that, we can go ahead an install the app, with some awareness of what it’s doing.

For example, one of the apps I spent a great deal of time testing is the built-in Weather app. If we search the App Store for “weather”, we find a number of choices, but we’re looking for “The Weather app, powered by Bing”. If you click on that app, to view the description and then click on the permissions link, we see the following:

Now that makes some sense, when you think about it. The app is going to try to deliver weather information, ideally for the location you’re at, and to do that, it’s going to talk to some servers out on the Internet that serve out weather information.

In order to enforce whether an app gains access to Home/Work networks or the Internet, Windows uses Network Isolation to define the difference between those networks, and forces an environment where the app can only access what it publicly requests access to. Therefore, when Windows starts, it needs to be able to determine the difference between your Home/Work network and the Internet. I mentioned that it does this automatically, but how? Well, this is all documented here, but for our purposes…

If not joined to a domain, Windows is assuming any address not on the local subnet of your Home/Work network is an Internet address, and it defines the local network based on the address and the subnet mask that is configured on each of your applicable network interfaces. Everything else is considered Internet address space.
If joined and authenticated to a domain, Windows takes things a step further, and queries Active Directory (AD) to locate the subnets defined for the various Sites in AD. These are all treated as part of your “home or work” networks in addition to everything on the local subnet. Everything else, then, is considered Internet address space.

Sounds easy enough: App determines location, talks to server, and server provides weather information. Now, all of that may sound straightforward, but how does that relate to the problem at hand? After all, the Weather app’s servers are out on the Internet, and it requested those permissions, right? So everything should be fine. Well, things aren’t always quite that simple.

Proxy Servers, Content Distribution, and other Twists To This Story

Let’s look at how Windows determines what the app is attempting to talk to. To an extent that depends on which way the traffic flows.

Outbound Traffic – Outbound traffic is initiated by the app, and from it’s perspective that traffic is destined for whatever IP address is in the Destination IP address field of the outgoing packets.
Inbound Traffic – The app makes requests to create listeners to handle unsolicited inbound packets, and from that system’s perspective, the traffic came from whatever IP address is in the Source IP field of the incoming packet.

In our case we’re dealing with outbound traffic, and you might be tempted to think that this destination address is always the address of some server setup specifically for the app or web page. But that is often not the case.

Proxy Servers – A traditional proxy server is a special server setup to handle outbound requests, usually http, and often to the internet. In order to work, proxy servers are typically configured on the client end of things, so that some or all, typically http, traffic is sent directly to the proxy server, and then the proxy server makes the connections to the actual web servers, passing the responses back to the client as it receives them. The important point here is that outgoing packets from an app running on a system that sends through a proxy server to get to the internet are destined to the IP address of the proxy server.
Reverse Proxy – Reverse proxies are similar to regular proxy servers except they work on the server side of the connection. For a variety of reasons you may want to put a server behind a reverse proxy to isolate it from the internet. To make this work, server administrators will configure the public DNS name of the server to point to the reverse proxy system instead of the server itself, and then configure the reverse proxy to send the appropriate traffic to the server(s). Typical reverse proxies are usually on a network near the server anyway, and are less likely to be a problem in the context of Network Isolation, but understand them is useful for thinking about content delivery services.
Content Delivery – Content delivery services, like Akamai, provide services to content providers, and in many ways are a lot like the reverse proxies discussed above. However, they typically rely on a intelligent DNS servers to direct clients toward what amounts to a reverse proxy with cache that is closest to the client. This is commonly done by using a series of intelligent CName records or aliases, where the response returned will most likely depend on the address of the DNS resolver that is attempting to resolve the name. This distributed caching mechanism is really the goal of this service, and the purpose of this is to reduce internet bandwidth required during periods of relatively heavy demand for specific content. For example, if Microsoft releases a Service Pack, there is usually a significant demand to download that at the time of release. By using a content delivery service, Microsoft is able to reduce the demand on their own servers and portions of the Internet, by having this cached on content delivery devices all over the world, closer to the clients that are requesting the content. The important point here is that outgoing packets from an app, that reference content that is delivered through a content delivery system like this, will actually be destined to the IP address of the nearest content delivery device to the client.

Putting It All Together

Now consider an app like The Weather app. It requires access to the Internet, but it will not get access to your Home/Work networks. You can imagine what will happen, when the app requests weather information from its servers, but because a proxy server or a content delivery service is involved, the address it ends up sending to is actually the address of a proxy server or content delivery system on the Home/Work network. Windows will see this as an attempt to access resources that the app has not requested access to, (Home/Work network), and blocks that traffic.

Recall also that I mentioned IE 11 on Windows 8.1 also began to exhibit some problems. When IE 11, running on Windows 8.1, has “Enhanced Protected Mode” enabled, as it does by default, it follows similar behavior. If you browse an Internet site, (Internet Zone), and the browser ends up targeting an address that happens to be within scope of the AD subnets defined in AD, Windows will block that traffic by default.

What To Do…

The key to solving these problems is to make sure the Windows will treat addresses that deliver internet content appropriately, and treat addresses that deliver intranet (Home/Work) traffic appropriately as well. Most of this is also documented in Isolating Windows Store Apps on Your Network, but I’ve added a couple of details.

Note/Disclaimer: Before implementing any of these in a production environment, always test your changes in a similar test environment to make sure you obtain the desired outcome. Editing the Windows Registry is dangerous, and can cause damage to your systems, if you don’t know what you’re doing. I can’t be held responsible for the outcome. I can only explain what worked in my environment.

Recommendation 1: Treat External Content Delivery Systems as Internet Address Space

If you have an external content delivery system on your network, you have little or no control over what it brings into your network. For that reason, I think the most appropriate way to handle this address space is to make sure it is always treated as internet address space. Ideally, keep all of these types of systems on dedicated subnets. Because of the way Network Isolation uses AD Site subnet assignments, that means you’ll typically want to exclude that address space from the subnets defined in AD.

I actually attempted this early on, as part of my troubleshooting, but that alone didn’t seem to fix the problem. Simply removing the applicable subnet from AD Sites and Services is not enough to make the problem go away from your existing Windows 8 and Windows 8.1 systems; at least not right away. Even if I ran “GPUpdate” and rebooted afterwards the problem was still there. It turns out there are some extra steps that I needed to do to make the symptoms go away, and these are included below.

Recalculate your subnets to separate out the address space dedicated to external Internet endpoints from Intranet (Home/Work) address space.
1. Locate any subnet that includes the external address space. This is a subnet you’ll need to remove.
2. If the address space includes address space outside of that which you’ll want to exclude, then list the address space you’ll need to include back in. For example if you have 172.1.0.0/16 defined in AD, and 172.1.1.0/24 is what you need to exclude, then might add the following address space back in: 172.1.0.0/24,172.1.2.0/23,172.1.4.0/22,172.1.8.0/21,172.1.16.0/20,172.1.32.0/19,172.1.64.0/18, and 172.1.128.0/17. (If you aren’t sure how to recalculate these, there are plenty of references on calculating subnets, and for now I’m leaving it outside the scope of this article.)
3. Alternatively, you can allocate new dedicated address space that is not or will not be included in AD subnet definitions. If so, you’ll need to move any external content delivery devices to the new subnet.
Redefine your Home/Work network address space, using any of the following options, to exclude address space dedicated to external Internet address space.
- As mentioned above, the first option to consider is to modify the subnets defined in AD Sites and Services. I can’t tell you what your new subnet definitions should be, but they should exclude the address space assigned to the external content delivery devices you’re trying to exclude. I describe using AD Sites and Services below, to give an idea of what to do.
  If you have Domain Controllers running Windows Server 2012 or later, there is also a Powershell option, using New-ADReplicationSubnet and Remove-ADReplicationSubnet.
  1. Open AD Sites and Services.
  2. On the left pane, expand sites, and the click subnets.
  3. Locate the subnet(s) you plan on removing, and note the Site(s) that they are assigned to.
  4. Delete the subnets you determined need to be deleted from up above.
  5. Add any subnets you determined needed to be added back to AD subnets, from above.
- If you can’t or don’t want to update the subnets in AD, you can consider overriding the them with a GPO
  1. Open the Group Policy Management snap-in (gpmc.msc) and edit the Default Domain Policy.
  2. From the Group Policy Management Editor, expand Computer Configuration, expand Policies, expand Administrative Templates, expand Network, and click Network Isolation.
  3. In the right pane, double-click Private network ranges for apps.
    In the Private network ranges for apps dialog box, click Enabled.
  4. In the Private subnets text box, type the subnets for your intranet, including all those defined in AD except the subnet(s) that you need removed, as determined above.
  5. In the Private subnets text box, add any additional subnets that you determined needed to be added back in.
  6. Double-click Subnet definitions are authoritative, click Enabled. (Here, we assume the AD subnets still contain the external address space, therefore you don’t want to allow these address ranges to merge with the ones defined in AD.
- If you are moving your external Internet devices to new address space that is not included in AD subnet definitions, then you’ll need to go ahead and do that. How you accomplish that is out of scope for this article.
If you made any changes to AD subnet definitions or used a GPO to override them, you’ll need to fix the Windows 8/8.1 systems that already experience these issues. The only way I found to accomplish this was by deleting some registry entries, and then rebooting the affected systems to force it to recreate them using the new subnet definitions. Ideally you’ll want to automate this, perhaps using SCCM, scripts, or other tools. The idea is to do the following on each affected Windows 8 or Windows 8.1 system.
Note: I suspect they would eventually fix themselves without this, but at this time, I’m not sure how long that would take or what has to happen.
1. On each affected Windows 8/8.1 system, navigate to the following registry key:
  HKLM:\System\CurrentControlSet\Services\iphlpsvc\Parameters\ADHarvest
2. If your system is like the ones I’ve been testing with, you’ll find 3 values: LastFetchContents, LastFetchDomain, and LastSuccessfulADS&SFetch. Delete all 3 of these values, but don’t delete the ADHarvest key.
3. Reboot the affected system.

Recommendation 2: If Necessary, Declare Internet Proxy Servers Using Network Isolation Group Policy.

If you have proxy servers on your network, you generally have much more control over what they can be used for. If you are reading this, it’s probably because you’re having a problem, so it is possible that Windows is not treating your proxy servers appropriately. The documentation suggests that Windows will typically auto-discover proxy servers.

However, if you need to, you can use Network Isolation Group Policies to declare the appropriate addresses as internet proxy servers. Be sure to investigate the “Proxy definitions are authoritative” and “Intranet proxy servers for apps” policy settings to see if the apply to your situation. For the scenarios we’re discussing these settings would not be needed.

Open the Group Policy Management snap-in (gpmc.msc) and edit the Default Domain Policy.
From the Group Policy Management Editor, expand Computer Configuration, expand Policies, expand Administrative Templates, expand Network, and click Network Isolation.
Double-click Internet proxy servers for apps.
Click Enabled, and then in the Domain Proxies text box, type the IP addresses of your Internet proxy servers, separated by semicolons.

MetzlerTech's Blog Tech Bits