As an SCCM admin for the last decade I have managed multiple versions of the product at various large companies. I was always thankful when budgets allowed for proper SCCM site infrastructure, which made my job a lot easier. The last company I worked for was a large retailer that had over 500 retail locations. SCCM 2007 was already managing clients at central offices, but not the retail locations. We spun up a project to migrate to SCCM 2012, and management wanted to expand SCCM into the retail locations.
Around this time, there was a separate project to install virtual hosts at these sites, so I was so happy when they told me I was able to install an SCCM distribution point at all of these locations. Everything seemed perfect. I wouldn’t have to deal with our networking guys hassling me when clients would download packages and patches over the WAN, and clients could get content from their local DP. SCCM is the world’s most powerful systems management platform, and the new architecture was going to make it run even better. However, with all this power I was going to face some new challenges I was not aware of...yet.
It’s Complicated
Now that I had 500+ new locations to manage, I started to look at the network configurations of these sites to prepare for new SCCM boundary and boundary group creation. Come to find out, documentation was inconsistent, but some SCCM 2007 clients were already coming online at these locations, so I had some data to work with. I spent weeks developing scripts, database queries, and spreadsheet formulas to discover networking information which I collected from the SCCM 2007 environment. I knew the boundary and boundary group configuration wasn’t complete, and I would need to run periodic audits and look for failed deployments to try to discover new information.
So what would I need to be able to support 500+ distribution points? As per Microsoft’s support guidance, a primary site can only support 250 standard DPs, so I had to install additional secondary site servers to be able to support 500+ DPs. In SCCM 2012, secondary sites now require SQL and even though SQL Express is free, I didn’t want to deal with any additional SQL replication headaches. I had no choice but to use secondary servers in my architecture.
As you can imagine, it takes time to install all of those DPs. Luckily for me, a different team was responsible for installing and configuring the OS and SCCM prerequisite operating system roles such as IIS and BITS. Using SCCM PowerShell Cmdlets, I was able to automate the DP role installation on the servers, but I still had approximately a 10% failure rate due to various issues with prerequisites missing, problematic IIS installations, and general networking failures. It took much longer to install the DPs than planned, thus impacting my project schedule.
In the end, the SCCM architecture looked something like this:
Rewards and Challenges
Content migration from SCCM 2007 to 2012 was a complicated process, but provided a great cleanup opportunity. The unexpected work really began when I had to start distributing that content to all of the distribution points. I tried to stage it, but It took several weeks to send 400+ packages to all of the DPs, and even longer to troubleshoot failed content distributions. The bottleneck at the secondary site servers was significant. I quickly realized that having all of these DPs in the environment was going to be a source of significant additional work as I began to deal with harsh QoS rules, random secondary site server failures, DP components failing, and port blocking. I mean, even after working things out with the networking team to tweak QoS and ports, and firming up the secondary sites and DPs, there was still about a 8% failure rate in content distribution. In order to work around this, I had no choice but to configure fallback distribution points, and work closely with the deployment team to make sure they only allow fall back on specific packages.
Having such a large SCCM infrastructure helped reduce WAN traffic, but brought up some new challenges (or as I like to call them, opportunities). In Part Two of this blog, I will go into the details of how Adaptiva OneSite Anywhere would have resolved some issues I faced in my recent SCCM 2012 deployment.
I’ll explain how using OneSite Anywhere, I could have:
- Reduced the number of boundaries
- Removed the unneeded secondary sites
- Saved time by not having to troubleshoot failed distribution point installations
- Streamline the environment by removing all 500+ DPs
Here’s a peek at what my infrastructure could have looked like with OneSite Anywhere.
To find out how it works, read Part 2 of this blog.
