Even with the most meticulous design, the day will come when your farm’s capacity is not sufficient any more. User numbers increase, applications become more resource-hungry and the amount of data to be handled increases steadily. So what do you do? Simply more of the same, i.e. buy more servers and add them to the farm? That is one way of increasing capacity, but it is not the only one and therefore may not be the best.

In this article I am assuming that your farm actually is near 100% capacity (with a safety margin, of course). Only then does the following discussion make sense. It may very well be that a farm appears to be fully loaded, but in reality the terminal servers are having a good time idling around. In such cases some other component slows things down. The list of potential sources of trouble is long and includes the network, infrastructure servers (Citrix data store, domain controllers, file servers, application servers, etc.) and even misconfiguration of the virus scanners. Before going out and spending money on new hardware or software, it might be a good idea to have an external consultant review your farm design to make sure you are spending your company’s money on the right things.

Identify the Capacity Bottleneck

Once you are sure that you actually have a capacity bottleneck, the obvious first step is to identify it. Just knowing “I cannot get more than x users on each of my servers” is not going to be enough.

A XenApp farm is a very complex thing. Finding a bottleneck may at first sound like looking for a needle in a haystack. So let’s simplify. Every terminal server consists of the following main components:

• Memory
• CPUs
• Hard disks
• Network interfaces

Today’s (dual) 1 GBit/s network cards are fast enough for typical XenApp servers. Hard disks are not used much, except for storing temporary and user profile data, and large parts of the profiles are redirected to a file server anyway. CPU management in XenApp works well. That leaves us with memory as the culprit. How fast was that for performance analysis?

I must admit I lied to you. Finding a capacity bottleneck may very well be a really tough job. Every farm is different. No two XenApp customers have the same application set. But still, if we are mostly looking at typical office or task workers and taking a typical network and farm design as a basis, it mostly boils down to memory being short.

There are two kinds of memory shortage on 32-bit systems. The first is exhaustion of kernel memory (see my earlier article on the subject for details). The second simply is lack of enough RAM to satisfy the running applications’ demands.

Depending on whether your servers reach the limit of their kernel or application memory pool, one or more of the following scaling options may apply to your situation.

Be sure to check out Helge Klein's blog post about how to increase the capacity of a XenApp Farm.

Click here to read all of it

First of all, there's the design of the User Store. If you're using Windows XP and/or Windows 2003, you'll be using "version 1" profiles, and if you're using Windows Vista or Windows 2008, you'll be using "version 2" profiles. That's Microsoft's terminology, but it can get really ambiguous when I'm also talking about UPM versions, so I'll refer to Windows XP and Windows 2003 profiles as Win5 profiles. Likewise Windows Vista and Windows 2008 profiles can be grouped under the term Win6 profiles. As an aside, it's looking like Windows 7 profiles share the same format as Vista and 2008, so at the time of writing, it looks like Windows 7 is also covered by the term Win6.

So what we're expecting is that you'll put both your Win5 and Win6 profiles on the same server, and we expect that you'll set up your User Store as suggested by Microsoft's article http://technet.microsoft.com/en-us/library/cc757013.aspx (Security Recommendations for Roaming User Profiles Shared Folders). For the sake of example, let's say you have the User Store set up as the share //OutlawSrv1/profiles/ . UPM lets you configure paths using Active Directory attributes, so you might use either #cn# or #sAMAccountName# to define the next level. Again, for the sake of example, let's use #cn#, giving us a path of //OutlawSrv1/profiles/#cn#/ . If we do this, all the different UPM directories we're going to create will just inherit the permissions from the top level directory, which is exactly what we want.

Now we recommend setting up a System Environment variable to differentiate between Win5 and Win6 profiles at the next level. UPM lets you incorporate System Environment variable into the profile path, by enclosing the variable in percent signs. In our example, we'll set up a System Environment variable %ProfileVer% which will take the value "v1" on Windows XP and Windows 2003 systems and "v2" on Windows Vista and Windows 2008 systems. So now the complete path you configure in the UPM GPO is //OutlawSrv1/profiles/#cn#/%ProfileVer%/

Here is the User Store layout that results:

//OutlawSrv1/profiles/William/v1/
                     .       /v2/
                     /Douglas/v1/
                     .       /v2/
                     /Henry/v1/
                     .     /v2/
                     /Ginger/v1/
                     .      /v2/

Sorry if that's a bit long winded, but it's important to understand that this is the basis of our planning for the next versions of UPM. We're going to assume that new features will be able to slot in alongside the v1 and v2 directories above, so that UPM v3/v4 will build per-feature data on the directory structure above, giving:

//OutlawSrv1/profiles/William/v1/
                     .       /v2/
                     .       /feature1/
                     .       /feature2/
                     /Douglas/v1/
                     .       /v2/
                     .       /feature1/
                     .       /feature2/
                     /Henry/v1/
                     .     /v2/
                     .     /feature1/
                     .     /feature2/
                     /Ginger/v1/
                     .      /v2/
                     .      /feature1/
                     .      /feature2/

That's the first recommendation. If you're only interested on how to future-proof your UPM installation against future UPM feature updates, you can stop reading here.

The next recommendations are about how you might manage upgrade scenarios. Typically, you've got UPM v2 deployed, and you'd like to take advantage of the new features in UPM v3 or UPM v4. Unfortunately, we don't expect to be able to offer co-existence of different UPM versions. Specifically, for any given user, all machines that he uses must be managed by the same version of UPM. Otherwise, older versions of UPM will just ignore the new feature directories, and will make changes in the v1 or v2 directories without making corresponding changes in the feature directories. Result: a corrupt profile - precisely what UPM is supposed to avoid!

The first step in solving this problem is that we'll be providing compatibility flags in the configuration of all future versions of UPM. If UPM v3 (say) can't find a configured compatibility flag, it reverts to the UPM v2 functional level. Similarly, if the flag is turned off, it operates at the UPM v2 functional level. So it's OK to install newer versions of UPM over the top of older versions - they'll work correctly. Likewise, the configuration of UPM v3 will be designed to be a superset of UPM v2, so you can replace the v2 ADM template with a v3 ADM template and UPM v2 will continue to operate correctly. (BTW, you're probably better off removing INI files, they could cause problems. My personal opinion is that INI files and production farms don't mix.)

There are two difficulties:

• AD changes don't propagate through the forest instantaneously, so we have to avoid the situation where some instances of UPM v3 have been told they can now operate at UPM v3 level, while others haven't seen the group policy update and continue to operate at UPM v2 level.

• You can't (in general) be sure that you've found and updated every copy of UPM v2 to UPM v3, so regardless of what group policy says, some copies of UPM can only operate at UPM v2 level

There are two ways around this problem.

The first is applicable in small networks, where you can be sure that you've managed to upgrade every machine.

1. Replace the UPM v2 ADM template in the GPO with the new UPM v3 template. Set the compatibility flag to off (0)
2. Upgrade all machines to UPM v3
3. Make whatever other changes you need to migrate (as per UPM v3 administrator guide)
4. At a scheduled maintenance period, log everybody off
5. Make the necessary configuration changes and set the compatibility flag to 1
6. Wait for the group policy changes to propagate - maybe run gpupdate on each machine to force the update
7. Let everybody log on again, and watch the fun.

Hmm, that might work in a small network, but you don't have much of a backup plan if it goes wrong. It's a "Big Bang" approach.

There's another way, which gives you a bit more safety, and makes it easier to revert if something goes wrong. It makes use of the processed user group feature of UPM.

The idea is to create a new OU to contain all machines that have been upgraded to UPM v3 and a new group which contains the all the users allowed to use those machines. Initially the OU is empty and the group has no members.

(The OU will most likely be a set of OUs, covering each of the different OS types, but we'll keep it simple for now. We'll call that OU "V2OU")

Step 1: If you have not already done so, ensure that all the users of the UPM v2 machines belong to a group, and configure this group as the processed group in the UPM v2 configuration. Let's call that group "V2Users"

Step 2: Create a "mini-farm" in the new OU. We'll call that new OU "V3OU" - again it could be several OUs, but we're keeping it simple. You could move some unused machines from V2OU into V3OU, or you could build fresh machines. Ensure that V2Users are directed to machines in V2OU, and V3Users are directed into V3OU (this will involve some XA / XD changes outside the scope of this article). Install and configure UPM v3 on all the machines in the mini-farm, with the v3 feature level enabled. For this OU, make the processed group V3Users.

Step 3: Now you can migrate your first department of users. Get them all to log off and remove them from V2Users. Back up their profiles.

Step 4: Add them into V3Users and ensure that their sessions are now directed to V3OU.

Step 5: Let the users from the first department log on. UPM v3 will migrate their profiles on-the-fly to the new structure.

Step 6: Having transferred some users out of V2OU, you should now have a bit of spare capacity in V2OU. Transfer some more machines out of V2OU, upgrade them from UPM v2 to UPM v3, and transfer them in to V3OU. Migrate your next department (steps 3-5)

Step 7: Repeat step 6 above until all machines and all users are using UPM v3.