I came across an issue over the last few weeks regarding regular Personalization failures for users who were browsing via Google Chrome. With help from AppSense support, and Ollie and Hiten from APS, we got this narrowed down to a known issue on the IIS end, so I'm going to quickly write it up so I don't forget about it!
Chrome is quite a popular browser these days, but it is one of these applications that insists on writing quite aggressively to the %LOCALAPPDATA% areas of the user profile. The reason we suffered so badly from this issue was that I didn't do my application analysis properly, and incorrectly assumed that Chrome usage would be fairly niche and limited in scope. However, if I'd studied the existing environment in the correct detail (or had been given the software to do this for me, another story sadly), I would have realized that it was quite possibly the most popular browser in the client environment. Unfortunately, I simply quickly cobbled a Google Chrome Application Group together from some templates on the AppSense forums and handed it off to the person who had requested it.
This was then refined by the requesting user, particularly by implementing the Google Chrome ADMX files which have a setting called "Set user data directory". By using this, the Chrome data directory can be redirected to "${ROAMING_APP_DATA}\Google\Chrome\User Data", effectively storing the Chrome settings in the traditional roaming folder rather than the local one. This is useful for ensuring compatibility with roaming profiles but not perhaps if you're using AppSense, but thought it was worth mentioning as it offers a way to ensure that the Chrome settings are written to a more familiar area.
All seemed well and good at first - users reported that Chrome settings such as logons, extensions and history were persisting as expected. Unfortunately at this point I simply ticked it off the list of items and went on to other issues that needed my attention.
Fast forward a few weeks and we had all sorts of odd issues that seemed to centre specifically around users who were also using Chrome. Settings disappearing - but never in their entirety. Little niggly things like the auto-start for Skype and Outlook colour schemes would reset themselves, sometimes in the middle of the user's session. Whilst not "show-stoppers", they were the sort of issues that not only frustrated the users but also decreased confidence in the solution we were putting together.
First remedial action we took was to turn on silent PersInfo logging for the affected users. Using PersInfo in this way is really one of the most valuable tools in the AppSense administrator's armoury when it comes to Personalization issues. We immediately started reviewing the logs and were surprised to see a whole host of synchronization failures. The pertinent entries are highlighted in bold below in the log snippet:-
16/03/2016 09:57:55,ConfigLoaded,"Personalization configuration loaded for user 'gklp76'\n\r\n\rUser Group is: Test\n\r\n\rThere are 17 Application(s) and 5 Application Group(s) whitelisted in this configuration"
16/03/2016 09:58:01,Information,"Personalization server is online\n\r\n\rActive Site:http://zzzzzzzz"
16/03/2016 10:32:20,SyncDownInProgress,"'Remote Desktop Client' is being synchronized with the client..."
16/03/2016 10:32:21,Information,"mstsc.exe (PID 4768) is being virtualized\n\rManaged as 'Remote Desktop Client (mstsc.exe)' [Grouped as 'Remote Desktop Client']"
16/03/2016 10:32:22,SyncDownComplete,"'Remote Desktop Client' was synchronized to the client\n\r\n\rTransferred 8.91 KB in 0.0 secs at a rate of 1.5 MB/sec"
16/03/2016 10:49:06,SyncUpInProgress,"'Microsoft Office 2016 / 365 Group' is being synchronized to the server..."
16/03/2016 10:49:07,SyncUpComplete,"'Microsoft Office 2016 / 365 Group' was synchronized to the server\n\r\n\rTransferred 90.49 KB in 0.1 secs at a rate of 1.5 MB/sec"
16/03/2016 10:50:39,ErrorOrFailure,"'Remote Desktop Client' failed to synchronize to the server..."
16/03/2016 10:50:39,SyncUpInProgress,"'Remote Desktop Client' is being synchronized to the server..."
16/03/2016 10:50:42,ConfigDownloaded,"Environment Manager has downloaded a new personalization configuration"
16/03/2016 10:52:38,SyncUpInProgress,"'Windows Settings' is being synchronized to the server..."
16/03/2016 10:53:17,ErrorOrFailure,"'Windows Settings' failed to synchronize to the server..."
16/03/2016 10:53:19,ErrorOrFailure,"The Personalization Server could not be reached. Link state is now offline.\n\r\n\rFailed to upload Windows Settings to the server\n\r\n\rRetrying at 10:54:34"
16/03/2016 10:54:07,SyncDownInProgress,"'Microsoft Office 2016 / 365 Group' is being synchronized with the client..."
16/03/2016 10:54:07,ErrorOrFailure,"'Microsoft Office 2016 0x2F; 365 Group' failed to synchronize to the client because the link state is currently offline..."
16/03/2016 10:54:08,Information,"OUTLOOK.EXE (PID 9084) is being virtualized\n\rManaged as 'Microsoft Office Outlook 2016' [Grouped as 'Microsoft Office 2016 / 365 Group']"
16/03/2016 10:54:35,Information,"Personalization server is now online\n\r\n\rActive Site: http://zzzzzzzzz"
16/03/2016 10:54:48,SyncUpInProgress,"'Microsoft Office 2016 / 365 Group' is being synchronized to the server..."
16/03/2016 10:54:50,SyncUpComplete,"'Microsoft Office 2016 / 365 Group' was synchronized to the server\n\r\n\rTransferred 104.69 KB in 0.0 secs at a rate of 2.2 MB/sec"
16/03/2016 10:54:54,SyncDownInProgress,"'Microsoft Office 2016 / 365 Group' is being synchronized with the client..."
16/03/2016 10:54:55,SyncDownComplete,"'Microsoft Office 2016 / 365 Group' was synchronized to the client\n\r\n\rTransferred 65.44 KB in 0.0 secs at a rate of 2.2 MB/sec"
16/03/2016 10:54:55,Information,"OUTLOOK.EXE (PID 3368) is being virtualized\n\rManaged as 'Microsoft Office Outlook 2016' [Grouped as 'Microsoft Office 2016 / 365 Group']"
16/03/2016 10:55:36,ErrorOrFailure,"The Personalization Server could not be reached. Link state is now offline.\n\r\n\rFailed to upload Windows Settings to the server\n\r\n\rRetrying at 10:56:51"
This was quite concerning on a number of levels, both for the amount of failures and the resulting loss of settings. It quickly became clear that certain settings would fail to save but then these settings would be overwritten with older data from the Personalization Server at next logon, giving the appearance of "disappearing" config items.
The next stage was to gather some debug logs, which allowed us to see that the users with Chrome were saving quite large volumes of data (up to 80 or 90MB, in some cases).
What the guys from AppSense support and APS mentioned was that there was an arbitrary limit, both on file sizes and file numbers, within IIS. Upload of a file larger than 30MB or a number of files exceeding 1000 would hit this limit, which was put in by design to reduce the potential for denial-of-service attacks upon IIS systems. Debug logs seemed to indicate that the Chrome users were saving data that exceeded both of these limits.
So to address this, we moved twofold - firstly, to increase the available limits, and secondly, to streamline the Chrome settings so that they would avoid saving datasets of this size and volume.
To increase the file upload size limit, we performed the following steps (also summarized in an AppSense TechNote):-
- Launch IIS (Start | Run | inetmgr)
- Select the Personalization Server Website (or the Default Website, if your Personalization Server is the only website configured on the IIS host)
- Select the 'Request Filtering' feature from the list
- From the actions pane on the right, select 'Edit Feature Settings'
- Change the 'Maximum Allowed Content Length) value to a more suitable number (in this case, we simply doubled it to 60MB - see image below)
- Restart IIS (run iisreset from an elevated command prompt on the IIS server)
Increasing the amount of files is slightly different, requiring an edit of the web.config file. Again, this is summarized in an AppSense TechNote.
- Run an elevated instance of Notepad
- Open the file %PROGRAMFILES%\AppSense\Environment Manager\Personalization Server\web.config (you may have to change the filetype setting in the Open dialog to All)
- Add the following text to the <appSettings> section - <add key="aspnet:MaxHttpCollectionKeys" value="1200" /> (see image below)
- Save the file
- Restart IIS (run iisreset from an elevated command prompt on the IIS server)
You can set the number to any size you require, I've just used 1200 as a good starting point over the initial limit of 1000.
The next step was to review the PersInfo logs and see if they had changed. As you can see from the excerpt below, they look far healthier:-
16/03/2016 13:03:16,ConfigLoaded,"Personalization configuration loaded for user 'gklp76'\n\r\n\rUser Group is: Test\n\r\n\rThere are 18 Application(s) and 6 Application Group(s) whitelisted in this configuration"
16/03/2016 13:03:24,Information,"Personalization server is online\n\r\n\rActive Site: http://cisvirappvir06.mds.ad.dur.ac.uk/PersonalizationServer"
16/03/2016 13:03:49,SyncDownInProgress,"'Google Chrome Group' is being synchronized with the client..."
16/03/2016 13:03:49,SyncDownComplete,"'Google Chrome Group' was synchronized to the client\n\r\n\rTransferred 442.09 KB in 0.2 secs at a rate of 5.9 MB/sec"
16/03/2016 13:04:03,Information,"chrome.exe (PID 9488) is being virtualized\n\rManaged as 'Google Chrome' [Grouped as 'Google Chrome Group']"
16/03/2016 13:04:46,SyncUpInProgress,"'Google Chrome Group' is being synchronized to the server..."
16/03/2016 13:04:50,SyncUpComplete,"'Google Chrome Group' was synchronized to the server\n\r\n\rTransferred 2.32 MB in 1 secs at a rate of 2.2 MB/sec"
16/03/2016 13:06:02,SyncDownInProgress,"'Microsoft Office 2016 / 365 Group' is being synchronized with the client..."
16/03/2016 13:06:03,SyncDownComplete,"'Microsoft Office 2016 / 365 Group' was synchronized to the client\n\r\n\rTransferred 124.16 KB in 0.1 secs at a rate of 2.1 MB/sec"
16/03/2016 13:06:06,Information,"OUTLOOK.EXE (PID 9468) is being virtualized\n\rManaged as 'Microsoft Office Outlook 2016' [Grouped as 'Microsoft Office 2016 / 365 Group']"
16/03/2016 13:07:57,SyncUpInProgress,"'Windows Settings' is being synchronized to the server..."
16/03/2016 13:07:58,SyncUpComplete,"'Windows Settings' was synchronized to the server\n\r\n\rTransferred 1.78 MB in 0.3 secs at a rate of 2.9 MB/sec"
16/03/2016 13:09:16,SyncDownInProgress,"'Google Chrome Group' is being synchronized with the client..."
16/03/2016 13:09:17,Information,"chrome.exe (PID 2904) is being virtualized\n\rManaged as 'Google Chrome' [Grouped as 'Google Chrome Group']"
16/03/2016 13:09:18,SyncDownComplete,"'Google Chrome Group' was synchronized to the client\n\r\n\rTransferred 1.19 MB in 0.3 secs at a rate of 3.2 MB/sec"
16/03/2016 13:11:18,SyncUpInProgress,"'Google Chrome Group' is being synchronized to the server..."
16/03/2016 13:11:21,SyncUpComplete,"'Google Chrome Group' was synchronized to the server\n\r\n\rTransferred 2.09 MB in 0.9 secs at a rate of 3.3 MB/sec"
16/03/2016 13:11:28,SyncDownInProgress,"'Google Chrome Group' is being synchronized with the client..."
16/03/2016 13:11:30,Information,"chrome.exe (PID 9264) is being virtualized\n\rManaged as 'Google Chrome' [Grouped as 'Google Chrome Group']"
16/03/2016 13:11:31,SyncDownComplete,"'Google Chrome Group' was synchronized to the client\n\r\n\rTransferred 1.24 MB in 0.2 secs at a rate of 3.8 MB/sec"
16/03/2016 13:11:52,SyncUpInProgress,"'Google Chrome Group' is being synchronized to the server..."
16/03/2016 13:11:54,SyncUpComplete,"'Google Chrome Group' was synchronized to the server\n\r\n\rTransferred 2.03 MB in 0.3 secs at a rate of 4.0 MB/sec"
16/03/2016 13:11:59,SyncDownInProgress,"'Google Chrome Group' is being synchronized with the client..."
16/03/2016 13:12:00,Information,"chrome.exe (PID 4756) is being virtualized\n\rManaged as 'Google Chrome' [Grouped as 'Google Chrome Group']"
16/03/2016 13:12:01,SyncDownComplete,"'Google Chrome Group' was synchronized to the client\n\r\n\rTransferred 1.24 MB in 0.3 secs at a rate of 4.1 MB/sec"
16/03/2016 13:12:17,SyncUpInProgress,"'Google Chrome Group' is being synchronized to the server..."
16/03/2016 13:12:19,SyncUpComplete,"'Google Chrome Group' was synchronized to the server\n\r\n\rTransferred 1.99 MB in 0.4 secs at a rate of 5.7 MB/sec"
16/03/2016 13:14:59,SyncUpInProgress,"'Windows Settings' is being synchronized to the server..."
16/03/2016 13:15:02,SyncUpComplete,"'Windows Settings' was synchronized to the server\n\r\n\rTransferred 1.70 MB in 0.6 secs at a rate of 4.8 MB/sec"
With this looking so much improved, we finally changed the Chrome settings to a more suitable configuration so that the mass of data would not be accumulated in future (thanks to Hiten for the template):-
Registry includes
HKEY_CURRENT_USER\Software\Google
HKEY_CURRENT_USER\Software\MozillaPlugins
Registry excludes
HKEY_CURRENT_USER\Software
Folder includes
{CSIDL_APPDATA}\Google\Chrome\User Data\Default\Storage
{CSIDL_APPDATA}\Google\Chrome\User Data\Default\GCM Store
{CSIDL_APPDATA}\Google\Chrome\User Data\Default\Sync Data
{CSIDL_APPDATA}\Google\Chrome\User Data\Default\Local Extension Settings
{CSIDL_APPDATA}\Google\Chrome\User Data\Default\Extensions
{CSIDL_APPDATA}\Google\Chrome\User Data\Default\Session Storage
{CSIDL_APPDATA}\Google\Chrome\User Data\Default\Sync Extension Settings
Folder excludes
{CSIDL_PROFILE}
File includes
{CSIDL_APPDATA}\Google\Chrome\User Data\Default\Shortcuts
{CSIDL_APPDATA}\Google\Chrome\User Data\Default\Login Data-journal
{CSIDL_APPDATA}\Google\Chrome\User Data\Default\History-journal
{CSIDL_APPDATA}\Google\Chrome\User Data\First Run
{CSIDL_APPDATA}\Google\Chrome\User Data\Default\Favicons-journal
{CSIDL_APPDATA}\Google\Chrome\User Data\Default\Login Data
{CSIDL_APPDATA}\Google\Chrome\User Data\Default\Last Session
{CSIDL_APPDATA}\Google\Chrome\User Data\Default\Cookies
{CSIDL_APPDATA}\Google\Chrome\User Data\Default\History
{CSIDL_APPDATA}\Google\Chrome\User Data\Default\Top Sites
{CSIDL_APPDATA}\Google\Chrome\User Data\Default\Last Tabs
{CSIDL_APPDATA}\Google\Chrome\User Data\Default\Favicons
{CSIDL_APPDATA}\Google\Chrome\User Data\Default\Origin Bound Certs
{CSIDL_APPDATA}\Google\Chrome\User Data\Local State
{CSIDL_APPDATA}\Google\Chrome\User Data\Default\Web Data
{CSIDL_APPDATA}\Google\Chrome\User Data\Default\Web Data-journal
{CSIDL_APPDATA}\Google\Chrome\User Data\Default\Bookmarks
{CSIDL_APPDATA}\Google\Chrome\User Data\Default\Current Tabs
{CSIDL_APPDATA}\Google\Chrome\User Data\Default\Origin Bound Certs-journal
{CSIDL_APPDATA}\Google\Chrome\User Data\Default\Top Sites-journal
{CSIDL_APPDATA}\Google\Chrome\User Data\Default\Preferences
{CSIDL_APPDATA}\Google\Chrome\User Data\Default\Current Session
{CSIDL_APPDATA}\Google\Chrome\User Data\Default\Cookies-journal
{CSIDL_APPDATA}\Google\Chrome\User Data\Default\Archived History
{CSIDL_APPDATA}\Google\Chrome\User Data\Default\Shortcuts-journal
Note that we have used {CSIDL_APPDATA} in all the file and folder includes because of the Chrome ADMX we configured to switch the user data directory. In a "normal" Chrome installation, all of the references to {CSIDL_APPDATA} would need to be changed to {CSIDL_LOCAL_APPDATA}.
As you can see from the screenshot below, the amount of Chrome data has significantly decreased straight away
And so far, we seem to have eliminated a lot of the niggly issues!
So some lessons to be learned:-
- Understand your environment - I made a fatal assumption by postulating that Chrome use would be quite limited
- Don't rely on config templates you grab from forums to be suitable for your environment
- Always monitor the size of your profiles (this problem would have been spotted much earlier had we been doing this)
- PersInfo is an excellent tool for proactively monitoring Personalization status
- Go and have a look on the AppSense support site (the TechNotes were there describing these exact issues)
- The AppSense support and APS guys are always helpful and can always teach you something new!
Finally, a quick note - I'm very busy with conference stuff and technical articles at the minute so apologies for the dearth of content recently. I've got stuff in the pipeline regarding Modern Apps, Start Tiles, more Windows 10 and hopefully some network stuff, so please bear with me while I try to clear my (virtual) desk :-)