Windows Search in Server 2019 and Multi-Session Windows 10

As always in our industry, small changes in one platform can result in significant impact across what is “standard practice” in many other areas. This week’s lucky contender is Windows Search.

Windows Search in both Windows Server 2019 and Windows 10 Multi-Session has changed how it operates, introducing the concept of per-user search natively into the Search Service. This is fundamentally different from previous versions (namely Windows Server 2016 etc) and changes how we need to think about search roaming with supporting technologies such as FSLogix Containers and Citrix UPM.

The biggest change is that the Windows Search index is now stored per user in the user profile, specifically C:\Users\%Username%\AppData\Roaming\Microsoft\Search\Data\Applications\{UserSID}\{UserSID}.edb. On each login, the Windows Search process creates a new instance of the search database for the user based on the existing EDB. If no EDB file exists, a new one is created by the Operating System.

User Based Search Index

There are two components associated with the Search functionality split into individual processes:

  • SearchIndexer.exe: This process runs under the “NT AUTHORITY\SYSTEM” account and accesses the index database for each user (profile based)
  • SearchProtocolHost.exe: an instance of this process runs per user and is a child process of SearchIndexer.exe. This process handles amongst other things, the indexing of the Outlook OST file, not the searching of the index file, but the actual index creation of it making content available to search. Searching should be operational with or without this process running. When new items need to be indexed, the process will run, once finished, it will stop

The impacts here, should you have dealt with FSLogix or Citrix UPM Search roaming should start to become obvious when you think about how these solutions tackle the Search side of things (they rip out a component of the index and store them within a container). The long and short of it is as follows:

  • For FSLogix environments, you must NOT enable search roaming within the FSLogix GPO. Specifically, you want SearchRoam set to 0 in the registry. This needs to be reversed in your master image should you have enabled it previously, in fact, given the what we know of Search challenges and the attempted hooks into it, I would be tempted to suggest that should you have enabled search roaming in either Windows Server 2019 or Windows 10 multi-session previously, then revert the setting, reinstall Microsoft Office and make sure that the event logs are clear from a Windows Search perspective
  • For Citrix UPM environments, the following article has been released which is quite confusing. TLDR, the key it’s discussing should not be set to anything other than 0, and ideally shouldn’t exist as a whole in Windows Server 2019. Citrix UPM configurations will need to cater for the new location of the Search Index, however, the OST container can stay in place and do its thing. I haven’t had time to test, but I would assume that the new search location in the user profile is an ideal candidate for the UPM container feature

Unfortunately, Windows Search is an ongoing challenge and there is a fair number of customers who are experiencing issues with the native multi-user search capability in both Windows 10 Multi-Session and Windows Server 2019. Some are experiencing repeated crashing of the service others are finding that search index files are not released on logoff resulting in locked files and corrupt indexes.

In the below walkthrough, I will try and outline what happens with Windows Search in Server 2019, and where things fall apart, along with a temporary resolution which may help until the problems are properly addressed by Microsoft.

First of all, an outline of the environment I am using to demonstrate the situation

  • Windows Server 2019 Datacenter, latest rollup at the time of writing
  • FSLogix release 2.9.7237.48865
  • Profile and Office Container configured
  • Microsoft Office 365 ProPlus, x64, Semi-Annual channel
  • No PVS or MCS provisioning

Secondly, below is an outline of some of the event log entries we will be referring to:

IDDetail
5Windows Search Service has created default configuration for new user ‘KINDO\JKindon5’
102SearchIndexer (4400,P,98) { S-1-5-21-2397015974-2202110191-2245630456-1134}: The database engine (10.00.17763.0000) is starting a new instance (3).
105SearchIndexer (4400,D,0) S-1-5-21-2397015974-2202110191-2245630456-1134: The database engine started a new instance (3). (Time=0 seconds)
637SearchIndexer (4400,D,35) S-1-5-21-2397015974-2202110191-2245630456-1134: New flush map file “C:\Users\JKindon5\AppData\Roaming\Microsoft\Search\Data\Applications\{ S-1-5-21-2397015974-2202110191-2245630456-1134}\{ S-1-5-21-2397015974-2202110191-2245630456-1134}.jfm” will be created to enable persisted lost flush detection.
325SearchIndexer (4400,D,35) { S-1-5-21-2397015974-2202110191-2245630456-1134}: The database engine created a new database (6, C:\Users\JKindon5\AppData\Roaming\Microsoft\Search\Data\Applications\{ S-1-5-21-2397015974-2202110191-2245630456-1134}\{ S-1-5-21-2397015974-2202110191-2245630456-1134}.edb). (Time=0 seconds)
103SearchIndexer (6672,T,97) { S-1-5-21-2397015974-2202110191-2245630456-1134}: The database engine stopped the instance (3).
2Unable to remove Windows Search Service indexed data for user ‘KINDO\JKindon5’ in response to user profile deletion.  Error code 0x80004002.
482SearchIndexer (3768,D,0) S-1-5-21-2397015974-2202110191-2245630456-1134: An attempt to write to the file “C:\Users\JKindon5\AppData\Roaming\Microsoft\Search\Data\Applications\S-1-5-21-2397015974-2202110191-2245630456-1134\S-1-5-21-2397015974-2202110191-2245630456-1134.jfm” at offset 0 (0x0000000000000000) for 8192 (0x00002000) bytes failed after 0.000 seconds with system error 21 (0x00000015): “The device is not ready. “.  The write operation will fail with error -1022 (0xfffffc02).  If this error persists then the file may be damaged and may need to be restored from a previous backup.
492SearchIndexer (9200,D,0) S-1-5-21-2397015974-2202110191-2245630456-1134: The logfile sequence in “C:\Users\JKindon5\AppData\Roaming\Microsoft\Search\Data\Applications\S-1-5-21-2397015974-2202110191-2245630456-1134\” has been halted due to a fatal error.  No further updates are possible for the databases that use this logfile sequence.  Please correct the problem and restart or restore from backup.
439SearchIndexer (3768,T,97) S-1-5-21-2397015974-2202110191-2245630456-1134: Unable to write a shadowed header for file C:\Users\JKindon5\AppData\Roaming\Microsoft\Search\Data\Applications\S-1-5-21-2397015974-2202110191-2245630456-1134\S-1-5-21-2397015974-2202110191-2245630456-1134.edb. Error -1022.
104SearchIndexer (3768,T,97) S-1-5-21-2397015974-2202110191-2245630456-1134: The database engine stopped the instance (2) with error (-1022).
103SearchIndexer (3768,T,97) Windows: The database engine stopped the instance (0).
300SearchIndexer (10076,R,98) S-1-5-21-2397015974-2202110191-2245630456-1134: The database engine is initiating recovery steps.
301SearchIndexer (10076,R,98) S-1-5-21-2397015974-2202110191-2245630456-1134: The database engine has finished replaying logfile  C:\Users\JKindon5\AppData\Roaming\Microsoft\Search\Data\Applications\S-1-5-21-2397015974-2202110191-2245630456-1134\edb.jtx.
302SearchIndexer (10076,U,98) S-1-5-21-2397015974-2202110191-2245630456-1134: The database engine has successfully completed recovery steps.
326SearchIndexer (9808,D,50) S-1-5-21-2397015974-2202110191-2245630456-1104: The database engine attached a database (2, C:\Users\JKindon\AppData\Roaming\Microsoft\Search\Data\Applications\S-1-5-21-2397015974-2202110191-2245630456-1104\S-1-5-21-2397015974-2202110191-2245630456-1104.edb). (Time=0 seconds)

My findings here are an aggregation of existing posts, as well as feedback from colleagues in the World of EUC slack channel. Testing has been validated in my own environments.

First Logon for the user

On the first logon for a new user, the following event logs entries will occur, basically detecting a new user without an index, creating a new database, configuring and leveraging the new instance as below:

Event ID 5: Windows Search Service has created the default configuration for new user “Kindo\JKindon5”

Event ID 102: SearchIndexer (4400,P,98) { S-1-5-21-2397015974-2202110191-2245630456-1134}: The database engine (10.00.17763.0000) is starting a new instance (3)

Event ID 105: SearchIndexer (4400,D,0) S-1-5-21-2397015974-2202110191-2245630456-1134: The database engine started a new instance (3). (Time=0 seconds)

Event ID 637: SearchIndexer (4400,D,35) S-1-5-21-2397015974-2202110191-2245630456-1134: New flush map file “C:\Users\JKindon5\AppData\Roaming\Microsoft\Search\Data\Applications\{ S-1-5-21-2397015974-2202110191-2245630456-1134}\{ S-1-5-21-2397015974-2202110191-2245630456-1134}.jfm” will be created to enable persisted lost flush detection

Event ID 325: SearchIndexer (4400,D,35) { S-1-5-21-2397015974-2202110191-2245630456-1134}: The database engine created a new database (6, C:\Users\JKindon5\AppData\Roaming\Microsoft\Search\Data\Applications\{ S-1-5-21-2397015974-2202110191-2245630456-1134}\{ S-1-5-21-2397015974-2202110191-2245630456-1134}.edb). (Time=0 seconds)

In summary, in a healthy operational state, for a new user, the following event log patterns will occur on logon:

  • 5 = Search Creates a default configuration
  • 102 = Search Indexer creates a new instance for the user
  • 105 = Search Indexer starts the new instance for the user
  • 637 = New flush map file created for the user
  • 325 = Search Indexer creates the new database

User Logoff

When the user logs off for the first time, the following event entry is logged:

Event ID 103: Event 103: SearchIndexer (6672,T,97) { S-1-5-21-2397015974-2202110191-2245630456-1134}: The database engine stopped the instance (3)

And this is where things go pear-shaped, you will most probably find the 103 entry is quickly followed by this one:

Event ID 2: Unable to remove Windows Search Service indexed data for user ‘KINDO\JKindon5’ in response to user profile deletion.  Error code 0x80004002

In this state, when the user next connects, they are going to be hit with a problem. What we are looking for, is a happy combination of the following events when the user logs on for the second time:

  • 102 = Search Indexer creates a new instance for the user
  • 105 = Search Indexer starts the new instance for the user
  • 326 = Search Indexer attaches an existing database

What we end up with though due to the above event ID 2 on Logoff, is the following:

Event ID 482: SearchIndexer (3768,D,0) S-1-5-21-2397015974-2202110191-2245630456-1134: An attempt to write to the file “C:\Users\JKindon5\AppData\Roaming\Microsoft\Search\Data\Applications\S-1-5-21-2397015974-2202110191-2245630456-1134\S-1-5-21-2397015974-2202110191-2245630456-1134.jfm” at offset 0 (0x0000000000000000) for 8192 (0x00002000) bytes failed after 0.000 seconds with system error 21 (0x00000015): “The device is not ready. “.  The write operation will fail with error -1022 (0xfffffc02).  If this error persists then the file may be damaged and may need to be restored from a previous backup.

The root cause of this is due to the following handles still being kept open by the SearchIndexer.exe process

Performance Monitor locked files

These files are held open; however, the physical files have gone (detached) and thus we loop.

Locked files with no access

If the user was to log back on at this point, we will experience some problematic symptoms and events:

Outlook indexing will present as below:

Failed Outlook Index

And no files in the %AppData%\Microsoft\Search\Data\Applications\{UserSID} directories will be touched

Restarting the search service

If we restart the Windows Search Service, a few things happen:

All existing handles are closed for SearchIndexer operations, both for any logged off users in a fail state, and for any other user on the Server, and the following event logs are written

  • Event ID 439: SearchIndexer (3768,T,97) S-1-5-21-2397015974-2202110191-2245630456-1134: Unable to write a shadowed header for file C:\Users\JKindon5\AppData\Roaming\Microsoft\Search\Data\Applications\S-1-5-21-2397015974-2202110191-2245630456-1134\S-1-5-21-2397015974-2202110191-2245630456-1134.edb. Error -1022.
  • Event ID 104: SearchIndexer (3768,T,97) S-1-5-21-2397015974-2202110191-2245630456-1134: The database engine stopped the instance (2) with error (-1022).
  • Event ID 103: SearchIndexer (3768,T,97) Windows: The database engine stopped the instance (0).
  • Event ID: 102, 105, 326: (happy combination representing success)
  • Event ID 300: SearchIndexer (10076,R,98) S-1-5-21-2397015974-2202110191-2245630456-1134: The database engine is initiating recovery steps.
  • Event ID 301: SearchIndexer (10076,R,98) S-1-5-21-2397015974-2202110191-2245630456-1134: The database engine has finished replaying logfile C:\Users\JKindon5\AppData\Roaming\Microsoft\Search\Data\Applications\S-1-5-21-2397015974-2202110191-2245630456-1134\edb.jtx.
  • Event ID 302: SearchIndexer (10076,U,98) S-1-5-21-2397015974-2202110191-2245630456-1134: The database engine has successfully completed recovery steps
  • Event ID: 105, 326 (Search index is OK again)

Outlook indexing is now OK

Outlook Indexing in Action

Along with successful alterations to the %AppData%\Microsoft\Search\Data\Applications\{UserSID} directories.

Fixed it would seem. So, logically here, we know that the resolution is to restart the Windows Search Service based on the failure Event, which is represented by Event ID 2. Specifically:

  • Event Log: Application
  • Event Level: Error
  • Event Source: Search-ProfileNotify
  • Event ID: 2
  • Event Data: Unable to remove Windows Search Service indexed data for user ‘KINDO\JKindon5’ in response to user profile deletion.  Error code 0x80004002

By implementing a scheduled task with the above Event ID as the trigger, and a simple configuration of PowerShell to restart the search service as the action:

Scheduled Task – General Properties
Scheduled Task – Trigger Properties
Scheduled Task – Action Properties

We can successfully clear the errors on log-off, release the files, and ensure that when the user logs back in, their index is good to go. The process will now look like the below when a user logs off:

  • Event ID: 103 for the user logging off
  • Event ID: 2 -> This triggers the restart of the service, which also triggers an Event ID: 103 for all other users on the Server
  • Event ID: 102, 105, 326 for all remaining user accounts on the server

Based on testing so far, this process does not hurt user experience, inclusive of existing Outlook sessions with existing search indexes, however, your mileage will vary, and this is at the end of the day, simply a workaround for a problem that needs to be fixed.

Summary

There are multiple streams of issues associated with native multi-user search currently, a running an interesting stream is located here on the Microsoft forums which pointed to the end resolution of a scheduled restart of the Search Service on the specific Event ID trigger, however, there have been multiple discussions around the World of EUC slack channel with people willing to test and share results – namely Kasper Johansen, Mike Streetz, Dennis Mohrmann

26 thoughts on “Windows Search in Server 2019 and Multi-Session Windows 10

Add yours

  1. Hi James,

    Thanks for your post.

    We are using 2019 Citrix session host with FsLogix with profile containers.
    If i unedrstand this post we should disable Search roaming within FsLogix?

    We curently are running the following settings

    Enable search roaming: Enabled, Multi-user search
    Store search database in profile container: Enabled, Multi-user search

    We also found this post on the citrix forum that should help to fix search
    https://support.citrix.com/article/CTX270433

    and in your previous post you stated we sould configure de following register key
    https://jkindon.com/2020/01/06/fslogix-containers-search-index-considerations-and-troubleshooting/

    “Multi-User Search reference

    When the multi-user search is enabled, the following registry value is changed from “0” to “1”

    Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Windows Search\Preferences\EnablePerUserCatalog(REG_DWORD)”

    This location is in a diferent location then the Citrix Article.

    So the correct setting would be the following.

    Enable search roaming: Disable (fix wit reg key)
    Store search database in profile container: Disable (fix wit reg key)

    Set the key from the Citrix Article
    HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Windows Search\EnablePerUserCatalog(REG_DWORD:000000)

    Do not apply the regkey

    Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Windows Search\Preferences\EnablePerUserCatalog(REG_DWORD)”

    Please let me know if this is the correct setting.

    Thanks!

    Liked by 1 person

    1. – Yes disable search roaming, you don’t need it
      – Ignore the UPM article
      – The key we mentioned in the post previously was automatically set, it did not require configuration
      – Correct configuration is simple now – don’t do anything with search in Server 2019, make sure FSLogix is NOT doing anything with it. That’s it

      Like

      1. James,

        Thank i’m curently testing this and it seems like it is working 🙂

        Would you still recommend the restart of the services task to implement this?

        Regards!
        Eric

        Like

      2. Error presents on log off – if you see the same errors in the log and experience the same symptoms, then yep, I’d put the fix in for now – but if not then no need

        Like

  2. Do you have these warning messages in event viewer (Application)

    After opening Outlook:

    EventID: 3036
    Content source the crawl could not be completed.
    Context: Application, SystemIndex Catalog
    Details: The parameter is incorrect. (HRESULT : 0x80070057) (0x80070057)

    Like

    1. I also have this warning message:
      EventID: 1534
      The Delete event profile notification (part {DE3F3560-3032-41B4-B6CF-F703B1B95640}) failed. The error code is See Tracelogging for error details.

      Like

  3. hi james, does this also apply to fslogix office containers only? what about windows search when using upm and fslogix office containers only?

    kind regards,

    sebastian

    Like

  4. Great article, however I’m a bit confused. Is the reg key from Citrix needed for w10 1809 desktops with fsl? Or does searchroam = 0 handle that for you? thank you james!

    Like

    1. Hey Nick – multi session win10 and server 2019 need nothing touched out of the box, I believe the key from Citrix is a reversal of a setting that may have been applied at some point, but isn’t required by default and should be ignored unless needed (as I understand it)

      Like

  5. Great post for a problem I thought we fixed back in March this year. 2019/Citrix and we remove local profiles on logout as we roam them – well, that stopped again due to this problem and now have a mess of .000, 001, 002 profiles to clean up. Is this a timing issue of the search service lock so user-profile cannot properly delete on logout? If so – I was going to try to add the search service restart on each user logout and hope the user-profile will have no issues with locked EDB files.

    Like

    1. Sounds like the same problem to me, definitely had issues with search not releasing the index – restart fixes it (still feels nasty having to do it)

      Like

  6. For configuration with RDS on 2019 with Profile & Office Container, using Windows native 2019 search Outlook 365 account are correctly indexed and search works…but not for PST file…do you have same tips/suggestion?

    Like

      1. To be honest I have no idea on PST files – potentially you may need to alter the outlook indexing options? But I’m guessing here as I don’t deal with them 🙂

        Like

  7. So when you leave the Default RoamSearch to 0, I understand Windows 2019 will handle it now. Does this mean you don’t set the 2 settings for Store Search DB in office 365 container and store search DB in profile container? I assume now it would be in the users C:\Users\%Username%\AppData\Roaming\Microsoft\Search\Data\Applications\{UserSID}\{UserSID}.edb. Where does outlook go now?

    Like

  8. I’m seeing the same behaviour that some already mentioned: mail search is working as expected, but calendar and contacts search is not working.
    Everything is configured ok, eventlog seems ok. Already tried to defrag the user index database (stop service, esentutl /d , start service). That resolved the warnings of ESENT eventid 642, but still no results for the contact searches.

    Like

      1. Helps or happens? 🙂
        I get it to work for Outlook and Onenote but asid from this my list of indexed locations is completely empty and I am not able to add more locations e.g. my local drives.

        Like

      2. This purely stops the issue of Search index corruption – I haven’t delved into the depths of what happens with indexing etc – I am just using this to stop the chaos of repeated broken searches due to the lock

        Like

  9. I have been able to get indexing working fine following this guide – great post! The problem I do seem to run into is at log off and then log on, the indexing seems to take place again. I have been able to get to a position on one VDA where once it’s indexed, it stays. When the user then logons to another multi-session host, re-indexing begins again.

    Has anyone had this? At a bit of a loss – unsure if this is expected behaviour.

    Like

  10. Having the exact same issue on a new Windows 2019 build with Office 365 and Profile containers. Search index breaks every time after first logon.

    The workaround to restart the service at user logoff is working but feels wrong. Surely with Fslogix owned by MS now they should fix this by now.

    Maybe time for Microsoft to ditch the abomination that is Windows search as it’s caused issues since office stopped using it’s own search DB. Might be time for Microsoft to look at bringing the Office search back!

    Has anyone got any updates on where we are with any kind of fix for this issue? we’re also running the latest Fslogix agent from a couple weeks ago and its still an issue in that.

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Create a website or blog at WordPress.com

Up ↑

%d bloggers like this: