Odd file behavior in virtual branches, perhaps after installing Windows Updates #70
Description
Hello,
My organization has been using a ProjFS-based file syncing setup since late last year. Since around March of this year, the number of machines using this system has grown from ~400 to ~700 now. This is the primary way people get and interact with the files they work on constantly every day.
I give this background because, as of yesterday, I've now seen what appears to be a new and very odd issue for a 2nd time (on a different machine) within the last ~week.
Nothing about the code managing ProjFS or the backing store information has changed in any substantial way for a few months now (some minor inspection improvements is about it, nothing that impacts interactions with ProjFS).
The only potentially interesting data point I have is that a few days before the first case our IT group rolled out a series of Windows Updates. That said, in both cases I had confirmed that the users did NOT have any pending windows updates at the time (they had installed and rebooted since), as that can cause all kinds of odd behavior.
The updates installed were KB5004245, KB5003539, KB5004748, KB5004772. The last batch of windows updates were installed a month prior (before this issue was first reported).
In both cases, the odd behavior mysteriously went away after a system restart.
Because that appears to fix it, and because our users sometimes like to attempt to fix things themselves and not let us know, it is possible that there have been other cases of this that I'm just not aware of. That said, the timing of 2 cases within a week, and just after the updates, combined with the reboot-as-a-fix, seems a little suspicious.
Anyway, on to the odd behavior...
The user notices the issue when a ProjFS file fails to be read, and so causes some user application to fail to start (if it is an exe/dll/etc) or fail to load some content file.
This behavior, though, only impacts a subset of the ProjFS files under a given virtual root, while other virtual files work perfectly fine (I am able to trigger placeholder creation, hydration, PrjDeleteFile back to virtual and repeat with other files). In the first case, all ProjFS files in a given directory were impacted (there were some full files mixed in the directory that worked normally).
I'm not sure if it was all files in the same directory for the 2nd case as the user rebooted before I could test that case thoroughly.
In the first case, though, I did have time to try a handful of things on these badly behaving files, with odd results:
- The files are listed in File Explorer
- I was able to right-click on them and open file properties
- Attempting to delete the file (hitting delete key in File Explorer), though, failed oddly with "Item Not Found" "This is no longer located at XXX"
...then, in C#, I tried:
var path = // bad path
var fi = new FileInfo(path);
// fi.Exists => false
// fi.Attributes => 0xFFFFFFFF
var di = new DirectoryInfo(Path.GetDirectoryName(path));
var file = di.EnumerateFiles().First(f => string.Compare(f.FullName, path) == 0);
// file is NOT NULL ... it was enumerated
// file.Exists => TRUE !!
// file.Attributes => 0x00440001 (RECALL_ON_DATA_ACCESS | RECALL_ON_OPEN | READONLY) -- looks like a placeholder, but no REPARSE_POINT attribute??
using (var fs = file.OpenRead())
{
fs.ReadByte();
}
// throws FileNotFoundException
FindFirstFileEx w/ FIND_FIRST_EX_ON_DISK_ENTRIES_ONLY lists no items in the directory !? (full files had been deleted at this point, leaving only the badly behaving files in the directory)
FindFirstFileEx without that flag lists the bad file with the above attributes
Calling CreateFile() directly to try and get any reparse point data also failed to invalid path.
I also tried:
- Stopping and restarting the virtual root
- Clearing the ProjFS negative path cache
... and re-running all of the above tests, all with the same result.
Throughout all of this, my ProjFS application logged no errors, and I suspect got not callbacks (though I can't fully prove that).
The file in question had been newly created on this machine the day before, then submitted to version control and the user had later sync'd. So, the file should have transitioned from Full-Writable -> Full-ReadOnly -> PrjDeleteFile to make it virtual -> Placeholder (app-triggered) and broken. All of this occurred a day or two after taking the windows updates.
I didn't notice any obvious critical or error system events in that time period.
And then, as a bit of a last ditch, I asked the user to reboot ... and after they did, the badly behaving placeholder files suddenly worked perfectly!
Rebooting also "magically" fixed the 2nd case.
Any ideas?
Any other things you'd want me to try if/when I see the next case?
Thanks!