Incremental builds with MSBuild/Visual Studio are broken

[Un1oR]

I think it problem related with #33, but a bit more complicated.
When I try incremental build, all solution fully rebuilds. I found it not happens if I comment body of PersistentJSONDict.save function. As I understand, MSBuild tracks writing a statistics file and mark projects in solution as outdated.
Affects only solutions with two or more projects with build dependencies between it!

[frerich]

Can you provide a sample project which triggers this? On first sight, it sounds like another incarnation of https://github.com/frerich/clcache/wiki/Caveats#no-cache-hits-when-building-via-visual-studio-ide-or-msbuild -- i.e. the file tracking feature needs to be disabled. Does that help with incremental builds, too? If so, I guess that Wiki page should be extended.

[Un1oR]

Disabling tracking is incompatible with incremental builds, as far I'm understand.
I'll add my test project a little bit later.

[Un1oR]

https://github.com/Un1oR/clcache-incremental-test
Interesting thing I found. It affects only hot cache case. If I run build.bat first three times I haven't rebuilds at result. But if I remove build directories and run it againt, I will always have full rebuild.
My log https://gist.github.com/Un1oR/81062c4a6e04ca3ec072ff986e0718e8.

[Un1oR]

In addition to my previous comment. If I comment registerCacheHit function body a problem goes away.

[frerich]

I suspect what happens is that due to the file tracing feature of Visual Studio, it notices that for any .cpp file being compiled, the stats.txt file is touched as well. So it considers that a 'dependency' of some sort: if a.cpp is touched, stats.txt needs to get touched -- and maybe Visual Studio also has a backling such that it thinks that if stats.txt is touched, a.cpp needs to be updated, too.

Thus, when doing an incremental build which merely compiles single .cpp files, it notices that doing so touches stats.txt (to count a cache hit) and thus it thinks that all other .cpp files will need to be recompiled as well? Sounnds unlikely...

I skimmed the documentation about Incremental Builds, but didn't spot anything which might help with avoiding this issue. My feeling is that if we just had a way to either disable tracking writes to stats.txt, this issue would be gone.

[Un1oR]

I have simplified example. Now it contains one project with two cpp files and problem still is reproduced.

I have tried to find workaround for disabling statistics file tracking. ExcludedInputPaths and TrackedOutputFilesToIgnore options wasn't have effect for me. Put cache directory into one of this folders is works, but it isn't useful solution, I think.

[frerich]

TrackedOutputFilesToIgnore sounds promising! I didn't see it mentioned too often though. It is listed in the Microsoft.Build.CommonTypes.xsd schema, but I see no mentioning of the term TrackedOutputFilesToIgnore in the C# source code of MSBuild. :-/

[frerich]

I think we should really start looking into integration tests which use clcache the same way Visual Studio does, maybe using msbuild instead of calling cl.exe triggers this already?

[ivaigult]

I'm observing the same behavior and even worse: compiled binaries don't contain my changes, but full rebuild helps.

[frerich]

Alas I don't use msbuild myself -- is it possible to detect incremental builds by looking at the cl command line? If so, we could at least treat such invocations as unsupported and don't try to restore any cached objects.

[ivaigult]

The command line is exactly same as for regular build, but I cannot reproduce this without Visual Studio or msbuild.

[izmmisha]

Would be nice if this problem solved by #319 PR.

[Un1oR]

I'll try to check it out soon!

[RomanIudintsev]

Hi, I compiled exe from https://github.com/izmmisha/clcache/tree/add-tracker-support and try to compile project https://github.com/Un1oR/clcache-incremental-test with it. Got fail https://gist.github.com/RomanYudintsev/9978f7dadcf090bb010f257e42837776#file-gistfile1-txt