Help
cancel
Showing results for 
Search instead for 
Did you mean: 
Highlighted
Pilot Lvl 1
Message 11 of 17

Re: Windows builds failing randomly

Thanks for looking into it Alena.

 

...but the build you're showing didn't fail. What you're showing here (and in the log) looks like the standard way CMake tests for compiler attributes. In that case it's supposed to fail.

 

(The code it points at is cmake's code for testing compilers for specific atttributes - not our code.)

 

The build works on a real Windows machine (the rest of the team are using them).

 

Right now with master it usually fails several times before it builds properly. Here's a more recent example:

 

https://github.com/CloudCompare/CloudCompare/runs/618602537

 

If I keep re-running that one it will pass eventually.

 

So maybe try re-running it until it fails?

 

Did you see my comment above about the worker logs and pipes? That seems more likely to be an issue.

Highlighted
Copilot Lvl 2
Message 12 of 17

Re: Windows builds failing randomly

Your comment makes sense. We were actually able to reproduce the issue from your forked repo, and the only errors we were able to get are the ones in the build I provided in the answer above (that build I attached did not fail because of applied logic in try...catch):

CheckIncludeFile.c(1): fatal error C1083: Cannot open include file: 'pthread.h': No such file or directory
ninja: build stopped: subcommand failed.

 

Could you please add the suggested patch to your code, and provide the logs you will get? it will help us to understand what is happening in your build. 

Highlighted
Pilot Lvl 1
Message 13 of 17

Re: Windows builds failing randomly

> did not fail because of applied logic in try...catch

 

I'm sorry, but that's not the case. As I mentioned above, the things you are showing as errors here will not cause the build to fail. They are normal - it's the way that cmake determines what's available on the machine.

 

So I don't think you have reproduced the error with the try...catch yet. It fails in the cmake config process, so it never starts the build step. What about the other error I mentioned with the runner? Again - it seems a likely candidate.

 

I will see what else I can do at my end, but please try to reproduce it again. The symptom is (I think) that cmake is trying to produce a warning but instead something (runner?) produces an error which kills the build. It doesn't happen all the time, but it happens very frequently. I've had to run them 16 or 17 times in some cases to get it to pass.

Highlighted
Copilot Lvl 2
Message 14 of 17

Re: Windows builds failing randomly

There's a slightly modified build.yml which fails in case if there is any sdterr otput from the build. It seems to be too strict because but you might find it usefull

https://github.com/dsame/CloudCompare/blob/9888fc2a5e26d7c8bb0059f79f6d2bd1db881921/.github/workflow...

 

Please try to apply it your build in order to see the stderr output that was lost in your build and find out the differences in success and failed builds. 

Alternatively, as the recent Windows images have bash installed i'd advise to switch to it from PowerShell:
https://github.com/CloudCompare/CloudCompare/compare/CloudCompare:master...dsame:dsame/use-cmd?expan...

With the bash you should not have the problem with CMake output to stderr

Highlighted
Pilot Lvl 1
Message 15 of 17

Re: Windows builds failing randomly

I like the idea of switching to bash since I'm completely unfamiliar with PowerShell anyways. It seems like the easiest option. I'll do that and report back.

 

Thanks for your help Sergey!

Highlighted
Solution
Pilot Lvl 1
Message 16 of 17

Re: Windows builds failing randomly

After switching from PowerShell to bash as suggested by @dsame, we have not had any of the random failures on Windows we were seeing before.

 

Solution: Don't use PowerShell.

 

Thanks @dsame, @AlenaSviridenko and @Yanjingzhu for looking into it.

Highlighted
Copilot Lvl 2
Message 17 of 17

Re: Windows builds failing randomly

The reason builds fails is either CMake or another tool invoked by CMake writes to stderr. PowerShell triggers terminating exception then.

 

The reason build fails randomly is a race condition we did not identified exactly becaues the build system is too complex and not obvious. 

 

In order to identify the exact the problem of CMake fail i suggest you to apply the code of this PR: https://github.com/CloudCompare/CloudCompare/pull/1232 which intended to intercept the stderr stream and to do not let it miss the logs and PowerShell output. 

 

With this changes the CMake Configure will not break on the warnings but print them to the build log, so you can analyze them and find out the unexpected behaviour. 

The sample build: https://github.com/dsame/CloudCompare/runs/694200554?check_suite_focus=true

 

The messages cause PowerShell to fail becaues of CMake writes to stder are marked with "ERR" prefix:

2020-05-21_574.png

Please, review the suggested PR and confirm it can help effectively to solve the problem