Self-hosted runners suddenly failing

Hi,

We have a self hosted runner set up on a Mac Mini here in the office. It’s been working well, but all of a sudden every build has started failing because build errors when setting up Ruby dependencies. We’re using Bundler to manage Ruby gems, installed for an installation of Ruby managed by rbenv on the Mac Mini.

I’ve been unable to reproduce the error while logged into the machine via SSH, but it consistently fails in the Actions runner. The action uses the shell directive shell: bash -l. If I understand the Actions documentation correctly, invoking bash with bash --noprofile --norc -eo pipefail -l should create the same kind of session - but running bundler to install said dependencies is fine in this case.

We had been using the following Gemfile:

source 'https://rubygems.org'

git_source(:github) {|repo_name| "https://github.com/#{repo_name}" }

# Minimum version of ruby required
ruby ">= 2.6.3"

gem 'cocoapods', '>= 1.8.4'
gem 'fastlane', '2.140.0'

which started having json fail as a dependency of CocoaPods:

# .../.bundle/ruby/2.6.0/extensions/x86_64-darwin-18/2.6.0/json-2.3.0/gem_make.out
current directory: .../.bundle/ruby/2.6.0/gems/json-2.3.0/ext/json/ext/generator
/Users/myuser/.rbenv/versions/2.6.3/bin/ruby -I /Users/myuser/.rbenv/versions/2.6.3/lib/ruby/2.6.0 -r ./siteconf20200622-62217-12uprqk.rb extconf.rb
creating Makefile

current directory: .../.bundle/ruby/2.6.0/gems/json-2.3.0/ext/json/ext/generator
make "DESTDIR=" clean

current directory: .../.bundle/ruby/2.6.0/gems/json-2.3.0/ext/json/ext/generator
make "DESTDIR="
compiling generator.c
In file included from generator.c:1:
In file included from ./../fbuffer/fbuffer.h:5:
In file included from /Users/myuser/.rbenv/versions/2.6.3/include/ruby-2.6.0/ruby.h:33:
In file included from /Users/myuser/.rbenv/versions/2.6.3/include/ruby-2.6.0/ruby/ruby.h:29:
/Users/myuser/.rbenv/versions/2.6.3/include/ruby-2.6.0/ruby/defines.h:123:10: fatal error: 'stdio.h' file not found
#include <stdio.h>
         ^~~~~~~~~
1 error generated.
make: *** [generator.o] Error 1

make failed, exit code 2

and unf_ext fail as a dependency of fastlane:

# .../.bundle/ruby/2.6.0/extensions/x86_64-darwin-18/2.6.0/unf_ext-0.0.7.6/mkmf.log
"clang -o conftest -I/Users/myuser/.rbenv/versions/2.6.3/include/ruby-2.6.0/x86_64-darwin18 -I/Users/myuser/.rbenv/versions/2.6.3/include/ruby-2.6.0/ruby/backward -I/Users/myuser/.rbenv/versions/2.6.3/include/ruby-2.6.0 -I. -I/Users/myuser/.rbenv/versions/2.6.3/include -I/usr/local/opt/node@12/include -D_XOPEN_SOURCE -D_DARWIN_C_SOURCE -D_DARWIN_UNLIMITED_SELECT -D_REENTRANT    -O3 -Wno-error=shorten-64-to-32  -fno-common -pipe conftest.c  -L. -L/Users/myuser/.rbenv/versions/2.6.3/lib -L. -L/Users/myuser/.rbenv/versions/2.6.3/lib -L/usr/local/opt/node@12/lib -fstack-protector-strong -L/usr/local/lib     -lruby.2.6   "
In file included from conftest.c:1:
In file included from /Users/myuser/.rbenv/versions/2.6.3/include/ruby-2.6.0/ruby.h:33:
In file included from /Users/myuser/.rbenv/versions/2.6.3/include/ruby-2.6.0/ruby/ruby.h:29:
/Users/myuser/.rbenv/versions/2.6.3/include/ruby-2.6.0/ruby/defines.h:123:10: fatal error: 'stdio.h' file not found
#include <stdio.h>
         ^~~~~~~~~
1 error generated.
checked program was:
/* begin */
1: #include "ruby.h"
2: 
3: int main(int argc, char **argv)
4: {
5:   return 0;
6: }
/* end */

I tried upgrading these Ruby dependencies, with the new Gemfile being:

source 'https://rubygems.org'

git_source(:github) {|repo_name| "https://github.com/#{repo_name}" }

# Minimum version of ruby required
ruby "~> 2.7.1"

gem 'cocoapods', '~> 1.9.3'
gem 'fastlane', '~> 2.149.1'

But now ffi fails as a dependency of Cocoapods:

# .../.bundle/ruby/2.7.0/extensions/x86_64-darwin-19/2.7.0/ffi-1.13.1/mkmf.log
"pkg-config --exists libffi"
| pkg-config --libs libffi
=> "-lffi\n"
"clang -o conftest -I/Users/myuser/.rbenv/versions/2.7.1/include/ruby-2.7.0/x86_64-darwin19 -I/Users/myuser/.rbenv/versions/2.7.1/include/ruby-2.7.0/ruby/backward -I/Users/myuser/.rbenv/versions/2.7.1/include/ruby-2.7.0 -I. -I/Users/myuser/.rbenv/versions/2.7.1/include -I/usr/local/opt/node@12/include -D_XOPEN_SOURCE -D_DARWIN_C_SOURCE -D_DARWIN_UNLIMITED_SELECT -D_REENTRANT   -O3 -ggdb3 -Wall -Wextra -Wdeprecated-declarations -Wdivision-by-zero -Wimplicit-function-declaration -Wimplicit-int -Wpointer-arith -Wshorten-64-to-32 -Wwrite-strings -Wmissing-noreturn -Wno-constant-logical-operand -Wno-long-long -Wno-missing-field-initializers -Wno-overlength-strings -Wno-parentheses-equality -Wno-self-assign -Wno-tautological-compare -Wno-unused-parameter -Wno-unused-value -Wunused-variable -Wextra-tokens  -fno-common -pipe conftest.c  -L. -L/Users/myuser/.rbenv/versions/2.7.1/lib -L. -L/Users/myuser/.rbenv/versions/2.7.1/lib -L/usr/local/opt/node@12/lib -fstack-protector-strong -L/usr/local/lib     -lruby.2.7   "
In file included from conftest.c:1:
In file included from /Users/myuser/.rbenv/versions/2.7.1/include/ruby-2.7.0/ruby.h:33:
In file included from /Users/myuser/.rbenv/versions/2.7.1/include/ruby-2.7.0/ruby/ruby.h:29:
/Users/myuser/.rbenv/versions/2.7.1/include/ruby-2.7.0/ruby/defines.h:126:10: fatal error: 'stdio.h' file not found
#include <stdio.h>
         ^~~~~~~~~
1 error generated.
checked program was:
/* begin */
1: #include "ruby.h"
2: 
3: int main(int argc, char **argv)
4: {
5:   return !!argv[argc];
6: }
/* end */

As you can see, it always boils down to this stdio.h header.

Has anyone else run into a similar issue? I’m starting to think this has something to do with macOS Catalina, and it not allowing the bash instance run by the Action Runner to use GCC properly… but I’m unsure how to confirm this. The machine is running on macOS 10.15.5 (19F101).

The logs are taken from the Action runner’s _work directory. I’ve changed some of the paths to hide private information.

@genkobar,

Do you mean that in the past your self-hosted macOS runner could work fine to execute the same jobs for the same project, but it goes down now?
Did you try installing a new self-hosted on the macOS machine to see if it can work?

If the problem still exists, please share your repository with us, I will help you report this issue to the appropriate engineering team for further investigation and evaluation.