Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Runners try to register before Gitlab installed and/or ready #146

Open
bschonec opened this issue Apr 13, 2022 · 8 comments
Open

Runners try to register before Gitlab installed and/or ready #146

bschonec opened this issue Apr 13, 2022 · 8 comments

Comments

@bschonec
Copy link

Affected Puppet, Ruby, OS and module versions/distributions

  • Puppet:

puppet-agent-7.15.0-1.el8.x86_64
puppetserver-7.6.1-1.el7.noarch

  • Ruby:

  • Distribution:
    Puppet server: Red Hat Enterprise Linux Server release 7.9 (Maipo)
    Puppet agent: Red Hat Enterprise Linux release 8.1 (Ootpa)

  • Module version:
    4.3.0

How to reproduce (e.g Puppet code you use)

Puppet manifest:

  # Resource ordering  THIS DOESN'T ORDER PROPERLY
  Class['gitlab'] -> Class['gitlab_ci_runner'] 
  include ::gitlab

  # Install Gitlab CI Runners
  include ::gitlab_ci_runner

Hiera data:

gitlab_ci_runner::concurrent: 4
gitlab_ci_runner::check_interval: 4
gitlab_ci_runner::metrics_server: "localhost:8888"
gitlab_ci_runner::manage_docker: true
gitlab_ci_runner::config_path: "/etc/gitlab-runner/config.toml"
gitlab_ci_runner::runners:
  ubi8:
    ensure: present
    url: "http://10.10.10.10/"
    registration-token: "token"
    tag-list: "aws,docker,example-tag"
    executor: "docker"
    docker:
      image: "ubi/ubi8:minimal"
  alpine:
    ensure: present
    url: "http://10.10.10.10/"
    registration-token: "token"
    tag-list: "alpine"
    executor: "docker"
    docker:
      image: "alpine:latest"
 
gitlab_ci_runner::runner_defaults:
  url: "https://localhost/ci"
  registration-token: "token"
  executor: "docker"
  docker:
    image: "ubi/ubi7:minimal"

What are you seeing

When running the puppet manifest, the gitlab_ci_runner https://github.com/voxpupuli/puppet-gitlab_ci_runner/blob/master/manifests/init.pp#L139 tries to connect to the local machine before gitlab is installed from the include ::gitlab code.

What behaviour did you expect instead

I expected Gitlab class module to run and install Gitlab CE and then the gitlab_ci_runner class module register the runners.

Output log

Info: Using environment '23_create_gitlab_ce_role_profile_for_installing_and_managing_gitlabce'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Error: Failed to apply catalog: Failed to open TCP connection to 10.100.100.10:80 (Connection refused - connect(2) for "10.100.100.119" port 80)

Any additional information you'd like to impart

Of course, since Gitlab isn't installed there's no way to configure the runners.

I've been out of the Puppet world for about four years so I'm rusty. I'm not familiar with Deferred Functions.

Is the resource ordering above not correct? Is there a way in the puppet module to ensure that the runners are registered later/last?

@tuxmea
Copy link
Member

tuxmea commented Apr 14, 2022

Have you tried using contain instead of include?

  # Resource ordering  THIS DOESN'T ORDER PROPERLY
  Class['gitlab'] -> Class['gitlab_ci_runner'] 
  contain ::gitlab

  # Install Gitlab CI Runners
  contain ::gitlab_ci_runner

@bschonec
Copy link
Author

I've tried your suggestion but I'm getting the same results. I did notice that the puppet run is failing at https://github.com/voxpupuli/puppet-gitlab_ci_runner/blob/master/lib/puppet/functions/gitlab_ci_runner/register_to_file.rb#L44 before the class module even has a chance to do anything or that resource ordering is enforced.

@alexjfisher
Copy link
Member

@tuxmea The registering takes place in a Deferred function that gets run before any of the rest of the catalog is actually applied.

If you're installing a runner on the same host as gitlab is being installed, then you might want to wrap the gitlab runner part of your puppet code in a fact that checks if gitlab itself has been installed yet. Doesn't look like https://github.com/voxpupuli/puppet-gitlab contains any facts, but creating a simple gitlab_installed fact should be pretty trivial.

Then your puppet code might look a bit like...

include gitlab

if $facts['gitlab_installed'] {
  include gitlab_ci_runner
} else {
  notify { 'Rerun puppet to install gitlab runner':}
}

Your custom fact (which you could put in your profile module) may be as simple as something like ...

# profile/lib/facter/gitlab_installed.rb
Facter.add(:gitlab_installed) do
  setcode do
    File.exist?('/path/to/some/file/that/exists/after/gitlab/is/installed')
  end
end

Hope that helps.

@alexjfisher
Copy link
Member

@alexjfisher
Copy link
Member

Just spotted this 'draft' PR. puppetlabs/puppet#8902 :)

@joshcooper
Copy link

joshcooper commented Apr 22, 2022

👋 Hope to have the deferred PR more ready to go next week. It'll fix two important limitations with Deferred functions.

  • Currently if a deferred function fails, then the entire run fails. With the PR, only the resource will fail (and other resources that depend on it, direectly or transitively).
  • Currently it's not possible to install a prerequisite (like a gem) for a deferred function and call the deferred function in the same run. However, due to the above limitation, you can't install the gem and run the agent again. In other words, it never converges.

@jkroepke
Copy link

Hi, this problem still exists in puppet 7.19...

# /opt/puppetlabs/bin/puppet agent -t
Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Error: Failed to apply catalog: Failed to open TCP connection to gitlab:80 (Connection refused - connect(2) for "gitlab" port 80)

here is a trace:

Error: Failed to apply catalog: Failed to open TCP connection to gitlab:80 (Connection refused - connect(2) for "gitlab" port 80)
/opt/puppetlabs/puppet/lib/ruby/2.7.0/net/http.rb:960:in `initialize'
/opt/puppetlabs/puppet/lib/ruby/2.7.0/net/http.rb:960:in `open'
/opt/puppetlabs/puppet/lib/ruby/2.7.0/net/http.rb:960:in `block in connect'
/opt/puppetlabs/puppet/lib/ruby/2.7.0/timeout.rb:95:in `block in timeout'
/opt/puppetlabs/puppet/lib/ruby/2.7.0/timeout.rb:105:in `timeout'
/opt/puppetlabs/puppet/lib/ruby/2.7.0/net/http.rb:958:in `connect'
/opt/puppetlabs/puppet/lib/ruby/2.7.0/net/http.rb:943:in `do_start'
/opt/puppetlabs/puppet/lib/ruby/2.7.0/net/http.rb:932:in `start'
/opt/puppetlabs/puppet/lib/ruby/2.7.0/net/http.rb:1483:in `request'
/opt/puppetlabs/puppet/cache/lib/puppet_x/gitlab/runner.rb:47:in `request'
/opt/puppetlabs/puppet/cache/lib/puppet_x/gitlab/runner.rb:18:in `post'
/opt/puppetlabs/puppet/cache/lib/puppet_x/gitlab/runner.rb:58:in `register'
/opt/puppetlabs/puppet/cache/lib/puppet/functions/gitlab_ci_runner/register_to_file.rb:44:in `register_to_file'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/functions/dispatch.rb:60:in `invoke'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/functions/dispatcher.rb:47:in `block in dispatch'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/functions/dispatcher.rb:46:in `catch'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/functions/dispatcher.rb:46:in `dispatch'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/functions/function.rb:46:in `block in call'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/functions/function.rb:45:in `catch'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/functions/function.rb:45:in `call'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/runtime3_support.rb:283:in `block in external_call_function'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/runtime3_support.rb:282:in `external_call_function'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/scope.rb:1102:in `call_function'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/deferred_resolver.rb:151:in `resolve_future'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/deferred_resolver.rb:102:in `resolve'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/deferred_resolver.rb:107:in `block in resolve'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/deferred_resolver.rb:107:in `each_pair'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/deferred_resolver.rb:107:in `resolve'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/deferred_resolver.rb:104:in `block in resolve'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/deferred_resolver.rb:104:in `map'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/deferred_resolver.rb:104:in `resolve'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/deferred_resolver.rb:107:in `block in resolve'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/deferred_resolver.rb:107:in `each_pair'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/deferred_resolver.rb:107:in `resolve'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/deferred_resolver.rb:166:in `block in map_arguments'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/deferred_resolver.rb:166:in `map'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/deferred_resolver.rb:166:in `map_arguments'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/deferred_resolver.rb:141:in `resolve_future'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/deferred_resolver.rb:102:in `resolve'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/deferred_resolver.rb:84:in `block (2 levels) in resolve_futures'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/deferred_resolver.rb:83:in `each_pair'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/deferred_resolver.rb:83:in `block in resolve_futures'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/deferred_resolver.rb:81:in `each'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/deferred_resolver.rb:81:in `resolve_futures'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/deferred_resolver.rb:44:in `resolve_and_replace'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:115:in `block in convert_catalog'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:567:in `block in thinmark'
/opt/puppetlabs/puppet/lib/ruby/2.7.0/benchmark.rb:308:in `realtime'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:566:in `thinmark'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:112:in `convert_catalog'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:481:in `run_internal'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:339:in `run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/agent.rb:83:in `block (6 levels) in run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:289:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/agent.rb:82:in `block (5 levels) in run'
/opt/puppetlabs/puppet/lib/ruby/2.7.0/timeout.rb:95:in `block in timeout'
/opt/puppetlabs/puppet/lib/ruby/2.7.0/timeout.rb:105:in `timeout'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/agent.rb:81:in `block (4 levels) in run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/agent/locker.rb:21:in `lock'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/agent.rb:71:in `block (3 levels) in run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/agent.rb:162:in `with_client'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/agent.rb:67:in `block (2 levels) in run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/agent.rb:127:in `run_in_fork'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/agent.rb:66:in `block in run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:172:in `controlled_run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/agent.rb:47:in `run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/agent.rb:433:in `onetime'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/agent.rb:393:in `block in run_command'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:289:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/agent.rb:390:in `run_command'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:421:in `block in run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:756:in `exit_on_fail'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:421:in `run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:143:in `run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:77:in `execute'
/opt/puppetlabs/puppet/bin/puppet:5:in `<main>'
Debug: Closing connection for https://puppet:8140

@mojibake-umd
Copy link

mojibake-umd commented Dec 1, 2023

I would like to comment that I just tested setting
[agent]
preprocess_deferred = false in puppet.conf
or via hiera.
puppet::agent_additional_settings:
# Disable deferred functions
# gitlab_ci_runner will try to register before
# anything is even installed and configured.
preprocess_deferred: false

Which seems to get around the issue for me of launching an EC2 runner from zero.

I tested against 4.4.0.
puppet 7.24.0

EDIT:

Actually that only gets you a little further along in puppet setup.
End up running into puppetlabs/puppet#9174 and still failing to get gitlab runner registered.

Error: /Stage[main]/Gitlab_ci_runner::Config/Concat[/etc/gitlab-runner/config.toml]/Concat_file[/etc/gitlab-runner/config.toml]: Failed to generate additional resources using 'eval_generate': undefined method `<<' for #<Puppet::Pops::Evaluator::DeferredValue:0x00000000087d1df8>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants