Command cron_01_set_leader failed

[sumit20rai]

After installing/configuring this gem, I'm seeing the following error in /var/log/cfn-init.log on my EC2 instance after running git aws.push from my local repo.

2014-10-21 08:08:37,602 [DEBUG] Running test for command cron_01_set_leader
2014-10-21 08:08:37,744 [DEBUG] Test command output:
2014-10-21 08:08:37,745 [DEBUG] Test for command cron_01_set_leader passed
2014-10-21 08:08:38,085 [ERROR] Command cron_01_set_leader (su -c "/usr/local/bin/bundle exec create_cron_leader --no-update" $EB_CONFIG_APP_USER) failed
2014-10-21 08:08:38,086 [DEBUG] Command cron_01_set_leader output: bash: /usr/local/bin/bundle: No such file or directory

2014-10-21 08:08:38,086 [ERROR] Error encountered during build of postbuild_0_tubett: Command cron_01_set_leader failed
Traceback (most recent call last):

I have added the whenever-elasticbeanstalk
Below is my cron.config file content..
Any idea ...what am i doing wrong?

files:

Reload the on deployment

/opt/elasticbeanstalk/hooks/appdeploy/post/10_reload_cron.sh:
mode: "00700"
owner: root
group: root
content: |
#!/usr/bin/env bash
. /opt/elasticbeanstalk/containerfiles/envvars
cd $EB_CONFIG_APP_CURRENT
su -c "/usr/local/bin/bundle exec setup_cron" $EB_CONFIG_APP_USER

Add Bundle to the PATH

"/etc/profile.d/bundle.sh":
mode: "000755"
owner: root
group: root
content: |
#!/usr/bin/env bash
export PATH=$PATH:/usr/local/bin
encoding: plain
container_commands:
cron_01_set_leader:
test: test ! -f /opt/elasticbeanstalk/containerfiles/.cron-setup-complete
leader_only: true
cwd: /var/app/ondeck
command: su -c "/usr/local/bin/bundle exec create_cron_leader --no-update" $EB_CONFIG_APP_USER
cron_02_write_cron_setup_complete_file:
cwd: /opt/elasticbeanstalk/containerfiles
command: touch .cron-setup-complete

Thanks
Sumit

[jufemaiz]

Cross post @ http://stackoverflow.com/questions/26483509/command-cron-01-set-leader-output-bash-usr-local-bin-bundle-no-such-file-or

Can you please provide further information on the instance you are attempting to use this with? What is the full path to bundle on your instance? If this has changed, we have an issue obviously.

[jufemaiz]

@sumit20rai ping

[tboyko]

Seeing the same issue with bundle not existing where the script expects it. I'm running the "64bit Amazon Linux 2014.03 v1.0.9 running Ruby 2.1 (Puma)" instance and bundle exists at "/opt/rubies/ruby-2.1.2/bin/bundle". Why not incorporate something along the lines of BUNDLE_EXE=which bundle to the script and running off of the result?

Digging further, this creates other issues. I'm now seeing this:

2014-11-03 21:00:05,949 [INFO](13482 MainThread) [directoryHooksExecutor.py-29] [root directoryHooksExecutor info] Executing script: /opt/elasticbeanstalk/hooks/appdeploy/post/10_reload_cron.sh
2014-11-03 21:00:06,443 [INFO](13482 MainThread) [directoryHooksExecutor.py-29] [root directoryHooksExecutor info] Output from script: bundler: command not found: setup_cron
Install missing gem executables with bundle install

2014-11-03 21:00:06,443 [ERROR](13482 MainThread) [directoryHooksExecutor.py-33] [root directoryHooksExecutor error] Script /opt/elasticbeanstalk/hooks/appdeploy/post/10_reload_cron.sh failed with returncode 127

[jufemaiz]

@tboyko that looks like a better approach. One of the problems I've found in investigating this on the older instance is the inability to find the gem even if the bundle is correct. Will post some extra info shortly.

[tboyko]

Looks like the path issue is throughout the gem. This is tough for me to test (with a forked repo) because my experience with EB is that it just doesn't play nicely with git repos or local paths in the Gemfile...

[jufemaiz]

@tboyko ah yeah… that's a "fun" part of EB eh. I've found ways and means to cache into /vendor (simlinked into /var/app/containerfiles for persistency across deployments )

[jufemaiz]

@sumit20rai and @tboyko can you please try release v1.1.5 on your codebase. Note: you'll need to remove your .ebextensions/cron.config before running wheneverize-eb again.

[tboyko]

There seems to be an issue. Here's what I got after upgrading, deleting and recreating cron.config, and deploying:

2014-11-07 17:50:08,242 [DEBUG] Running command a_elasticbeanstalk_support_path
2014-11-07 17:50:08,242 [DEBUG] No test for command a_elasticbeanstalk_support_path
2014-11-07 17:50:08,248 [ERROR] Command a_elasticbeanstalk_support_path (. /tmp/update_appenv_include_eb_config_support.sh) failed
2014-11-07 17:50:08,248 [DEBUG] Command a_elasticbeanstalk_support_path output: containerfiles
export EB_CONFIG_SUPPORT=/opt/elasticbeanstalk/support
/tmp/update_appenv_include_eb_config_support.sh: line 8: /opt/elasticbeanstalk/support/envvars.d/appenv: No such file or directory

2014-11-07 17:50:08,248 [ERROR] Error encountered during build of prebuild_2_[REMOVED]: Command a_elasticbeanstalk_support_path failed
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/cfnbootstrap/construction.py", line 511, in run_config
CloudFormationCarpenter(config, self._auth_config).build(worklog)
File "/usr/lib/python2.6/site-packages/cfnbootstrap/construction.py", line 247, in build
changes['commands'] = CommandTool().apply(self._config.commands)
File "/usr/lib/python2.6/site-packages/cfnbootstrap/command_tool.py", line 113, in apply
raise ToolError(u"Command %s failed" % name)
ToolError: Command a_elasticbeanstalk_support_path failed
2014-11-07 17:50:08,249 [ERROR] Unhandled exception during build: Command a_elasticbeanstalk_support_path failed
Traceback (most recent call last):
File "/opt/aws/bin/cfn-init", line 122, in
worklog.build(detail.metadata, configSets)
File "/usr/lib/python2.6/site-packages/cfnbootstrap/construction.py", line 117, in build
Contractor(metadata).build(configSets, self)
File "/usr/lib/python2.6/site-packages/cfnbootstrap/construction.py", line 502, in build
self.run_config(config, worklog)
File "/usr/lib/python2.6/site-packages/cfnbootstrap/construction.py", line 511, in run_config
CloudFormationCarpenter(config, self._auth_config).build(worklog)
File "/usr/lib/python2.6/site-packages/cfnbootstrap/construction.py", line 247, in build
changes['commands'] = CommandTool().apply(self._config.commands)
File "/usr/lib/python2.6/site-packages/cfnbootstrap/command_tool.py", line 113, in apply
raise ToolError(u"Command %s failed" % name)
ToolError: Command a_elasticbeanstalk_support_path failed

[jufemaiz]

Crap. Sorry. Hadn't committed one of the files :(

Change .ebextensions/cron.config to:

    mode: "000755"
    owner: root
    group: root
    content: |
      #!/usr/bin/env bash
      # update_appenv_include_eb_config_support.sh
      if test -d /opt/elasticbeanstalk/containerfiles; then
        echo "containerfiles"
        grep -Fwq "export EB_CONFIG_SUPPORT" /opt/elasticbeanstalk/containerfiles/envvars.d/appenv
        if [ $? -eq 1 ]; then
          echo "export EB_CONFIG_SUPPORT=/opt/elasticbeanstalk/containerfiles"
          echo "export EB_CONFIG_SUPPORT=/opt/elasticbeanstalk/containerfiles" >> /opt/elasticbeanstalk/support/envvars.d/appenv;
        fi
      elif test -d /opt/elasticbeanstalk/support; then
        echo "support"
        grep -Fwq "export EB_CONFIG_SUPPORT" /opt/elasticbeanstalk/support/envvars.d/appenv
        if [ $? -eq 1 ]; then
          echo "export EB_CONFIG_SUPPORT=/opt/elasticbeanstalk/support"
          echo "export EB_CONFIG_SUPPORT=/opt/elasticbeanstalk/support" >> /opt/elasticbeanstalk/support/envvars.d/appenv;
        fi
      fi
    encoding: plain

[tboyko]

2014-11-10 17:33:12,126 [DEBUG] Command a_elasticbeanstalk_support_path output: containerfiles
export EB_CONFIG_SUPPORT=/opt/elasticbeanstalk/containerfiles
/tmp/update_appenv_include_eb_config_support.sh: line 8: /opt/elasticbeanstalk/support/envvars.d/appenv: No such file or directory

2014-11-10 17:33:12,126 [ERROR] Error encountered during build of prebuild_2_[removed]: Command a_elasticbeanstalk_support_path failed
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/cfnbootstrap/construction.py", line 511, in run_config
    CloudFormationCarpenter(config, self._auth_config).build(worklog)
  File "/usr/lib/python2.6/site-packages/cfnbootstrap/construction.py", line 247, in build
    changes['commands'] = CommandTool().apply(self._config.commands)
  File "/usr/lib/python2.6/site-packages/cfnbootstrap/command_tool.py", line 113, in apply
    raise ToolError(u"Command %s failed" % name)
ToolError: Command a_elasticbeanstalk_support_path failed
2014-11-10 17:33:12,127 [ERROR] Unhandled exception during build: Command a_elasticbeanstalk_support_path failed
Traceback (most recent call last):
  File "/opt/aws/bin/cfn-init", line 122, in <module>
    worklog.build(detail.metadata, configSets)
  File "/usr/lib/python2.6/site-packages/cfnbootstrap/construction.py", line 117, in build
    Contractor(metadata).build(configSets, self)
  File "/usr/lib/python2.6/site-packages/cfnbootstrap/construction.py", line 502, in build
    self.run_config(config, worklog)
  File "/usr/lib/python2.6/site-packages/cfnbootstrap/construction.py", line 511, in run_config
    CloudFormationCarpenter(config, self._auth_config).build(worklog)
  File "/usr/lib/python2.6/site-packages/cfnbootstrap/construction.py", line 247, in build
    changes['commands'] = CommandTool().apply(self._config.commands)
  File "/usr/lib/python2.6/site-packages/cfnbootstrap/command_tool.py", line 113, in apply
    raise ToolError(u"Command %s failed" % name)
ToolError: Command a_elasticbeanstalk_support_path failed

[tboyko]

I think there may be bigger issues at hand here. TL;DR - Amazon Linux AMIs are changing their structure and breaking the whole $EB_ variable notion, which this gem depends on. Seems like a major gem restructuring may be involved :/

https://forums.aws.amazon.com/thread.jspa?messageID=583440&#583440

[jufemaiz]

Oh crap. WTF is Amazon doing? Surely this is stuff that should be well documented and notified to users :(

[tboyko]

No kidding. Looks like the OpsWorks platform isn’t immune to this sort of thing either:

http://blogs.aws.amazon.com/application-management/post/Tx2G95J53SKI2E0/AWS-OpsWorks-supports-application-environment-variables http://blogs.aws.amazon.com/application-management/post/Tx2G95J53SKI2E0/AWS-OpsWorks-supports-application-environment-variables

Read the first comment…

On Nov 13, 2014, at 2:01 PM, Joel Courtney [email protected] wrote:

Oh crap. WTF is Amazon doing? Surely this is stuff that should be well documented and notified to users :(

—
Reply to this email directly or view it on GitHub #18 (comment).

[jufemaiz]

Ugh :(

WHY?!?!? BUT WHY?!?!?

[jufemaiz]

Ok @tboyko I've done some digging.

As root, you have access to the following shell script (which in turn calls a ruby script using a gem not available as ec2-user).

/opt/elasticbeanstalk/bin/get-config

With that you can identify which data you want to access with the following options:

Usage: get-config CATEGORY [OPTIONS]

Categories:
    optionsettings                   environment option settings that affect instance
    container                        container specific configurations
    addons                           addon configurations
    environment                      environment variables
    meta                             EB environment meta-data

Options:
    -k, --key KEY                    Key of specific configurations.
        --output OUTPUT_FORMAT       Output format. Can be JSON or YAML. Default is JSON.
    -n, --namespace NAMESPACE        Otion Setting namespace for retrieval. Only applied to Category optionsettings.
    -o, --option-name OPTION_NAME    Option Setting name for retrieval. Only applied to Category optionsettings.
    -a, --add-on ADDON               Add-on name. Only applied to Category addons.
    -h, --help                       Help

This itself is obtained from files on the instance, I believe anyway, at /opt/elasticbeanstalk/deploy/configuration/containerconfiguration

So now the question is, do older instances have this available to them. If so not so bad. If not… :(

[benmkramer]

Has there been any progress or updates to this on Elastic Beanstalk? Are there any alternatives that would be recommended?

[rmahnovetsky]

anyone made any progress on this one?

[sanjayginde]

@rmahnovetsky I hacked this together to get things functional for my own project. I'm hoping to clean it up at some point: https://github.com/contently/whenever-elasticbeanstalk/pull/1/files

[rmahnovetsky]

@sanjayginde thanks. I'll take a look and let you know how I get on

[mcbsys]

Not specifically about this gem, but here are a couple resources I found recently on the environment variable issue:

• AWS employee shows workaround on 11/14/2014 (but watch out for the typos that I mentioned 2/18/2015):
  https://forums.aws.amazon.com/thread.jspa?messageID=583183
• Blog article about solving this issue for delayed_job:
  http://www.dannemanne.com/posts/post-deployment_script_on_elastic_beanstalk_restart_delayed_job

In summary you probably want to do something like this starting from the 2014.09 AMIs:

# Loading environment data
EB_SCRIPT_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k script_dir)
EB_SUPPORT_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k support_dir)
EB_APP_USER=$(/opt/elasticbeanstalk/bin/get-config container -k app_user)
EB_APP_CURRENT_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k app_deploy_dir)
EB_APP_PIDS_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k app_pid_dir)

Here are some commands to check the AMI version:

cat /etc/system-release
cat /etc/image-id
cat /etc/yum.conf | grep releasever # Beanstalk locks repositories using this line

[langtianlang]

Is there any update on this? I'm looking to use whenever on ebs. Should I be using this project? Any guidance would be appreciated, thanks.

[rmahnovetsky]

I ended up creating a worker environment and using the periodic scheduler. http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features-managing-env-tiers.html

[JaisonBrooks]

I would love to know if there is a solution in the works or something out there to show how to use this gem and eb.

[mkafkas]

is there any update on this issue?

[binodmainali]

I am also facing something similar and stuck with Unauthorized access error

[binodmainali]

Above is your schedule file converted to cron syntax; your crontab file was not updated. I am still getting this as the output

[CyborgMaster]

I got this completely working with the latest AMI (2015.03). I based my work of the fork from @sanjayginde, but had to make a few more tweaks. I've got a pull request back to his fork open here: trilogy-group#2. I recommend that he accept my pull request and then open another one from his fork back here to the main repo, because he did 95% of the work.

[samhamilton]

Has anyone got this working with the latest AMI 2015.09? Seems to be broken again.

[jufemaiz]

#26

For those with issues who still would like this to work, please provide me with some information on whether or not this works for you.

[dharmdip]

Problem is EB is not checking stack ruby version and complains bundle with this error

Activity execution failed, because: bundler: command not found Install missing gem executables withbundle install(ElasticBeanstalk::ExternalInvocationError) caused by: bundler: command not found Install missing gem executables withbundle install(Executor::NonZeroExitStatus)

need to tell ruby version where gems are installed by adding this line or manage by environment variables.

. /opt/elasticbeanstalk/support/scripts/use-app-ruby.sh

Hope this solves issue.

[bamonde]

I had the same issue and found this article:
http://www.dannemanne.com/posts/post-deployment_script_on_elastic_beanstalk_restart_delayed_job

Maybe that helps to solve somebody problem and/or gem`s context.

[feliperaul]

I was reading about cron jobs with ElasticBeanstalk and the oficial AWS docs now state this:

Note: If you have a periodic task to be run on only one instance, consider using the cron-leaderonly-linux.config file for web environments. If you're using a dedicated worker environment, consider using periodic tasks.

The file they link to is this one, which has a commit date of May 1st 2017, which is after the last comment on this issue is from April 4 2017.

I have no idea if that can help solve the main issue this gem faced after AWS did the breaking changes (which, as far as I could understand, was related to leader ellection), but now AWS links to an official script to detect the leader.

Maybe there's a chance of fixing this gem?

If not, how is everybody handling this problem (making sure that cron jobs run only on one instance at a time in Elastic Beanstalk) ?

[jufemaiz]

@feliperaul sounds like a good option for you to raise as a separate issue?

For the record, we shifted to using containers and thus able to run them via ECS for background tasks.

[feliperaul]

@jufemaiz Any writeup on that path? We really would like to use EBS to deploy (specially because autoscalling due to ephemeral instances), but Sidekiq and CRON are two of our main blocks this path right now.