Rultor Build Fails on Recent Versions of yegor256/rultor

[original-brownbear]

This is the explaination for #81 failing.
You can find more information on this here: yegor256/rultor#1018

In essence this is what happened here specific to this project:
Currently the project sees this error in the build when merging for example here:

Caused by: org.sonatype.aether.resolution.ArtifactResolutionException: Could not find artifact com.jcabi:jcabi-x:jar:0.0.0 in Maven (http://repo.maven.apache.org/maven2)

This is caused by:
yegor256/rultor@61ce6e1#diff-137272550fb0d38c0e229b21d901e20eR6
... forcing the use of http://repo.maven.apache.org/maven2 on Rultor.

Rultor needed this to build on Maven 3.3.9, but here it evidently is causing trouble.
As explained in the above Rultor ticket, this needs to be fixed in projects breaking supplying their own Docker image, not in Rultor. Rultor's Dockerfile/Image is there to build Rultor itself and maintaining the build across multiple users of the same base Docker image is simply an impossible task.

That being said, looking at the last PR here, that the merge worked fine with: The below Dockerfile results in a working image. Building it and setting the image property in the .rultor.yml to wherever you push the build result, will resolve the issues you're experiencing with merging #81.

From https://hub.docker.com/r/yegor256/rultor/builds/bh4tyahxbh2bfcngvfxnyt4/:

# Copyright (c) 2009-2015, rultor.com
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met: 1) Redistributions of source code must retain the above
# copyright notice, this list of conditions and the following
# disclaimer. 2) Redistributions in binary form must reproduce the above
# copyright notice, this list of conditions and the following
# disclaimer in the documentation and/or other materials provided
# with the distribution. 3) Neither the name of the rultor.com nor
# the names of its contributors may be used to endorse or promote
# products derived from this software without specific prior written
# permission.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT
# NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND
# FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
# THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
# INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
# (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
# STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED
# OF THE POSSIBILITY OF SUCH DAMAGE.

FROM ubuntu:14.04
MAINTAINER Yegor Bugayenko <[email protected]>
LABEL Description="This is the default image for Rultor.com" Vendor="Rultor.com" Version="1.0"
WORKDIR /tmp

ENV DEBIAN_FRONTEND=noninteractive

# UTF-8 locale
RUN locale-gen en_US en_US.UTF-8
RUN dpkg-reconfigure locales
ENV LC_ALL en_US.UTF-8
ENV LANG en_US.UTF-8
ENV LANGUAGE en_US.UTF-8

# Basic Linux tools
RUN apt-get update
RUN apt-get install -y wget bcrypt curl
RUN apt-get install -y unzip zip
RUN apt-get install -y gnupg gnupg2
RUN apt-get install -y jq
RUN apt-get install -y bsdmainutils
RUN apt-get install -y libxml2-utils
RUN apt-get install -y build-essential
RUN apt-get install -y automake autoconf

# Git 2.0
RUN apt-get install -y software-properties-common python-software-properties
RUN add-apt-repository ppa:git-core/ppa
RUN apt-get update
RUN apt-get install -y git git-core

# SSH Daemon
RUN apt-get install -y ssh && \
  mkdir /var/run/sshd && \
  chmod 0755 /var/run/sshd

# Ruby
RUN apt-get install -y libmagic-dev zlib1g-dev
RUN apt-add-repository ppa:brightbox/ruby-ng
RUN apt-get update
RUN apt-get install -y ruby2.2 ruby2.2-dev
RUN gem update && gem install nokogiri && gem install bundler

# Ruby via RVM
#RUN apt-add-repository -y ppa:rael-gc/rvm
#RUN apt-get update
#RUN sudo apt-get install -y rvm
#RUN /bin/bash -l -c "rvm install ruby"
#RUN /bin/bash -l -c "gem update"
#RUN /bin/bash -l -c "gem install nokogiri"
#RUN /bin/bash -l -c "gem install bundler"
#RUN echo 'source /usr/share/rvm/scripts/rvm' >> /etc/bash.bashrc

# PHP
RUN apt-get install -y php5 php5-dev php-pear
RUN curl --silent --show-error https://getcomposer.org/installer | php
RUN mv composer.phar /usr/local/bin/composer
RUN mkdir jsl && \
  wget --quiet http://www.javascriptlint.com/download/jsl-0.3.0-src.tar.gz && \
  tar xzf jsl-0.3.0-src.tar.gz && \
  cd jsl-0.3.0/src && \
  make -f Makefile.ref && \
  mv Linux_All_DBG.OBJ/jsl /usr/local/bin && \
  cd .. && \
  rm -rf jsl
# RUN pecl install xdebug-beta && \
#   echo "zend_extension=xdebug.so" > /etc/php5/cli/conf.d/xdebug.ini

# Java
RUN apt-get install -y default-jdk
RUN apt-get install -y gdebi && \
  wget --quiet http://ppa.launchpad.net/natecarlson/maven3/ubuntu/pool/main/m/maven3/maven3_3.2.1-0~ppa1_all.deb && \
  gdebi --non-interactive maven3_3.2.1-0~ppa1_all.deb && \
  ln -s /usr/share/maven3/bin/mvn /usr/bin/mvn && \
  rm -rf maven3_3.2.1-0~ppa1_all.deb
RUN git clone https://github.com/yegor256/rultor.git && \
  cd rultor && \
  gem install jekyll && \
  mvn clean install -Pqulice --quiet && \
  cd .. && \
  rm -rf rultor

# LaTeX
RUN apt-get install -y texlive-latex-extra
RUN apt-get install -y xzdec
RUN tlmgr init-usertree
RUN tlmgr install helvetic
RUN tlmgr install psnfss

# PhantomJS
RUN apt-get install -y phantomjs

# S3cmd for AWS S3 integration
RUN apt-get install -y s3cmd

# NodeJS
RUN apt-get install -y nodejs npm
RUN sudo npm install -g npm
RUN ln -s /usr/bin/nodejs /usr/bin/node

# One more time, right before the end
RUN apt-get update -y
RUN apt-get upgrade -qq -y

# Clean up
RUN rm -rf /tmp/*
RUN rm -rf /root/.ssh

---

• 82-ba434d45/Aether.java:97-100: Clean unnecessary docker setup (src/docler/Dockerfile),... #84 (by Mihai Emil Andronache)

[dmarkov]

@yegor256 please take a look at it and dispatch (see par.21)

[yegor256]

@original-brownbear can't we just modify the code of this project so that it starts to work with the newest Docker image from Rultor?

[original-brownbear]

@yegor256 yes you can, but adding the custom image really is just:

• Build image from old Rultor image version, or simply create a tag for that image in fact
• add that one line to .rultor.yml specifying the image

... while fixing it in the code is adding a line to load some Maven settings.xml to the Rultor script adding the correct mirror. Well and adding that settings.xml to the repo..

To me it looks like the effort of both approaches is pretty much the same, but fixing the code instead of creating the image still keeps the risk of this happening again around.
Same as Travis and the like don't have a one size fits all environment, Rultor shouldn't have one either for exactly this reason.

[yegor256]

@original-brownbear I think, the best approach is to start tagging Docker images in Rultor project. now there is just one tag: latest. If we start tagging them, we will be able to specify a tagged image in this project, which will solve the problem. see?

[original-brownbear]

@yegor256 jup as I said above that would be the easiest :)

[yegor256]

@original-brownbear let's fix that in Rultor first? to make that tags available. also, let's make it possible to provide a tag in .rultor.yml config. and then, we'll get back to this ticket. right?

[original-brownbear]

@yegor256 ticket created: yegor256/rultor#1033, this is a job for you I'm afraid.

[yegor256]

@original-brownbear I'll take a look

[original-brownbear]

@yegor256 now that the Rultor ticket is dealt with you should be able to simply do this (tried it with my fork):

git push origin :refs/tags/1.61.3 
git push origin --tags

That should trigger the build of 1.61.3 which was working with this project as far as I can see and then just put:

docker:
  image: "yegor256/rultor:1.61.3"

in the .rultor.yml and be happy in all likelihood :)

[yegor256]

@original-brownbear done: https://hub.docker.com/r/yegor256/rultor/builds/bbvxn3arbq5bhcdk7ucyxxe/

[original-brownbear]

@yegor256 it errored :/, can you try again ? Maybe just some apt-get timeout.

[yegor256]

@original-brownbear trying again: https://hub.docker.com/r/yegor256/rultor/builds/bsyzeopmvzpkvfafaap5bxs/

[original-brownbear]

@yegor256 damn it failed again but the reason is clear at least:
The build uses the 1.61.3 Dockerfile but the Dockerfile contains running a Rultor git pull and then build. This cannot work because it pulls the latest Rultor version (for which this file misses the new deps.). For all future tags this will be fine, but rebuilding past tags will break for all tags before the upgrading of Rultor dependencies I fear.
Really I think the approach that wastes the least amount of time here is to:

• Just create a Dockerfile for this project from the 1.61.3 Rultor Dockerfile minus the part where it pulls ad builds Rultor itself.
• Add this file to this project and set up the same kind of automated build Rultor has in Dockerhub
• Set the resulting image in the .rultor.yml.

This is a thing of 10 minutes and fixes the situation for good. All other projects working fine currently can simply lock in their version of the image, but with this one we need to intervene manually.

[yegor256]

@original-brownbear maybe it's better to create a new tag 1.61.3.1, which will be cloned from 1.61.3 and will have no rultor build inside?

[original-brownbear]

@yegor256 honestly I only see downsides to this approach. It's the same amount of work and creates a much harder to maintain situation for this project. If you want to upgrade dependencies here, you'll have to either count on there being a working tag of Rultor for the upgraded dependencies or add a Dockerfile here afterall.
If you add the file here, all future dependency issues can be resolved here and won't require any action on the Rultor side.
To me this situation looks equivalent to creating a .travis.yml.
It's simply better to keep the definition of the necessary dependencies for a project with that project and not elsewhere isn't it?

[yegor256]

@original-brownbear yes, I agree. let's go for it.

[amihaiemil]

@original-brownbear what's the status here? :)

[original-brownbear]

@amihaiemil I'm not sure, as far as I understand the above conversation this ticket is a valid bug and needs to be handled.

[original-brownbear]

@yegor256 see above 2 comments, you don't need anything from me here right?

[yegor256]

@original-brownbear no, we're good, it's a bug

[dmarkov]

@original-brownbear thanks a lot for reporting, 15 mins added to your acc, pmt ID AP-226479170T671113C

[dmarkov]

@amihaiemil it's yours, please go ahead

[amihaiemil]

@original-brownbear I'm new to docker (trying to get a decent grip of it now, since I have this task)

I got the big picture of what needs to be done from your comments, but I have a few questions, if you're willing to answer:

First, what I know is that docker acts like a virtual machine and using the Dockerfile you simply setup the build environment and then specify some commands (in our case mvn builds on projects that we git clone form their repos).

What is a unclear to me is:

1. What is an image and how you specify it in .rultor.yml? You said that a Docker hub job for jcabi-aether should create an image that is to be specified in .rultor.yml.

2. How does rultor know to look under /src/docker/ for the Docker file (I found nothing in rultor's source code). Or it's not rultor that does that, but the DockerHub job, in which case I would guess there's a setting there?

3. If all the projects currently use rultor's Dockerfile, does that mean every build of a project runs all the RUN commands in that Dockerfile? (that would mean every build of every project also runs mvn clean install -Pqulice on rultor project?) - I most likely misunderstood this since it would be pointless to be like this.

4. To my understanding, ideally, each jcabi project should have a Dockerfile like rultor's, but pull and build the matching project, not rultor. (e..g Dockerfile of jcabi-aether would git clone and build the code at https://github.com/jcabi/jcabi-aether.git). It would be nice, of course, to reuse the common environment setup part using the FROM directive.

[original-brownbear]

@amihaiemil let me try :)

answers can be found here:

1. What is an image ? => Read this
2. About Rultor + images => Read this
3. No :) those commands are only run once, see the link in 1. ;)
4. No the git interaction needs to happen in Rultor, otherwise you simply remove the Rultor functionality from the merging part ;). See the link in 2. to understand this better.

[amihaiemil]

@dmarkov The PR #83 for this issue has been merged and closed. Can you close this ticket? Thanks

[original-brownbear]

@amihaiemil I'm the issue author, you need to ask me to close it ;)
=> closed :)

[amihaiemil]

@original-brownbear sure, thanks. I asked the PM out of habbit.

[dmarkov]

@amihaiemil Thanks for your contribution, 30 mins was added to your account, payment ID is AP-1WE51508E7353410P, 221 hours and 7 mins spent total. +30 to your rating, your total score is +240

[dmarkov]

@original-brownbear the code made here contains a puzzle 82-ba434d45/#84, which will be resolved soon

[dmarkov]

@original-brownbear the last puzzle 82-ba434d45/#84 originated from here solved

[amihaiemil]

@original-brownbear I think something like this:

would be more accurate for Dan's avatar...