Core TravisCI not submitting results to hosting automated tests.

[getsource]

The WordPress Hosting Team has a distributed automated test project that collects PHPUnit test results from hosts.

As part of this, WordPress.org's TravisCI reports to that page. Currently, this is used to email hosts if Core's tests are passing, but theirs are not.

However, currently, core's tests aren't being submitted:
https://travis-ci.com/WordPress/wordpress-develop/jobs/293698369#L3345

From the logs, it looks like a couple of packages might be missing on the docker environment where the tests are being reported with:

$ if [[ "$WP_TEST_REPORTER" == "true" ]]; then
    git clone https://github.com/WordPress/phpunit-test-runner.git test-runner
    docker-compose run --rm -e WPT_PREPARE_DIR=/var/www -e WPT_TEST_DIR=/var/www php php test-runner/report.php
  fi
  
Cloning into 'test-runner'...
Starting wordpress-develop_mysql_1 ... 
ent variables pass checks.
Getting SVN Revision
sh: 1: git: not found
Getting SVN message
sh: 1: git: not found
Copying junit.xml results
rsync -rv '/var/www'/tests/phpunit/build/logs/* '/var/www'
sh: 1: rsync: not found
Error: Failed to perform operation.

@aaronjorbin recommended creating an issue here, so that the packages can be added, to the appropriate image, which is why I'm here!

If that doesn't work out, @ocean90 also recommended seeing if we could change how some of this works on the runner side, or making a new image just for submissions.

Conversation on both of those started here, on Slack.

[pento]

Does test-runner/report.php need to be run inside the container, or can it be run directly on the Travis instance?

(I know I originally wrote and committed that change, but I don't recall if there was a specific reason I configured test-runner/report.php to run inside the container, or if it was just to make the same as all the others. Not recalling anything important makes me suspect it's the latter.)

If it needs to be run in the container, we can explore adding the tools it needs. Adding rsync probably wouldn't be a problem, since it's tiny. git weighs in at ~30MB, which gives me a little pause.

[getsource]

Off the top of my head, I can't think of a reason it would need to be run inside a separate container. Assuming everything it needs is available in the Travis instance, of course.

[pento]

It should be available, and it's much easier to install on Travis (and change in the future, if necessary) if there's anything missing.

I'll close this issue, since it shouldn't require any docker image changes, but feel free to ping me on tickets/PRs elsewhere if you need extra eyeballs. 🙂

[getsource]

Okay. I think I figured out why it was likely set up that way.

I suspect it's because environment information is collected as part of the report, and that information is different where the tests are run (in the php docker container) vs. the TravisCI instance.

That's useful information for figuring out why the tests might be failing from one test reporter and not another.

[getsource]

Thinking on this a little more, an extra note:
We could fix it here, but I'm going to see if there's a good way to gather the environment information from an appropriate php docker instance, while still reporting back from the Travis instance.

The reporting part of the test runner could use some refactoring that might line up with this, too.

[pento]

Oh, yeah. That sounds familiar.

Exploring whether the info can be collected from the Docker container is a good option. (Ticket, for reference: /WordPress/phpunit-test-runner#111)

[getsource]

Thanks for the ping!

I've made progress on the refactoring, but it'd definitely be simpler if the environment check/gathering doesn't have to be separately uploaded or piped in as its own command.

Do you think it'd make sense to add rsync here, since that's the only one left now that git is in the image?

[pento]

Yeah, I think it's fine to put rsync in there, it's tiny.

[getsource]

Cool, I can put together a PR.
Right now, in core Travis, tests are only being submitted for the results from PHP 7.0.

I think ideally that would get bumped to a higher version right now (probably the subject for a PR/patch for core Travis), and certainly will be in the future.

Do you think it's better to only have the package added to the one version of PHP that is currently being used for result submission, or to add it to all of them? If only to one, we'll have to remember to update this when that happens.

If only adding to the one being used for tests, could consider a config argument/variable (like is used for composer) to make it clearer why the package is only on one version, and make it a little harder to miss when it gets changed on the core side.

[pento]

All of them are fine. Decisions, not options. 😉

[getsource]

Okay, thank you!

Still learning how this works, but submitted a PR to build the images for testing.

[getsource]

Thanks so much @pento !

Looks like we're almost there:
https://travis-ci.com/github/WordPress/wordpress-develop/jobs/316610104#L3394

I think this finishes things for this repo.

I'm guessing the next step is to pass the WPT_REPORT_API_KEY env variable to docker-compose for authentication here https://github.com/WordPress/wordpress-develop/blob/master/.travis.yml#L138.

I can start a trac ticket/patch for that part.

[getsource]

Trac ticket: https://core.trac.wordpress.org/ticket/49846