Setup cruft for cookiecutter refresh #330
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Run `cruft create`, with answers matching Flyingpigeon then manually merge the 2 created repos.
|
Did someone forgot a file |
Fix following error:
```
$ make docs Generating docs with Sphinx ...
make[1]: Entering directory '/zstore/repos/flyingpigeon/docs'
Running Sphinx v3.0.4
Configuration error:
There is a programmable error in your configuration file:
Traceback (most recent call last): File "/home/lvu/.conda/envs/flyingpigeon-docs/lib/python3.7/site-packages/sphinx/config.py", line 319, in eval_config_file
execfile_(filename, namespace)
File "/home/lvu/.conda/envs/flyingpigeon-docs/lib/python3.7/site-packages/sphinx/util/pycompat.py", line 81, in execfile_
exec(code, _globals)
File "/zstore/repos/flyingpigeon/docs/source/conf.py", line 121, in <module>
import sphinx_rtd_theme
ModuleNotFoundError: No module named 'sphinx_rtd_theme'
Makefile:20: recipe for target 'html' failed
```
|
pymetalink not installable in |
It's done here. #329 |
|
I merged the PR (a bit too fast). Then opened another PR to fix remaining pep8 issues. |
…kiecutter-refresh
|
Can I remove |
See Travis-CI logs: https://api.travis-ci.org/v3/job/692554776/log.txt (https://travis-ci.org/github/bird-house/flyingpigeon/jobs/692554776#L1405) Fail make develop: ``` RROR: Could not find a version that satisfies the requirement pymetalink (from flyingpigeon==1.5.1) (from versions: none) ERROR: No matching distribution found for pymetalink (from flyingpigeon==1.5.1) Makefile:50: recipe for target 'develop' failed make: [develop] Error 1 (ignored) ``` Which in turn fail make test-notebooks because the pip install in make develop did not finish: ``` $ make test-notebooks Copying notebook output sanitizer ... Running notebook-based tests ERROR: usage: pytest [options] [file_or_dir] [file_or_dir] [...] pytest: error: unrecognized arguments: --nbval --sanitize-with /home/travis/build/bird-house/flyingpigeon/docs/source/output-sanitize.cfg inifile: /home/travis/build/bird-house/flyingpigeon/setup.cfg rootdir: /home/travis/build/bird-house/flyingpigeon Makefile:125: recipe for target 'test-notebooks' failed ``` ``` $ make docs Generating docs with Sphinx ... make[1]: Entering directory '/home/travis/build/bird-house/flyingpigeon/docs' Running Sphinx v3.0.4 Extension error: Could not import extension nbsphinx (exception: No module named 'nbsphinx') Makefile:20: recipe for target 'html' failed make[1]: *** [html] Error 2 make[1]: Leaving directory '/home/travis/build/bird-house/flyingpigeon/docs' Makefile:142: recipe for target 'docs' failed make: [docs] Error 2 (ignored) ```
|
We'll have to put it back again later because it is necessary for a number of processes. pymetalink has to be installed from the github repo url I believe. |
Yes you can remove it. I had to install pymetalink manually, since the pip is not working well. Check the notebooks and comment pymetalink out there. |
huard
approved these changes
May 29, 2020
Some FP processes would fail? Then pymetalink should have been in the environment.yml then? Because stuff in requirements_dev.txt is not in the docker image, these are extra installs for running tests and generating docs only. |
Which one? I included pymetalink while setting up the notebook examples |
|
Scrap that, it's only needed by the client accessing processes with META4 ComplexOutputs. |
Removed .travis.yml from skip list since this one is not diverging too much.
…kiecutter-refresh
pymetalink is needed for `make test-notebooks`:
Copying notebook output sanitizer ...
Running notebook-based tests
============================= test session starts ==============================
platform linux -- Python 3.7.6, pytest-5.4.2, py-1.8.1, pluggy-0.13.1 -- /home/lvu/.conda/envs/flyingpigeon/bin/python
cachedir: .pytest_cache
rootdir: /zstore/repos/flyingpigeon, inifile: setup.cfg
plugins: notebook-0.6.0, flake8-1.0.6, nbval-0.9.5
collecting ... collected 16 items
docs/source/notebooks/subset_processes::ipynb::Cell 0 PASSED [ 6%]
docs/source/notebooks/subset_processes::ipynb::Cell 1 PASSED [ 12%]
docs/source/notebooks/subset_processes::ipynb::Cell 2 PASSED [ 18%]
docs/source/notebooks/subset_processes::ipynb::Cell 3 PASSED [ 25%]
docs/source/notebooks/subset_processes::ipynb::Cell 4 FAILED [ 31%]
docs/source/notebooks/subset_processes::ipynb::Cell 5 PASSED [ 37%]
docs/source/notebooks/subset_processes::ipynb::Cell 6 PASSED [ 43%]
docs/source/notebooks/subset_processes::ipynb::Cell 7 PASSED [ 50%]
docs/source/notebooks/subset_processes::ipynb::Cell 8 PASSED [ 56%]
docs/source/notebooks/subset_processes::ipynb::Cell 9 PASSED [ 62%]
docs/source/notebooks/subset_processes::ipynb::Cell 10 FAILED [ 68%]
docs/source/notebooks/subset_processes::ipynb::Cell 11 FAILED [ 75%]
docs/source/notebooks/subset_processes::ipynb::Cell 12 FAILED [ 81%]
docs/source/notebooks/subset_processes::ipynb::Cell 13 PASSED [ 87%]
docs/source/notebooks/subset_processes::ipynb::Cell 14 PASSED [ 93%]
docs/source/notebooks/subset_processes::ipynb::Cell 15 PASSED [100%]
=================================== FAILURES ===================================
_____________ docs/source/notebooks/subset_processes.ipynb::Cell 4 _____________
�[91mNotebook cell execution failed�[0m
�[94mCell 4: Cell outputs differ
Input:
�[0m# Enter `fp?` for general exploration on processes provided by flyingpigeon
help(fp.subset_continents) # or type: fp.subset_countries?
�[94mTraceback:�[0m
�[94m mismatch 'stdout'�[91m
assert reference_output == test_output failed:
'Help on meth... outputs.\n\n' == 'Help on meth... outputs.\n\n'
Help on method subset_continents in module birdy.client.base:
- subset_continents(resource=None, region='Africa') method of birdy.client.base.WPSClient instance
+ subset_continents(resource=None, region='Africa', mosaic=False) method of birdy.client.base.WPSClient instance
? ++++++++++++++
Return the data whose grid cells intersect the selected continents for each input dataset.
Parameters
----------
region : {'Africa', 'Asia', 'Australia', 'North America', 'Oceania', 'South America', 'Antarctica', 'Europe'}string
Continent name.
+ mosaic : boolean
+ If True, selected regions will be merged into a single geometry.
resource : ComplexData:mimetype:`application/x-netcdf`, :mimetype:`application/x-tar`, :mimetype:`application/zip`
NetCDF Files or archive (tar/zip) containing netCDF files.
Returns
-------
output : ComplexData:mimetype:`application/x-netcdf`
NetCDF output for first resource file.
metalink : ComplexData:mimetype:`application/metalink+xml; version=4.0`
Metalink file with links to all NetCDF outputs.
�[0m
____________ docs/source/notebooks/subset_processes.ipynb::Cell 10 _____________
�[91mNotebook cell execution failed�[0m
�[94mCell 10: Cell execution caused an exception
Input:
�[0m# Run the process
resp = fp_i.subset_continents(resource=urls, region=['Europe', 'Africa'], mosaic=True)
timeout = time.time() + 60*5 # 5 minutes from now
while resp.getStatus() != 'ProcessSucceeded':
time.sleep(1)
if time.time() > timeout: # to avoid endless waiting if the process failed
break
cont = resp.get()
�[94mTraceback:�[0m
�[0;31m---------------------------------------------------------------------------�[0m
�[0;31mTypeError�[0m Traceback (most recent call last)
�[0;32m<ipython-input-11-ccabe938873f>�[0m in �[0;36m<module>�[0;34m�[0m
�[1;32m 1�[0m �[0;31m# Run the process�[0m�[0;34m�[0m�[0;34m�[0m�[0;34m�[0m�[0m
�[0;32m----> 2�[0;31m �[0mresp�[0m �[0;34m=�[0m �[0mfp_i�[0m�[0;34m.�[0m�[0msubset_continents�[0m�[0;34m(�[0m�[0mresource�[0m�[0;34m=�[0m�[0murls�[0m�[0;34m,�[0m �[0mregion�[0m�[0;34m=�[0m�[0;34m[�[0m�[0;34m'Europe'�[0m�[0;34m,�[0m �[0;34m'Africa'�[0m�[0;34m]�[0m�[0;34m,�[0m �[0mmosaic�[0m�[0;34m=�[0m�[0;32mTrue�[0m�[0;34m)�[0m�[0;34m�[0m�[0;34m�[0m�[0m
�[0m�[1;32m 3�[0m �[0;34m�[0m�[0m
�[1;32m 4�[0m �[0;31m# using fp_i you need to wait until the processing is complete!�[0m�[0;34m�[0m�[0;34m�[0m�[0;34m�[0m�[0m
�[1;32m 5�[0m �[0mtimeout�[0m �[0;34m=�[0m �[0mtime�[0m�[0;34m.�[0m�[0mtime�[0m�[0;34m(�[0m�[0;34m)�[0m �[0;34m+�[0m �[0;36m60�[0m�[0;34m*�[0m�[0;36m5�[0m �[0;31m# 5 minutes from now�[0m�[0;34m�[0m�[0;34m�[0m�[0m
�[0;31mTypeError�[0m: subset_continents() got an unexpected keyword argument 'mosaic'
____________ docs/source/notebooks/subset_processes.ipynb::Cell 11 _____________
�[91mNotebook cell execution failed�[0m
�[94mCell 11: Cell execution caused an exception
Input:
�[0m# plot the test file (url of output) with the flyingpigon plot function
resp = fp.plot_map_timemean(resource=cont.output)
�[94mTraceback:�[0m
�[0;31m---------------------------------------------------------------------------�[0m
�[0;31mNameError�[0m Traceback (most recent call last)
�[0;32m<ipython-input-12-bc56cd51fcc0>�[0m in �[0;36m<module>�[0;34m�[0m
�[1;32m 1�[0m �[0;31m# plot the test file (url of output) with the flyingpigon plot function�[0m�[0;34m�[0m�[0;34m�[0m�[0;34m�[0m�[0m
�[0;32m----> 2�[0;31m �[0mresp�[0m �[0;34m=�[0m �[0mfp�[0m�[0;34m.�[0m�[0mplot_map_timemean�[0m�[0;34m(�[0m�[0mresource�[0m�[0;34m=�[0m�[0mcont�[0m�[0;34m.�[0m�[0moutput�[0m�[0;34m)�[0m�[0;34m�[0m�[0;34m�[0m�[0m
�[0m
�[0;31mNameError�[0m: name 'cont' is not defined
____________ docs/source/notebooks/subset_processes.ipynb::Cell 12 _____________
�[91mNotebook cell execution failed�[0m
�[94mCell 12: Cell execution caused an exception
Input:
�[0m# The plot process returnes one graphic file.
resp.get(asobj=True).plotout_map
�[94mTraceback:�[0m
�[0;31m---------------------------------------------------------------------------�[0m
�[0;31mAttributeError�[0m Traceback (most recent call last)
�[0;32m<ipython-input-13-3203d44d7305>�[0m in �[0;36m<module>�[0;34m�[0m
�[1;32m 1�[0m �[0;31m# The plot process returnes one graphic file.�[0m�[0;34m�[0m�[0;34m�[0m�[0;34m�[0m�[0m
�[1;32m 2�[0m �[0;31m# It can be plotted directly by asking birdy to get python objects, instead of links to files.�[0m�[0;34m�[0m�[0;34m�[0m�[0;34m�[0m�[0m
�[0;32m----> 3�[0;31m �[0mresp�[0m�[0;34m.�[0m�[0mget�[0m�[0;34m(�[0m�[0masobj�[0m�[0;34m=�[0m�[0;32mTrue�[0m�[0;34m)�[0m�[0;34m.�[0m�[0mplotout_map�[0m�[0;34m�[0m�[0;34m�[0m�[0m
�[0m
�[0;31mAttributeError�[0m: 'subset_bboxResponse' object has no attribute 'plotout_map'
=============================== warnings summary ===============================
/home/lvu/.conda/envs/flyingpigeon/lib/python3.7/site-packages/nbval/plugin.py:115
/home/lvu/.conda/envs/flyingpigeon/lib/python3.7/site-packages/nbval/plugin.py:115: PytestDeprecationWarning: direct construction of IPyNbFile has been deprecated, please use IPyNbFile.from_parent
return IPyNbFile(path, parent)
/home/lvu/.conda/envs/flyingpigeon/lib/python3.7/site-packages/nbval/plugin.py:312: 16 tests with warnings
/home/lvu/.conda/envs/flyingpigeon/lib/python3.7/site-packages/nbval/plugin.py:312: PytestDeprecationWarning: direct construction of IPyNbCell has been deprecated, please use IPyNbCell.from_parent
cell, options)
docs/source/notebooks/subset_processes.ipynb::Cell 0
/home/lvu/.conda/envs/flyingpigeon/lib/python3.7/site-packages/jupyter_client/manager.py:63: DeprecationWarning: KernelManager._kernel_spec_manager_changed is deprecated in traitlets 4.1: use @observe and @unobserve instead.
def _kernel_spec_manager_changed(self):
-- Docs: https://docs.pytest.org/en/latest/warnings.html
=========================== short test summary info ============================
FAILED docs/source/notebooks/subset_processes.ipynb::Cell 4
FAILED docs/source/notebooks/subset_processes.ipynb::Cell 10
FAILED docs/source/notebooks/subset_processes.ipynb::Cell 11
FAILED docs/source/notebooks/subset_processes.ipynb::Cell 12
================== 4 failed, 12 passed, 18 warnings in 12.76s ==================
Makefile:125: recipe for target 'test-notebooks' failed.
Had to move it out of requirements_dev.txt and into make develop itself
as a work-around.
Below is error when pymetalink install link is in requirements_dev.txt:
```
$ make develop
Installing development requirements for tests and docs ...
Obtaining file:///zstore/repos/flyingpigeon
ERROR: Command errored out with exit status 1:
command: /home/lvu/.conda/envs/flyingpigeon/bin/python3.7 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/zstore/repos/flyingpigeon/set
up.py'"'"'; __file__='"'"'/zstore/repos/flyingpigeon/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n
'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-qqaszq3v
cwd: /zstore/repos/flyingpigeon/
Complete output (1 lines):
error in flyingpigeon setup command: 'extras_require' must be a dictionary whose values are strings or lists of strings containing valid project/v
ersion requirement specifiers.
----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
```
I was not able to add pandoc to requiremnts_dev.txt, it was hanging for me so had to add to environment.yml. https://travis-ci.org/github/bird-house/flyingpigeon/builds/692680294 (https://api.travis-ci.org/v3/job/692680296/log.txt) ``` $ make docs Generating docs with Sphinx ... make[1]: Entering directory '/home/travis/build/bird-house/flyingpigeon/docs' Running Sphinx v3.0.4 making output directory... done building [mo]: targets for 0 po files that are out of date building [html]: targets for 15 source files that are out of date updating environment: [new config] 15 added, 0 changed, 0 removed ../README.rst:2: WARNING: Duplicate explicit target name: "wikipedia". ../README.rst:2: WARNING: Duplicate explicit target name: "wikipedia". Notebook error: PandocMissing in notebooks/subset_processes.ipynb: Pandoc wasn't found. Please check that pandoc is installed: http://pandoc.org/installing.html Makefile:20: recipe for target 'html' failed make[1]: *** [html] Error 2 make[1]: Leaving directory '/home/travis/build/bird-house/flyingpigeon/docs' Makefile:143: recipe for target 'docs' failed make: [docs] Error 2 (ignored) ```
|
I got everthing working in Travis-CI (pytest, flakes8 (thanks David for flake8 fix), make docs) except the notebooks (yes notebooks now run part of Travis-CI !). I will need help for these errors. Locally, you can reproduce with
https://travis-ci.org/github/bird-house/flyingpigeon/jobs/692686563 (https://api.travis-ci.org/v3/job/692686563/log.txt) |
|
I won't be able to fix those before June 12th, so we either merge this as is or someone else fixes this in the meantime. |
Zeitsperre
reviewed
May 29, 2020
I can help out next week. |
As of May 2020, the bump2version fork has adopted bumpversion to continue development. Co-authored-by: Trevor James Smith <[email protected]>
|
@nilshempelmann I just allowed Let me know if you still intend to fix the notebook failures this week or I can simply merge this PR as-is and you can look at the notebooks later. At least now the notebooks are running and we have a trace or their status. Just to be extra sure, can I have admin on https://flyingpigeon.readthedocs.io to test RtD build for this PR? |
tlvu
added a commit
to bird-house/cookiecutter-birdhouse
that referenced
this pull request
Jun 5, 2020
Backport from Flyingpigeon and Emu. Backport a few differences when synching with Flyingpigeon and Emu. Some noticeable changes: * Fixed wrong `OUTPUT_URL` in target refresh-notebooks. * Fixed wrong http port in Travis-CI breaking `make test-notebooks`. * Added `make docs` in Travis-CI. Matching FlyingPigeon PR bird-house/flyingpigeon#330 and Emu PR bird-house/emu#103
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR is part of issue bird-house/cookiecutter-birdhouse#61, rolling out cruft as a mechanism to easily update cookiecutter template changes to existing project.
Require cookiecutter-birdhouse PR bird-house/cookiecutter-birdhouse#89
In
.cruft.json, I set"project_short_description": "A Web Processing Service Testbed."from theREADME.rst, let me know if that is still correct.In
README.rst, I took the Flyingpigeon bird description fromdocs/source/index.rst.Noticeable changes:
Travis-CI build working: https://travis-ci.org/github/bird-house/flyingpigeon/builds/694013491
RtD build working: https://readthedocs.org/projects/flyingpigeon/builds/11184473/