-
Notifications
You must be signed in to change notification settings - Fork 66
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
various minor changes, and updated opendap mapper for Arome Arctic forecast data #533
Open
mortenwh
wants to merge
71
commits into
nansencenter:master
Choose a base branch
from
mortenwh:master
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
71 commits
Select commit
Hold shift + click to select a range
56996cc
test that history metadata is added correctly
mortenwh 7e5a1f6
updated baseURLs to s1 thredds
mortenwh fa71c83
Merge branch 'nansencenter:master' into master
mortenwh 9677223
Merge branch 'nansencenter:master' into master
mortenwh fceb4b5
removed print line which hampers readability - the line was printed e…
mortenwh 6d715b9
added exception handling of value error
mortenwh 68c01b7
Merge branch 'master' of github.com:nansencenter/nansat
mortenwh 1a53f4b
use != instead of is not. Mappername default changed to None would be…
mortenwh f754a8a
Merge branch 'master' of github.com:mortenwh/nansat
mortenwh d90e2e3
This is not an error. Might be a warning but it is still annoying, so…
mortenwh 18fc477
Merge remote-tracking branch 'nersc/master'
mortenwh 365ec55
Merge remote-tracking branch 'nersc/master'
mortenwh 21e4334
#525: added function to export with xarray, some cleaning, and tests
mortenwh 0ca6093
#525: ipdb lines were not meant to be committed..
mortenwh d1be8e9
#525: removed ipdb lines
mortenwh 110789c
#525: remover xarray based export function and modified the export fu…
mortenwh 2c45363
#525: removed unnecessary test
mortenwh d4d6495
#525: cleaned test code and added one test to demonstrate issue when …
mortenwh f276d0e
#525: static methods
mortenwh a671dc8
#525: adjusted netcdf-cf mapper to account for new attribute names wi…
mortenwh 54f17ab
#525: gdal adds its own Conventions attribute with value 'CF-1.5' whi…
mortenwh cfc5a28
#525: history is cut by gdal CreateCopy. This needs to be overridden.
mortenwh 5562633
get proj4 string from grid mapping variable
mortenwh f797b68
specify bands in opendap arome mapper since files are too big
mortenwh eb5e6fb
update to new netcdf attributes
mortenwh aebc93c
working mapper for opendap arome arctic
mortenwh 7876ece
opendap-arome now works
mortenwh d1bffc6
update box
mortenwh 1a62f3d
some error handling
mortenwh 4ed3fae
Merge branch 'master' of github.com:mortenwh/nansat
mortenwh 918e0ec
merged with nersc, and resolved conflicts
mortenwh 6c77c9a
Merge remote-tracking branch 'nersc/master'
mortenwh 3f488fb
start meps mapper
mortenwh 0e8447f
Should work now but does not..
mortenwh 4e71e74
update
mortenwh 6cdea78
reorganise
mortenwh 90e09f9
MEPS mapper works
mortenwh 2e7dedf
Add MET Nordic mapper
mortenwh b26dedb
add source filename to metadata
mortenwh f4306c5
Now add MET Nordic mapper
mortenwh 474f05b
Merge branch 'master' of github.com:mortenwh/nansat
mortenwh f43e9b5
fix encoding of metadata
mortenwh 251dba1
reorganise and change apostrophes
mortenwh a9556bb
Handle non-string attributes
mortenwh cd92904
Remove warnings
mortenwh 731e5fb
Make it possible to avoid creating gcps
mortenwh f410d5b
Add time metadata to bands
mortenwh 913c7aa
allow missing history, and set timezone to utc
mortenwh 7097cfd
find correct nc-file based on input time
mortenwh e222105
Assert correct gcp shape
mortenwh a3457f1
better handle input dict
mortenwh cdb110d
remove ipdb lines
mortenwh 03e579a
bug fix
mortenwh a559d61
started
mortenwh 2744857
Fix gcp shape by looping
mortenwh d0d4f72
improve gcp reading
mortenwh a7c92af
log gcp shape
mortenwh 09c6079
Init from lonlat
mortenwh cf52890
Merge branch 'master' of github.com:mortenwh/nansat
mortenwh 7c5af87
Use lat/lon grids for initialization
mortenwh a6af99f
bug fix?
mortenwh ed63700
bug fix
mortenwh 4be193c
resolve conflict
mortenwh 7e576c3
Adjust gcp sizes
mortenwh 3e98589
get rid of potential norwegian characters
mortenwh ae61f24
Merge branch 'master' of github.com:mortenwh/nansat
mortenwh 6b244c6
new mapper
mortenwh d17666b
Merge branch 'master' of github.com:mortenwh/nansat
mortenwh b1d7ede
correct docstring
mortenwh 191914f
fix docstring
mortenwh 2d634ce
reorganise arrays
mortenwh File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,112 @@ | ||
import json | ||
import pytz | ||
import datetime | ||
|
||
import pythesint as pti | ||
|
||
from osgeo import gdal | ||
from pyproj import CRS | ||
from netCDF4 import Dataset | ||
|
||
from nansat.nsr import NSR | ||
from nansat.exceptions import WrongMapperError | ||
from nansat.mappers.mapper_netcdf_cf import Mapper as NetcdfCF | ||
from nansat.mappers.opendap import Opendap | ||
|
||
|
||
class Mapper(NetcdfCF, Opendap): | ||
|
||
def __init__(self, url, gdal_dataset, gdal_metadata, file_num=0, bands=None, *args, | ||
**kwargs): | ||
|
||
if not url.endswith(".nc"): | ||
raise WrongMapperError | ||
|
||
try: | ||
ds = Dataset(url) | ||
except OSError: | ||
raise WrongMapperError | ||
|
||
if "title" not in ds.ncattrs() or "meps" not in ds.getncattr("title").lower(): | ||
raise WrongMapperError | ||
|
||
metadata = {} | ||
for attr in ds.ncattrs(): | ||
content = ds.getncattr(attr) | ||
if isinstance(content, str): | ||
content = content.replace("æ", "ae").replace("ø", "oe").replace("å", "aa") | ||
metadata[attr] = content | ||
|
||
self.input_filename = url | ||
|
||
xsize = ds.dimensions["x"].size | ||
ysize = ds.dimensions["y"].size | ||
|
||
# Pick 10 meter height dimension only | ||
height_dim = "height6" | ||
if height_dim not in ds.dimensions.keys(): | ||
raise WrongMapperError | ||
if ds.dimensions[height_dim].size != 1: | ||
raise WrongMapperError | ||
if ds.variables[height_dim][0].data != 10: | ||
raise WrongMapperError | ||
|
||
varnames = [] | ||
for var in ds.variables: | ||
var_dimensions = ds.variables[var].dimensions | ||
if var_dimensions == ("time", height_dim, "y", "x"): | ||
varnames.append(var) | ||
|
||
# Projection | ||
try: | ||
grid_mapping = ds.variables[ds.variables[varnames[0]].grid_mapping] | ||
except KeyError: | ||
raise WrongMapperError | ||
|
||
grid_mapping_dict = {} | ||
for index in grid_mapping.ncattrs(): | ||
grid_mapping_dict[index] = grid_mapping.getncattr(index) | ||
crs = CRS.from_cf(grid_mapping_dict) | ||
nsr = NSR(crs.to_proj4()) | ||
|
||
# Geotransform | ||
xx = ds.variables["x"][0:2] | ||
yy = ds.variables["y"][0:2] | ||
gtrans = xx[0], xx[1]-xx[0], 0, yy[0], 0, yy[1]-yy[0] | ||
|
||
self._init_from_dataset_params(xsize, ysize, gtrans, nsr.wkt) | ||
|
||
meta_dict = [] | ||
if bands is None: | ||
bands = varnames | ||
for band in bands: | ||
if band not in ds.variables.keys(): | ||
continue | ||
dimension_names, dim_sizes = self._get_dimension_info(band) | ||
self._pop_spatial_dimensions(dimension_names) | ||
index = self._get_index_of_dimensions(dimension_names, {}, dim_sizes) | ||
fn = self._get_sub_filename(url, band, dim_sizes, index) | ||
band_metadata = self.get_band_metadata_dict(fn, ds.variables[band]) | ||
# Add time stamp to band metadata | ||
tt = datetime.datetime.fromisoformat(str(self.times()[index["time"]["index"]])) | ||
if tt.tzinfo is None: | ||
tt = pytz.utc.localize(tt) | ||
band_metadata["dst"]["time"] = tt.isoformat() | ||
meta_dict.append(band_metadata) | ||
|
||
self.create_bands(meta_dict) | ||
|
||
# Copy metadata | ||
for key in metadata.keys(): | ||
self.dataset.SetMetadataItem(str(key), str(metadata[key])) | ||
|
||
# Get dictionary describing the instrument and platform according to | ||
# the GCMD keywords | ||
mm = pti.get_gcmd_instrument("computer") | ||
ee = pti.get_gcmd_platform("models") | ||
|
||
self.dataset.SetMetadataItem("instrument", json.dumps(mm)) | ||
self.dataset.SetMetadataItem("platform", json.dumps(ee)) | ||
|
||
# Set input filename | ||
self.dataset.SetMetadataItem("nc_file", self.input_filename) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,40 @@ | ||
import os | ||
import pytz | ||
import netCDF4 | ||
import datetime | ||
|
||
import numpy as np | ||
|
||
from nansat.exceptions import WrongMapperError | ||
from nansat.mappers.mapper_meps import Mapper as NCMapper | ||
|
||
class Mapper(NCMapper): | ||
|
||
|
||
def __init__(self, ncml_url, gdal_dataset, gdal_metadata, netcdf_dim=None, *args, **kwargs): | ||
|
||
if not ncml_url.endswith(".ncml"): | ||
raise WrongMapperError | ||
|
||
dt = datetime.timedelta(0) | ||
if netcdf_dim is not None and "time" in netcdf_dim.keys(): | ||
ds = netCDF4.Dataset(ncml_url) | ||
time = netcdf_dim["time"].astype(datetime.datetime).replace( | ||
tzinfo=pytz.timezone("utc")) | ||
dt = time - datetime.datetime.fromisoformat(ds.time_coverage_start.replace( | ||
"Z", "+00:00")) | ||
url = self._get_odap_url(ncml_url, np.round(dt.total_seconds()/3600)) | ||
|
||
super(Mapper, self).__init__(url, gdal_dataset, gdal_metadata, *args, **kwargs) | ||
|
||
|
||
def _get_odap_url(self, fn, file_num=0): | ||
""" Get the opendap url to file number 'file_num'. The | ||
default file number is 0, and yields the forecast time. | ||
""" | ||
url = ( | ||
"" + os.path.split(fn)[0] + "/member_%02d" | ||
"/meps_" + os.path.basename(fn).split("_")[2] + | ||
"_%02d_" + os.path.basename(fn).split("_")[3][:-2] | ||
) % (int(os.path.basename(fn)[8:11]), file_num) | ||
return url |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This change highlights a problem in this test:
the arguments to
-te
are given in the wrong order.Could you change
"-te -180 180 60 90 -ts 500 500"
to"-te -180 60 180 90 -ts 500 500"
?