Skip to content

Commit

Permalink
Merge pull request #69 from HDI-Project/issue_3_use_entry_points_for_…
Browse files Browse the repository at this point in the history
…mlprimitives_jsons

Issue 3 use entry points for mlprimitives jsons
  • Loading branch information
csala authored Jan 11, 2019
2 parents eb08f7c + 82c84c5 commit d0e1c8a
Show file tree
Hide file tree
Showing 104 changed files with 19 additions and 33 deletions.
1 change: 1 addition & 0 deletions MANIFEST.in
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ include CONTRIBUTING.rst
include HISTORY.md
include LICENSE
include README.md
include mlprimitives/jsons/*.json

recursive-include tests *
recursive-exclude * __pycache__
Expand Down
20 changes: 10 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,16 +30,6 @@ either combine third party tools or implement new functionalities from scratch.

The project is divided in three parts:

### The `mlblocks_primitives` folder

The `mlblocks_primitives` folder contains the JSON annotations for the MLBlocks primitives.

This folder has a flat structure, without subfolders, and all the primitive JSONs are named
after the Fully Qualified Name of the annotated primitive (function or class).

As a result of this, sorting the JSON files alphabetically shows them grouped by library, which
makes browsing them and seeing what tools are implemented easy.

### The `mlprimitives` package

The mlprimitives folder is where all the Python code can be found.
Expand All @@ -48,6 +38,16 @@ Several sub-modules exist inside it, for the different types of primitives imple
the `mlprimitives.adapters` module, which has a special role in the integration of third
party tools that do not directly fit the MLBlocks requirements.

### The `mlprimitives/jsons` folder

The `mlprimitives/jsons` folder contains the JSON annotations for the primitives.

This folder has a flat structure, without subfolders, and all the primitive JSONs are named
after the Fully Qualified Name of the annotated primitive (function or class).

As a result of this, sorting the JSON files alphabetically shows them grouped by library, which
makes browsing them and seeing what tools are implemented easy.

### The `tests` folder

Here are the unit tests for the Python code, as well as some validation tests for the JSON
Expand Down
1 change: 1 addition & 0 deletions mlprimitives/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@

from mlprimitives.evaluation import score_pipeline

MLPRIMITIVES_JSONS_PATH = os.path.join(os.path.dirname(__file__), 'jsons')
LOGGER = logging.getLogger(__name__)


Expand Down
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
15 changes: 4 additions & 11 deletions setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@


tests_require = [
'mlblocks>=0.2.4',
'mlblocks>=0.3.0',
'pytest>=3.4.2',
'google-compute-engine==2.8.12', # required by travis
]
Expand Down Expand Up @@ -84,15 +84,6 @@
}


json_primitives = glob.glob('mlblocks_primitives/**/*.json', recursive=True)
data_files = defaultdict(list)
for primitive_path in json_primitives:
parts = primitive_path.split('/')
dir_path = parts[1:-1]
install_path = os.path.join('mlblocks_primitives', *dir_path)
data_files[install_path].append(primitive_path)


setup(
author="MIT Data To AI Lab",
author_email='[email protected]',
Expand All @@ -105,12 +96,14 @@
'Programming Language :: Python :: 3.5',
'Programming Language :: Python :: 3.6',
],
data_files = list(data_files.items()),
description="MLBlocks Primitives",
entry_points = {
'console_scripts': [
'mlprimitives=mlprimitives:_main'
],
'mlprimitives': [
'jsons_path=mlprimitives:MLPRIMITIVES_JSONS_PATH'
]
},
extras_require=extras_require,
install_requires=install_requires,
Expand Down
15 changes: 3 additions & 12 deletions tests/test_mlblocks_primitives.py → tests/test_jsons.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,18 +2,10 @@

import json
import os
from unittest.mock import patch

from mlblocks import MLPipeline

PRIMITIVES_PATH = os.path.normpath(
os.path.join(
os.path.dirname(__file__),
os.pardir,
'mlblocks_primitives'
)
)

from mlprimitives import MLPRIMITIVES_JSONS_PATH

HYPERPARAMETER_DEFAULTS = {
'int': 1,
Expand All @@ -24,14 +16,13 @@
}


@patch('mlblocks.primitives._PRIMITIVES_PATHS', new=[PRIMITIVES_PATH])
def test_jsons():
"""Validate MLBlocks primitive jsons"""

primitives = (f for f in os.listdir(PRIMITIVES_PATH) if f.endswith('.json'))
primitives = (f for f in os.listdir(MLPRIMITIVES_JSONS_PATH) if f.endswith('.json'))
for primitive_filename in primitives:
try:
primitive_path = os.path.join(PRIMITIVES_PATH, primitive_filename)
primitive_path = os.path.join(MLPRIMITIVES_JSONS_PATH, primitive_filename)
with open(primitive_path, 'r') as f:
primitive = json.load(f)

Expand Down
File renamed without changes.

0 comments on commit d0e1c8a

Please sign in to comment.