Repository structure & build process

Introduction

The code in this repository is organized into four types of things:

• Package mapping
• Pattern mapping
• Builder files
• Templates

These four types of things also generally follow the build process, where the OSID spec is first parsed, then methods are mapped to patterns, and code templates applied to generate the final implementation.

Package mapping

This mapper is in the xosid_mapper.py file, and converts the xosid XML files into JSON for each package. It generates a package_maps/ directory one level up (as a sibling to the builder directory), with a JSON file for each package. Each of the interfaces is broken out, with these characteristics:

• Methods (with arguments, doc string, and return type)
• Doc string
• Inheritance
• Category (i.e. manager, object, session)

Example package map

An example package map for the resource service can be found here.

Build command

This can be run individually via python build_dlkit.py map.

Pattern mapping

It is assumed that you have run the xosid_mapper.py already and created the package maps, before trying to run the pattern mappers. If you just run the builder generically (python build_dlkit.py or python build_dlkit.py --all), then it handles the sequence of steps for you automatically.

This mapper is in the pattern_mapper.py file, and scans through the package maps in the ../package_maps/ directory. It then generates a pattern_maps/ directory as a sibling to the package maps (and sibling to the builder repo), with a JSON file for each package. For each of the interfaces and methods in the package maps, the pattern mapper uses a set of rules (located in pattern_mappers/) to determine if the method / interface follows a pattern. If it does, then the rules specify which package / interface / method is the base template to follow. For example, in pattern_mappers/objects.py, you can see that form methods which set a single argument all follow a pattern, which is defined in the resource package, Resource interface, set_group template. For the persistence implementation, that is in jsonosid_templates/resource.py.

Example pattern map

An example pattern map for the resource service can be found here.

Build command

This can be run individually via python build_dlkit.py patterns.

Builders

The builders read from the pattern and package maps, then use that data to pull in either the specified template or a hand-written implementation, and output the final code into the specified directory. Hand-written implementations take precendence over templates, if provided. If no template exists and no hand-written implementation is provided, then the method / interface is not output.

There exist three "core" builder files, that provide the scaffolding and structure for the build process.

• build_dlkit.py
• interface_builders.py
• method_builders.py

There are also adapter or persistence-specific adapters which inherit from the core builders. These builders either use the default methods from the core builders or override the default methods to accommodate special conditions. Many of these builder files appear in the repository, but most are old / deprecated / experimental. The ones actively used for the Python implementation are:

• azbuilder.py
• jsonbuilder.py
• kitbuilder.py
• mdatabuilder.py
• testbuilder.py

Core builders

The core builder classes encapsulate the generic builder workflow and provide the foundation for the adapter and platform-specific builders. The build process in general is described by this flowchart.

For Python builders, you probably only need to override the methods from method_builders.py and build_dlkit.py -- the boxes in gray and blue.

For other languages, you may need to override certain methods from interface_builders.py, too (the green-ish boxes). For example, if the target language is JavaScript, you probably don't need to include the __init__.py file that Python requires for modules. Or for Java, you may need to include multiple nested levels of directories.

You can look at the various persistence and adapter builders to see which methods they override, and hopefully get a sense of which ones you may need to override as well.

build_dlkit

build_dlkit.py

This file holds a lot of utility methods common to both the interface builder and the method builders, that help identify various directories or package names.

Interface builder

interface_builders.py

This builder tries to only deal with this at the interface level -- class-level init methods, imports, and doc strings.

This will create or update a set of interface map files, in interface_maps/ located in a sibling directory to the builder. These are referred to later on in the builder to make sure inherited classes are correctly imported.

Method builder

method_builders.py

This builder tries to only deal with building at the individual method level, including imports, method definition, and doc strings.

Templates

Each builder has its own set of templates, which are identified in the init methods for each builder. For example, jsonbuilder.py looks for its templates in jsonosid_templates.

The templates are organized according to package, and within each, there are classes that correspond to each interface.

Patterned methods

For the patterned methods, you will see an attribute name that ends in _template, like here. These templates use Python's simple string templating to let the builders insert context variables. For example:

    supports_visible_federation_template = """
        # Implemented from template for
        # osid.resource.ResourceProfile.supports_visible_federation
        return '${method_name}' in profile.SUPPORTS"""

The template variable ${method_name} is defined in method_builders.py's context, generated here. A similar context method exists in interface_builders.py, for the init methods.

Imports for templated methods

You'll notice that all interfaces with templates also have an attribute called import_statements_pattern. This lets these imports be also imported by any interfaces / methods that inherit from one of the templated methods. For example, any method that uses supports_visible_federation_template will also pull in these imports to it's module definition.

Hand-written implementations

For non-patterned methods and interfaces, you need to provide a hand-written implementation. These are typically one-off instances, many of which reside in the assessment package. In this case, you'll see that the attribute names do not include _template, and instead of import_statements_pattern we just use import_statements.

If you want to override a template, you also need to provide a hand-written implementation, and call it the expected name of the method. For example, if there is a method get_thing that inherits from Resource::get_group_template, but you want to provide a hand-written implementation, you would do:

class MySpecialSession:
    get_thing = """
        print('Are you happy?')"""

Argument defaults

To include default argument values, you can include a list of values, like here. This provides the default values for the match_keyword method's 2nd and 3rd arguments. The dictionary key is the argument index (starting with 0), so in this case 1 => 2nd argument.

    match_keyword_arg_template = {
        1: 'DEFAULT_STRING_MATCH_TYPE',
        2: True
    }

    match_keyword = """
        # Note: this currently ignores match argument
        match_value = self._get_string_match_value(keyword, string_match_type)
        for field_name in self._keyword_fields:
            if field_name not in self._keyword_terms:
                self._keyword_terms[field_name] = {'$in': list()}
            self._keyword_terms[field_name]['$in'].append(match_value)"""

Note that you must follow Python syntax in our case, where you cannot have non-kwarg arguments after kwargs. So in the above example, you cannot just provide this, because the method has 3 arguments:

    match_keyword_arg_template = {
        1: 'DEFAULT_STRING_MATCH_TYPE'
    }

Additional methods

Sometimes we need to add non-OSID methods for convenience. Sometimes, these are included in the init blocks, like here:

    init = """
    def __init__(self):
        self._runtime = None
        self._config = None

    def _initialize_manager(self, runtime):
        \"\"\"Sets the runtime, configuration and json client\"\"\"
        if self._runtime is not None:
            raise errors.IllegalState('this manager has already been initialized.')
        self._runtime = runtime
        self._config = runtime.get_configuration()
        set_json_client(runtime)
"""

Another way to add these is via an additional_methods attribute, like here:

    additional_methods = """
    def get_object_map(self):
        obj_map = dict(self._my_map)
        if 'agentIds' in obj_map:
            del obj_map['agentIds']
        return osid_objects.OsidObject.get_object_map(self, obj_map)
    object_map = property(fget=get_object_map)"""

You can also add additional interfaces this way, like seen here with AssetContentLookupSession.

class AssetLookupSession:

    additional_methods = """
class AssetContentLookupSession(abc_repository_sessions.AssetContentLookupSession, osid_sessions.OsidSession):
    \"\"\"This session defines methods for retrieving asset contents.
    An ``AssetContent`` represents an element of content stored associated
    with an ``Asset``.