Scanners

This is a work in progress...

Basic call hierarchy

In Executor.scan() we have

    def scan(self, scanner, node_list):
        """Scan a list of this Executor's files (targets or sources) for
        implicit dependencies and update all of the targets with them.
        This essentially short-circuits an N*M scan of the sources for
        each individual target, which is a hell of a lot more efficient.
        """
        env = self.get_build_env()

        # TODO(batch):  scan by batches)
        deps = []
        if scanner:
            for node in node_list:
                node.disambiguate()
                s = scanner.select(node)
                if not s:
                    continue
                path = self.get_build_scanner_path(s)
                deps.extend(node.get_implicit_deps(env, s, path))
        else:
            kw = self.get_kw()
            for node in node_list:
                node.disambiguate()
                scanner = node.get_env_scanner(env, kw)
                if not scanner:
                    continue
                scanner = scanner.select(node)
                if not scanner:
                    continue
                path = self.get_build_scanner_path(scanner)
                deps.extend(node.get_implicit_deps(env, scanner, path))

        deps.extend(self.get_implicit_deps())

        for tgt in self.get_all_targets():
            tgt.add_to_implicit(deps)

So we either have a fixed scanner set for the currently used Builder, or (in the else path) we try to pick one from the SCANNERS as defined in the current environment.

Remark: Because Executor.get_build_scanner_path() is called in the context above

    def get_build_scanner_path(self, scanner):
        """Fetch the scanner path for this executor's targets and sources.
        """
        env = self.get_build_env()
        try:
            cwd = self.batches[0].targets[0].cwd
        except (IndexError, AttributeError):
            cwd = None
        return scanner.path(env, cwd,
                            self.get_all_targets(),
                            self.get_all_sources())

we have to ensure that whatever is returned by the Scanner.select() method, offers a path() method with a matching function signature.

Node.get_env_scanner(env) calls env.get_scanner(self.scanner_key()), so the node generates its skey, and then looks up a proper scanner in its build environment.

In env.get_scanner(skey), the env-wise dictionary _gsm gets initialized lazily. It contains the mapping of single keys (mostly suffixes, but can also be an arbitrary string) to a Scanner (or plain function).

When the dict _gsm gets created, the list of SCANNERS for the current environment is processed (in reverse order! -> provides first match for the same suffixes/skeys). For each scanner the method get_skeys(env) gets called, and returns the expanded (like for $CPPSUFFIXES!) list of suffixes/skeys the scanner can handle allegedly. For each of these list items (skeys), the current scanner is then registered by

  _gsm[skey] = scanner

in the env-global dict of scanners. This implicitly assumes that the list of skeys, as returned from scanner.get_skeys() is unique.

This mechanism allows us to have a different mapping of skey -> scanner, by using an environment variable like $CPPSUFFIXES when registering the Scanner in the current environment. We can then set CPPSUFFIXES to new values, but don't have to change anything in the Scanner (C or Swig for example).