repository -> storage

CHANGELOG.md

@@ -2,16 +2,18 @@
 
 ### 3.5.0 (pending)
 
+* node storage now only handles checkouts -- local maven repository or svn no longer supported
 * token handling
   * load token from environment POMMES_REPOSITORY_TOKEN_<name> variable, not file
   * github + gitlab
     * load github token from "git credentials"
-    * explicitly declare "§§" if repository needs a token
-* default repositories
-  * the first repository defined in config is the default repository
-  * setup automatically defines default repository local = $POMMES_ROOT
-  * changed queries to only search the default repository by default;
-    prefix a query with "/" <repo> to search a different repository; prefix with "/" to search all repositories
+    * explicitly declare "§§" if storage needs a token
+* major repository cleanup, they are now called storage. The config file changed accordingly
+* default storage
+  * the first storage defined in config is the default storage
+  * setup automatically defines default storage local = $POMMES_ROOT
+  * changed queries to only search the default storage by default;
+    prefix a query with "/" <repo> to search a different storage; prefix with "/" to search all storages
 * simplified setup: 
   * root directory is an optional argument to `setup` now
   * default root directory changed from `Pommes` to `Projects`
@@ -22,8 +24,8 @@
   * more robust: handle io exception to that user interaction can be continued
 * tweaks and fixes
   * normalize git urls to avoid conflicting checkout problems
-  * fixed repository scm for file repositories
-  * fixed RawDescriptor to properly set repositoryScm
+  * fixed file storage scm
+  * fixed RawDescriptor to properly set storageScm
   * pom loading: use developer connection instead of connection, because it's usually better maintained
   * do not abort on pom loading errors, just report them
 * build
@@ -52,9 +54,9 @@
     the former POMMES_HOME (i.e `.pommes`); default is 'Pommes'
   * renamed pommes.properties to config
   * property `checkouts` is gone, it's now configured via `POMMES_ROOT`
-  * renamed import to repository
+  * renamed import to storage
 * tailing `.git` in git scm url is now optional
-* renamed zone to repository
+* renamed zone to storage
 * fixed NodeRepository to detect raw descriptors in none-root directories
 * fixed NodeRepository to check for trunk/branches/tags only for svn nodes (because ~/Projects/branches confused it)
 * updates
@@ -169,7 +171,7 @@
   (and pushed for git projects) now, and it prints an 'X' status if the project is unknwon but in scm.
 * Improved action selection: return quits without actions now, a applies all actions.
 * Dumped -svnuser and -svnpassword options. Specify credentials in the url instead.
-  (I've removed them because I don't want special handling for Subversion repository authentication.)
+  (I've removed them because I don't want special handling for Subversion storage authentication.)
 * Improved find argument: use can use query and format macros; the format is appended now; you can merge duplicate lines '-fold'.
   Merged database-export into find. Find now has an optional output argument (which also accepts files now) and -json and -dump
   formats to print pretty-printed or raw json.

README.md

@@ -16,7 +16,7 @@ This command searches the database for `lavender` projects and lets you choose b
 simply cd into the already existing checkout, choosing `2` also creates the checkout. 
 
 Technically, Pommes is a command line tool that maintains a database with project metadata. Pommes can:
-* index repositories (e.g. github or github) to populate the database
+* index storages (e.g. github or github) to populate the database
 * search the database by coordinates, dependencies, scm location, etc. 
 * perform (bulk-) checkouts for all projects that match a query.
 
@@ -58,25 +58,25 @@ Install the application
 ## Managing the Database
 
 Pommes maintains a [Lucene](http://lucene.apache.org) database to store project metadata. The database is filled
-by indexing so-called repositories. The database is stored in `$POMMES_ROOT/.pommes/database`, it's safe to delete this
+by indexing so-called storages. The database is stored in `$POMMES_ROOT/.pommes/database`, it's safe to delete this
 directory to wipe the database.
 
-`.pommes/config` defines databases in the form *repository.<name> = <url> {<option>}*
+`.pommes/config` defines databases in the form *storage.<name> = <url> {<option>}*
 * name is an arbitrary name
-* url is the primary definition of the repository, the protocol distinishes the repository type
+* url is the primary definition of the storage, the protocol distinishes the storage type
   * github:<url-of-github-api>
   * gitlab:<url-of-gitlab-server>
   * json:<url-pointing-to-json-file>
   * file:///path-to-local-directory-with checkouts
-* options depend on the repository type
+* options depend on the storage type
 
 Example urls
 
-       repository.mlhartme=github:https://api.github.com %~mlhartme
-       repository.company=gitlab:https://gitlab.company.com 
+       storage.mlhartme=github:https://api.github.com %~mlhartme
+       storage.company=gitlab:https://gitlab.company.com 
 
 You can configure access token for github and gitlab by storing them in a file $POMMES_ROOT/.pommes/<reponame>-<protocol>.token.
-For example, a token in file $POMMES_ROOT/.pommes/github-mlhartme.token will be used to index the above mlhartme repository.
+For example, a token in file $POMMES_ROOT/.pommes/github-mlhartme.token will be used to index the above mlhartme storage.
 
 ## Find Command
 
@@ -151,7 +151,7 @@ Descriptor: references a project, various implementation to load Maven, Json, et
 
 Project: holds project metadata: gav, parent, dependencies, scm, url
 
-Repository: can be scanned for descriptors
+Storage: can be scanned for descriptors
 
 Database: stores projects (in Lucene)
 
@@ -181,14 +181,14 @@ commands
                         '?' - checkout is not in database
                         '!' - checkout in wrong directory
                         '#' - error checking this checkout
-  'index' {repo}        re-index the specified (default: all) repositories.
+  'index' {storage}     re-index the specified (default: all) storages.
   'setup' ['-batch'] {name'='value}
-                        creates '.pommes' directory with initial configuration containing name/values as repositories; 
-                        indexes all repositories to create intial database;
+                        creates '.pommes' directory with initial configuration containing name/values as storages; 
+                        indexes all storages to create initial database;
                         '.pommes' is created in the directory specified by $POMMES_ROOT, default is ~/Pommes.
 
 fields in the database: (field id is the first letter of the field name.)
-  origin      Where this project was loaded from. Used as unique identifier. <repositoryName>:<path>
+  origin      Where this project was loaded from. Used as unique identifier. <storageName>:<path>
   revision    Last modified timestamp or content hash of this pom. Used to detect changes.
   parent      Coordinates of the parent project.
   artifact    Coordinates of this project.

src/main/java/net/oneandone/pommes/cli/Base.java

@@ -40,7 +40,7 @@ public Base(Environment environment) {
 
     public void run() throws Exception {
         try (Database database = environment.lib.loadDatabase()) {
-            run(new Scope(environment.lib.properties().defaultRepository,
+            run(new Scope(environment.lib.properties().defaultStorage,
                     database, environment, new CentralSearch(environment.world(), environment.maven())));
         }
     }

src/main/java/net/oneandone/pommes/cli/Environment.java

@@ -47,7 +47,7 @@ public Environment(Console console, World world) throws IOException {
         this.console = console;
         this.world = world;
         this.lib = Lib.load(world);
-        console.verbose.println("default repository: " + lib.properties().defaultRepository);
+        console.verbose.println("default storage: " + lib.properties().defaultStorage);
         this.lazyMaven = null;
         this.lazyCurrentPom = null;
         this.lazyExcludes = null;
@@ -124,7 +124,7 @@ private Project scanPom(FileNode directory) throws IOException {
                 Descriptor.Creator m = Descriptor.match(child.getName());
                 if (m != null) {
                     descriptor = m.create(this, child, "unused", child.getPath(), child.sha(), scm.getUrl(directory));
-                    return descriptor.load(this);
+                    return descriptor.load();
                 }
             }
         }

src/main/java/net/oneandone/pommes/cli/Index.java

@@ -22,14 +22,13 @@
 import net.oneandone.pommes.database.Project;
 import net.oneandone.pommes.descriptor.Descriptor;
 import net.oneandone.pommes.descriptor.ErrorDescriptor;
-import net.oneandone.pommes.repository.Repository;
+import net.oneandone.pommes.storage.Storage;
 import net.oneandone.pommes.scm.Git;
 import net.oneandone.pommes.scm.Scm;
 import net.oneandone.sushi.util.Separator;
 import org.apache.lucene.document.Document;
 
 import java.io.IOException;
-import java.io.PrintWriter;
 import java.util.HashMap;
 import java.util.Iterator;
 import java.util.List;
@@ -40,76 +39,70 @@
 import java.util.concurrent.BlockingQueue;
 
 public class Index extends Base {
-    private final List<String> repositories;
+    private final List<String> storages;
 
-    public Index(Environment environment, List<String> repositories) {
+    public Index(Environment environment, List<String> storages) {
         super(environment);
 
-        Set<String> available = environment.lib.properties().repositories.keySet();
-        this.repositories = repositories;
-        for (String repository : repositories) {
-            if (!available.contains(repository)) {
-                throw new ArgumentException("repository not found: " + repository);
+        Set<String> available = environment.lib.properties().storages.keySet();
+        this.storages = storages;
+        for (String storage : storages) {
+            if (!available.contains(storage)) {
+                throw new ArgumentException("storage not found: " + storage);
             }
         }
     }
 
     @Override
     public void run(Scope scope) throws Exception {
-        Repository repository;
+        Storage storage;
         Indexer indexer;
-        PrintWriter log;
 
-        log = new PrintWriter(environment.lib.logs().join("pommes.log").newWriter(), true);
-        for (Map.Entry<String, String> entry : environment.lib.properties().repositories.entrySet()) {
-            if (!repositories.isEmpty() && !repositories.contains(entry.getKey())) {
+        for (Map.Entry<String, String> entry : environment.lib.properties().storages.entrySet()) {
+            if (!storages.isEmpty() && !storages.contains(entry.getKey())) {
                 // not selected
                 continue;
             }
-            repository = null;
+            storage = null;
             for (String str : Separator.SPACE.split(entry.getValue())) {
                 if (str.equals("§§")) {
-                    String host = repo(repository, str).getTokenHost();
+                    String host = storage(storage, str).getTokenHost();
                     if (host != null) {
                         Git.UP up = Scm.GIT.getCredentials(environment.console(), environment.world().getWorking(), host);
-                        repo(repository, str).setToken(up.password());
+                        storage(storage, str).setToken(up.password());
                     }
                 } else if (str.startsWith("-")) {
-                    repo(repository, str).addExclude(str.substring(1));
+                    storage(storage, str).addExclude(str.substring(1));
                 } else if (str.startsWith("%")) {
-                    repo(repository, str).addOption(str.substring(1));
+                    storage(storage, str).addOption(str.substring(1));
                 } else {
-                    if (repository != null) {
-                        throw new ArgumentException("duplicate repository");
+                    if (storage != null) {
+                        throw new ArgumentException("duplicate storage: " + storage);
                     }
-                    repository = Repository.create(environment, entry.getKey(), str, log);
+                    storage = Storage.createStorage(environment, entry.getKey(), str);
                 }
             }
 
+            console.info.println("indexing " + entry.getKey() + ": " + entry.getValue());
+            indexer = new Indexer(environment, scope.getDatabase(), entry.getKey());
+            indexer.start();
             try {
-                console.info.println("indexing " + entry.getKey() + ": " + entry.getValue());
-                indexer = new Indexer(environment, scope.getDatabase(), entry.getKey());
-                indexer.start();
-                try {
-                    repository.scan(indexer.src, environment.console());
-                } finally {
-                    indexer.src.put(ErrorDescriptor.END_OF_QUEUE);
-                    indexer.join();
-                }
-                if (indexer.exception != null) {
-                    throw indexer.exception;
-                }
+                storage.scan(indexer.src, environment.console());
             } finally {
-                log.close();
+                indexer.src.put(ErrorDescriptor.END_OF_QUEUE);
+                indexer.join();
+            }
+            if (indexer.exception != null) {
+                throw indexer.exception;
             }
         }
     }
 
-    private Repository repo(Repository repository, String str) {
-        if (repository == null) {
+    private Storage storage(Storage storage, String str) {
+        if (storage == null) {
             throw new ArgumentException("missing url before '" + str + "'");
         }
-        return repository;
+        return storage;
     }
 
     /** Iterates modified or new documents, skips unmodified ones */
@@ -118,7 +111,7 @@ public static class Indexer extends Thread implements Iterator<Document> {
 
         public final BlockingQueue<Descriptor> src;
         private final Database database;
-        private final String repository;
+        private final String storage;
 
         private Exception exception;
 
@@ -128,15 +121,15 @@ public static class Indexer extends Thread implements Iterator<Document> {
 
         private final Map<String, String> existing;
 
-        public Indexer(Environment environment, Database database, String repository) {
+        public Indexer(Environment environment, Database database, String storage) {
             super("Indexer");
 
             this.environment = environment;
 
             this.src = new ArrayBlockingQueue<>(25);
 
             this.database = database;
-            this.repository = repository;
+            this.storage = storage;
             this.exception = null;
 
             // CAUTION: current is not defined until this thread is started (because it would block this constructor)!
@@ -151,7 +144,7 @@ public void run() {
 
             try {
                 started = System.currentTimeMillis();
-                database.list(repository, existing);
+                database.list(storage, existing);
                 environment.console().verbose.println("scanned " + existing.size() + " existing projects: "
                         + (System.currentTimeMillis() - started) + " ms");
                 current = iter();
@@ -209,13 +202,13 @@ private Document iter() {
                     return null;
                 }
                 count++;
-                existingRevision = existing.remove(descriptor.getRepository() + Field.ORIGIN_DELIMITER + descriptor.getPath());
+                existingRevision = existing.remove(descriptor.getStorage() + Field.ORIGIN_DELIMITER + descriptor.getPath());
                 if (descriptor.getRevision().equals(existingRevision)) {
                     console.info.println("  " + descriptor.getPath());
                     continue;
                 }
                 try {
-                    project = descriptor.load(environment);
+                    project = descriptor.load();
                 } catch (IOException e) {
                     console.error.println(e.getMessage());
                     e.printStackTrace(console.verbose);

src/main/java/net/oneandone/pommes/cli/Lib.java

@@ -39,7 +39,7 @@ public static Lib load(World world) throws IOException {
         return new Lib(lib);
     }
 
-    public static void create(World world, FileNode root, Console console, Map<String, String> repositories) throws IOException {
+    public static void create(World world, FileNode root, Console console, Map<String, String> storages) throws IOException {
         FileNode lib;
 
         lib = root.join(DIR);
@@ -50,7 +50,7 @@ public static void create(World world, FileNode root, Console console, Map<Strin
         lib.mkdirs();
         lib.join("logs").mkdir();
         world.resource("profiles").copyDirectory(lib.join("profiles").mkdir());
-        configFile(lib).writeLines(Properties.defaultConfig(repositories));
+        configFile(lib).writeLines(Properties.defaultConfig(storages));
     }
 
     /** @return .pommes directory to use or null if not defined */
@@ -72,11 +72,6 @@ public Lib(FileNode home) throws IOException {
         this.properties = Properties.load(configFile(home));
     }
 
-    public String tokenOpt(String repositoryName) {
-        // TODO merge with git credential access
-        return System.getenv("POMMES_REPOSITORY_TOKEN_" + repositoryName.toUpperCase());
-    }
-
     public FileNode projectDirectory(Project project) throws IOException {
         if (project.scm == null) {
             throw new IOException(project + ": missing scm");

src/main/java/net/oneandone/pommes/cli/Main.java

@@ -58,19 +58,19 @@ public static void main(String[] args) {
                 + "  'remove' root?        remove checkouts under the specified root directory;\n"
                 + "                        offers selection before changing anything on disk;\n"
                 + "                        checkouts with modifications are marked in the list\n" */
-                + "  'index' {repo}        re-index the specified (default: all) repositories.\n"
+                + "  'index' {storage}     re-index the specified storages (default: all).\n"
                 + "  'setup' ['-batch'] root? {name'='value}\n"
                 + "                        sets up pommes in the specified directory (default: ~/" + Lib.DEFAULT_ROOT + ")\n"
-                + "                        creates '" + Lib.DIR + "' directory inside with initial configuration containing name/values as repositories; \n"
-                + "                        indexes all repositories to create intial database;\n"
+                + "                        creates '" + Lib.DIR + "' directory inside with initial configuration containing name/values as storages; \n"
+                + "                        indexes all storages to create initial database;\n"
                 + "\n"
                 + "fields in the database: (field id is the first letter of the field name.)\n"
                 + fieldList()
                 + "\n"
                 + "query syntax\n"
                 + "  query     = '@' MACRO | or\n"
-                + "  or        = repo? (and (' ' and)*)? index?\n"
-                + "  repo      = '/' STR?                  ; default is first repository defined in config\n"
+                + "  or        = storage? (and (' ' and)*)? index?\n"
+                + "  storage   = '/' STR?                  ; default is first storage defined in config\n"
                 + "  index     = NUMBER\n"
                 + "  and       = term ('+' term)*\n"
                 + "  term      = field | lucene\n"
@@ -96,7 +96,7 @@ public static void main(String[] args) {
             cli.add(Remove.class, "remove root?=.");
             cli.add(Status.class, "st root?=.");
             cli.add(Goto.class, "goto -x=false query*");
-            cli.add(Index.class, "index repo*");
+            cli.add(Index.class, "index storage*");
             cli.add(Find.class, "find -output=null -fold queryOrFormat* { arg*(queryOrFormat)}");
 
         System.exit(cli.run(args));