Rules vs languages vs global folders

[tg44]

👋 Welcome!

Currently, we have rules "by name". So we have 4 python, 2 java etc. rules right now. It would be good to group these rules into one file or to something similar concept.

There are some ideas about the global directory handling like ".yarn" or ".maven" where we either want to globally ignore or globally exclude them (or maybe left it as is to still walk by rules). I think this is a must-have requirement, but adding them as a new rule-type or something new concept, will add a lot of unnecessarily complicated things to the already lot of files.

I think it would be worth talking about this, we have some users already, and I want to see others' opinions here.

My goto idea right now is to make bigger files, and merge multiple file-trigger type rules with multiple (new) "global" types as a big list. Add helper CLI functions to switch the globals by name (sth like heptapod rules globalExclude java/maven). BUT do you really want to enable/disable rules, or how would you like to use a tool like this?

[tg44]

@jgoldhammer @GBrachetta what do you too think? (also related to #7 and partly to #3)

[GBrachetta]

Thank you for sharing your ideas!

In my opinion there could be basically two types:

• The folders that are created once, such as .yarn or .nvm or Library or .Trash, which the user may choose to always ignore or not (I would!)
• The folders that are constantly created and deleted, such as .venv or node_modules. These are new every time there's a new project and disappear eventually every time a project is deleted.

I can imagine that for the first type there could be a kind of more general (over the rules) type config, maybe a list of folders to always be ignored because in most cases they will be there nonetheless (they are not changing).

For the second type I can imagine the current rules system would work, but I also imagine those rules will need to be ran periodically (automatically?) so the user doesn't need to manually do it. It's an easy thing to forget!

I also advocate for some heptapod clean command which would check what folders previously added to the exclude have been deleted, so they can be as well deleted from the list of exclusions.

The only other little thing I see that needs some addressing is a way to see more clearly what are currently the folders excluded, maybe on a table or with some basic formatting because as it keeps growing, the command heptapod tm ls can be very tricky to read.

Related to this would be nice to have a better verbose mode, because currently (with all the "general" excludes added to each rule) it is really long and difficult to read.

P.S. It would be nice to add a little recap of what heptapod did after heptapod run. Currently there's no message saying everything went okay.

Thanks for the great work!!

[jgoldhammer]

Good discussion @tg44 and @GBrachetta !

My thoughts:

• I generally like the rule system, but have to admit than I would not change rules very often. I guess their might be some users who do not want to exclude node_modules directory and commit to git, so it is good for users to enable or disable it. Also it is good to disable rules for projects you are not working with.
• I would favor a simple list of preconfigured directories like the m2 repository or yarn cache which will be excluded by time machine and can be adapted by the user. Maybe some users want to backup the m2 repository and some do not want that, so this list could contain a list of useful directories and should be adaptable by the user.
• I would also favor a simple list of ignored directories heptapod would not search in. In many cases users store code in a specific folder in the users directory, so in 99% users will ignore the users Library folder for all rules... So you could simplify ignoring folders from specific rules to a global ignored directory list.
• I also vote for the automatic run of heptapod once an hour configurable by the user.
• I also vote for a small statement after running heptapod run

Thx!

[tg44]

Nice! Thanks for the feedback!

• Small statement added to heptapod run.
• Verbose is now leveled so you can run with v 1 for minimal info or v 4 for debugging.
• Currently excluded folders are tricky, our listing method is only "good enough" I freeze this for now (check the readme).
• heptapod clean not needed, flags deleted with the directories, it is working out of the box already
• For automation, I started my research, but any idea is interesting at this point, I'm not committed to anything

My main questions right now;

• should I group the current rule system with the "global" exclude list?
  • I'm thinking of sth like "js" which would contain node, npm, rules from the existing list, and also the global exclude making possible to switch on/off whole groups (languages?), and sub leaves like the global .yarn exclude
  • or should I make a bigger one file list with all the global excludes in it?

I don't think that writing new rules will be a frequent thing, but I want to make the rules as clean, compact, and easy to manage as possible. (I think the biggest challenge with the other tools is that it is hard to customize them, or add new rules.) I think the biggest commitment with this tool is the initial setup which should be as painless as possible. Maybe even a graphical interface would be good.

[jgoldhammer]

I am voting for global directory exclude lists. I think some things fit to the rules, but some are not, especially if users want to exclude custom directories not to backup. Users may ask where to put the excludes when they have to put it into a specific rule, but no rule really fits to the excluded directory.

Besides that the global directory exclude list can be handled differently because you have to regularly scan the file for new directories and add the directory via tmutil.

My 2 cents regarding a graphhical interface: A graphical interface would not be necessary in my eyes. Most of developers who need your tool can handle the cli (I hope)...

[tg44]

I would simply solve this with a "global" (or "os specific") ruleset, the trash and library folders would go there by default.

[GBrachetta]

What kind of graphical interface do you envision? It could be a very nice addition to the cli and could make things visually clearer. Exciting!

[jgoldhammer]

What do you think @tg44 ?

[tg44]

I just sit down to my laptop to work on this a bit :D no idea yet.

About the GUI, no strict idea either. I want to fine-tune the rules and periodic running before I make it more user friendly.

[GBrachetta]

I think it might be better to have a global rule separated. So one global for those folders that are unique (Library, .Trash, .yarn, etc) and then the rules that apply to folders that are created all the time, such as node_modules, etc.

Unless I'm seeing it wrong...

[tg44]

What do you all think about this format? https://github.com/tg44/heptapod/blob/main/rules/osx.yaml

I made two things today;

• added a global rule
• added a list rule

Now we can group the rules! I haven't released this version yet. If you think this is generally a good approach;

• I need to reorganize the current rules (PRs welcomed ofc)
• I need to rewrite the enable/disable mechanism to work with subs too (cli commands)
• We need to decide if a lists searchPaths and ignorePaths is set, do we want that to be an override to all of the subs, or do we want to merge these arrays? (so a list have a searchPaths set to [/dev] and a subRule has [] we want to run the sub on ~ and ~/dev or just ~/dev?)
• I probably need to add a new table to list the globals only

[jgoldhammer]

Hi, in general this is a good idea to organize the paths in subrules. On a first look it looks a little bit complicated, but it is very flexible this way. The name of the rule is currently node-multi - maybe you can rename it?

My simplified way would just to have a single rule file with a list of paths.

name: "global ignore/excludes"
ignores: 
 - "~/Library"

excludes:
- "~/Downloads"

Thanks for all your work
Jens

[tg44]

My problem with this approach is the same as why I made it way more complicated (and much more flexible) than asimov.

If you check asimovs list it is both clean and simple, and totally unmanageable at the same time.

I think this is the point when the UX and the flexibility clash, and I should provide a clear UI for it rather than let the users manually modify the rules, and make the rules itself more user friendly.

Will work on this as one of the next steps. (after the autorun)

[GBrachetta]

I'm sorry for taking time to take a look, I'm away for a couple days. Will clone asap and try to understand, although golang is unknown to me! 😊

Thank you so much for all your work!

[GBrachetta]

I wonder if the file trigger could also be an array, so in this case:

name: "python_env-multi"
enabled: true
searchPaths: []
ignorePaths: []
ruleType: "list"
ruleSettings:
  subRules:
    - name: "python_env"
      enabled: true
      searchPaths: [ "~/dev" ]
      ignorePaths: [ "~/dev/alpakka", "~/dev/akka" ]
      ruleType: "file-trigger"
      ruleSettings:
        fileTrigger: [ "requirements.txt", "Pipfile" ]
        excludePaths:
          - ".venv"

we don't need two different rules for .venv folders but one with an array of triggers.

[tg44]

This feels like a double-edged sword, but ofc. we can do it without a big hassle.

Is that a valid use-case that I as a python programmer want to exclude the .venvs where I have Pipfile in the project, but I don't want to when there is a requirements.txt? If it is not, we can merge some of the .venv excludes to a single rule, by adding multiple file triggers.

[GBrachetta]

Not really a valid case. We would never care about .venv folders, as we never commit them either. Same as with node_modules they can be created with pip install or pipenv install.