irmin as libgit2 substitute

[kluvin]

Hi all! I am working on a markdown reimplementation of WikiWikiWeb, where I am now extending this to similar problems as the federated wiki -- which is essentially taking git semantics and applying them to a wiki. The idea was originally to use git directly, although, since it lives only in the filesystem, that brings some problems; so I looked at several technologies that are labelled as git-for-x, irmin stood out in that it could directly operate on .git repositories.

I am curious about how the storage-agnostic nature of irmin, together with the git-backend compares to libgit2, which is used in GitHub and others to programmatically manage bare repos. In fact libgit2 operates on disk as much as git itself does, and so I figure there is significant performance gains to be achieved by operating in memory, and only flushing to disk sometimes. Can irmin do something like that? Please let me know if anything is not clear.

PS: I am getting something like this:

$ irmin set x 1
$ irmin get x
> irmin: [ERROR] The client wants to refill the internal buffer by a bigger input.
  ERROR: Chunk loaded from ./.git/objects/be/eac1f72988177236341222008601144f5950da is too big (65536 bytes) for the decoder (max: 65536 bytes)
$ git gc
> ...done
$ irmin get x
> 1

When getting an object after writing a commit with irmin set, a simple git gc fixes this until the next write. It seems similar to #1001. Shall I publish an issue with more details?

PPS: The welcome message still says to comment on it, but it's now locked.

[samoht]

Hello 👋 The new release of irmin (2.3.0) should fix the issue you got with Git. Can you check it works for you?

Also:

Can irmin do something like that? Please let me know if anything is not clear.

Yes, that's possible to use Irmin with an in-memory backend (that you can then push to remote server). Can you describe your use-case to see if the current hooks exposed by the CLI are enough to do what you need?

[kluvin]

Trying to do anything with the instance I was working with earlier now yields the following:

irmin: [ERROR] ./.git/objects/pack/./.git/objects/pack/pack-<hash>.idx does not exists.
ERROR: (Invalid_argument List.combine)

Where the /.git/objects/pack/pack-<hash>.idx can be verified to exist. Notice that irmin isn't actually looking in the correct directory, which I can't figure out why. I think it may be due to the size, again.

Can you describe your use-case

I am afraid I'm not interpreting the question correctly, but I will try to give you something. What I want and need is the following:

• Revisioning, and a method of looking at past revisions
• Branching, by the notion popularized by git
• Merging, a method of recombining branches
• Logging of the history of branches and merges, so that it is essentially possible to get the network of branches and their interaction
• Deduplication with respect to a branch
• Deduplication with respect to the pool of branches. GitLab and I suspect also GitHub only does this one level deep.
• Metadata, some way of attaching information to an object

The notion of forks and branches seems to be encapsulated by the namespacing irmin provides, i.e., irmin get namespace/value, although I see that there exists a --branch option. It seems irmin can do most of these, though I am a bit unsure logging of history w.r.t. branching and merging, as well as deduplication of pools.

The reason I ask if irmin substitutes libgit2 is because that essentially the feature set I desire, as I am trying to impose the ideas implemented by GitHub on top of something that looks like a wiki. And as much as libgit2 implements a programmatic interface for git, irmin does both simultaneously; so I am currently picking up OCaml so that I am not restricted by the CLI nor bothered by having to deal with subprocesses all the time--and also for the joy of it, very fun so far!

Between TerminusDB (which I also seriously evaluated), libgit2, and irmin; they probably all fit the bill, it's just that irmin provides the best compromise between portability and ease-of-use, and I can pretty much see immediately how I'm going to do this.

[dinosaure]

irmin: [ERROR] ./.git/objects/pack/./.git/objects/pack/pack-<hash>.idx does not exists.
ERROR: (Invalid_argument List.combine)

You probably should make an issue about that. It's not really about the size but the path given to irmin and given then toocaml-git. Something is wrong and it's hard to know which library prepend a second time the path. An other issue should exists in ocaml-git to catch the List.combine exception.

The reason I ask if irmin substitutes libgit2 is because that essentially the feature set I desire, as I am trying to impose the ideas implemented by GitHub on top of something that looks like a wiki. And as much as libgit2 implements a programmatic interface for git, irmin does both simultaneously; so I am currently picking up OCaml so that I am not restricted by the CLI nor bothered by having to deal with subprocesses all the time--and also for the joy of it, very fun so far!

Irmin + ocaml-git reimplements Git as you expect - we can manipulates Git objects as libgit2/git does. By this way, Irmin + ocaml-git is compatible with an usual use of Git (regardless higher operations such as git status/git rerere/etc.). Any commits/others objects created by libgit2/git will be available with Irmin + ocaml-git.

Irmin + ocaml-git is able to import a Git repository in memory too. That mostly means that you can manipulate a Git repository without any access to a file-system. Such case appears mostly for MirageOS which does not have a file-system. Then, we provides synchronization mechanisms such as git push/git pull.

As @samoht said, it depends on your case:

• you can use Irmin + ocaml-git in memory and use synchronization mechanism then (the other part is: if your program shutdowns (for any reasons) and you did not git push, you will lost everythings)
• you can use Irmin + ocaml-git with a file-system, by this way any creation of a Irmin object will be available with libgit2/git manually. Of course, synchronization mechanisms are available too

I said Irmin + ocaml-git because Irmin is much more an abstract layer of a KV-store with some backends (such as git/libgit2/ocaml-git). Finally, you have a simple interface/signature and you can build your application from it and delay the choice of the backend:

• ocaml-git
• graphql
• irmin-pack

They are compatible each others as long as you use Irmin - and switch from one to another should be free.

Between TerminusDB (which I also seriously evaluated), libgit2, and irmin; they probably all fit the bill, it's just that irmin provides the best compromise between portability and ease-of-use, and I can pretty much see immediately how I'm going to do this.

I don't know about TerminusDB but I can say that: if you use irmin, you will be compatible with git/libgit2 as long as you use Irmin + ocaml-git - then, from this specific backend, Irmin opens the door for several others backends (like, as I said, graphql or irmin-pack).