Please add semi.el so we can continue to distribute this on Melpa

[tarsius]

/cc @riscy

Melpa requires that packages contain a library that matches the name of the package. I.e., in the case of the semi package, the library semi.el MUST exist.

I suggested that you do that all the way back in 2017, in #16, which I closed after waiting for a response for a few months. Please see that pull-request for the minimal-viable approach to satisfying Melpa's requirement.

I am currently improving package-build.el, the tool used to build the packages distributed on Melpa. In the process I am tightening enforcement of some requirements, such as the one at hand.

semi is the only package on Melpa which violates this requirement. I do not intend to keep a special-case in package-build.el just for this one package.

So please add the semi.el file.

[ikazuhiro]

Melpa requires that packages contain a library that matches the name of the package. I.e., in the case of the semi package, the library semi.el MUST exist.

I've not been able to find about it in the document. Where is it described? And, if it is MELPA's own requirement, could you tell me why MELPA requires PACKAGENAME.el?

semi is the only package on Melpa which violates this requirement. I do not intend to keep a special-case in package-build.el just for this one package.

AFAIK, neither apel, flim nor wanderlsut have PACKAGENAME.el file. Is semi really the only one package?

So please add the semi.el file.

I don't like to keep such meaningless file for only one unofficial package system. I hope you resolve the issue within your package system. If you won't accept semi without semi.el, for example, how about changing package name to existing file name?

[riscy]

Where is it described?

I believe it is here - https://github.com/melpa/melpa/blob/master/CONTRIBUTING.org#create-a-recipe-file - re: "The filename should match the name of the package’s provided feature." which I realize is a technically a bit different, but if a package is aligned with all of our (present-day) checks and balances then this is equivalent. I can see that semi was merged back in 2014.

how about changing package name to existing file name?

On a quick glance, renaming the package name to mime looks like it would fix several things, and the name is available. And we could use the :old-name specifier in the recipe to note that the package used to be "semi".

[tarsius]

@ikazuhiro I'll reply later today.

I believe it is here - https://github.com/melpa/melpa/blob/master/CONTRIBUTING.org#create-a-recipe-file

@riscy They asked for "official" rules, but these are the rules of the "unofficial package system archive" Melpa. I'll later argue why these conventions are never-the-less relevant and nearly universally followed, and should therefore be followed here too.

[tarsius]

semi is the only package on Melpa which violates this
requirement. I do not intend to keep a special-case in
package-build.el just for this one package.

AFAIK, neither apel,
flim nor
wanderlsut have
PACKAGENAME.el file. Is semi really the only one package?

You are right, there exist more packages that lack a library whose name matches the name of the package. Shortly after opening this issue, I realized that the claim that this is the only such package cannot be true, not least because I remembered specifically that multiple packages from the wanderlust project don't do so.

So I ran a reliable test on all packages currently available on Melpa, and this revealed that there are seven packages which don't provide the matching library. Four under your control and three others. This is out of a total of 5836 packages on Melpa. I have high confidence in the accuracy of this test.

I then proceeded to open issues for the three other packages. The maintainers of two of those responded very quickly and positively, so I concentrated on those and forgot to come back here and correct my earlier statement. The maintainer of the last package hasn't responded yet.

So now only 0.085% of package on Melpa do not provide the matching library. IMO it is thus safe to say that the practice of providing the matching library is nearly universally followed.

Melpa requires that packages contain a library that matches the name
of the package. I.e., in the case of the semi package, the library
semi.el MUST exist.

I've not been able to find about it in the document. Where is it
described?

I also was unable to find anything about this in Preparing Lisp code for distribution in the elisp manual. However, from that does not automatically follow that the authors of that document do not agree that every package should/must provide such a file. It is IMO more likely that it just did not occur to them that this is something that would have to be mentioned, since they had never seen a package that did not do so.

Also note that the package-x.el library, which is mentioned in that manual, as the tool to use to build package archives, is not sufficient to do that in practice, for anything but at minimal scale. GNU ELPA and NonGNU ELPA (the "official" package archives) use elpa-admin.el instead (which I have studied in detail and contributed to), and MELPA uses package-build.el (which I maintain).

The elisp manual mentions PACKAGE-pkg.el, but isn't clear about whether package authors should include such a file in the upstream repository of their package, or whether that is only intended to be generated by the tools used to build the package tarballs, which are ultimately distributed on in the package archive.

It is possible that originally the intention was for that file to be included in upstream repository and serve as a source of metadata from the perspective of the package archive maintenance tool, not just from the perspective of package.el, the tool used by the end-user .

But if that is the case, then that is a practice that, over a decade ago, stopped being the intended way of providing metadata.

elpa-admin.el (as mentioned the tool used to build the packages on the "official" [Non]GNU ELPA) never used this file as an input. It gets metadata exclusively from the "mainlib". It then stores some of that metadata in the PACKAGE-pkg.el file, if that existed, completely overriding whatever may have been there before.

MELPA and its package-build.el are much older than the "official" ELPA and their tooling. Because of that it still supports using PACKAGE-pkg.el as an input.

Only few package authors actively take advantage of this and provide metadata in that file that in the absent from PACKAGE.el. More authors duplicate the information in both files. And sadly what is most common is that authors added PACKAGE-pkg.el a long long time ago and now have mostly forgotten about it, i.e., the file contains obsolete information, which is only very infrequently brought up-to-date, if ever.

For these two reason ((1) elpa-admin.el does use it either, and (2) the provided metadata is more often than not bad), I am in the process of making package-build.el and thus MELPA ignore the PACKAGE-pkg.el as an input.

Contacting you, and the three other package maintainers who still don't include PACKAGE.el, is part of that process.

And, if it is MELPA's own requirement, could you tell me
why MELPA requires PACKAGENAME.el?

Above I have tried to make the point that it is actually MELPA which is the only ELPA that still is able to build a package if PACKAGE.el is missing. This is going to change soon, at which point all ELPA will require the PACKAGE.el file.

In other words, this is very much not "MELPA's own requirement".

So please add the semi.el file.

I don't like to keep such meaningless file for only one unofficial
package system.

1. MELPA is not a "package system". The term commonly used for MELPA is "package archive" or "an ELPA" (an abbreviation for "Emacs Lisp Package Archive"). MELPA is one of the well known ELPA. While it is not one of the "official" archives, it is by far the oldest and largest, and still very much relevant. One could say that all the ("official" and "unofficial") ELPA together with package.el form a "package system".

   Since wanderlust and your other three packages are only on MELPA, it is via MELPA that these packages are at all installable using the "official package system" (though via an "unofficial" package archive).

   You could of course also/instead distribute your packages via one of the official archives, but then you would also have to satisfy the PACKAGE.el requirement. In fact you would have to do so at the time when you add the package to the archive; with MELPA you still have a few weeks to add these files, though I would appreciate it if you could do so soon.

2. Even if we ignored the needs of the official package system, this file would not be meaningless. Users know these packages by their public name and, because nearly every package does that, there is the assumption that packages come with a library whose name matches.

   Users may then try to configure the package known as wanderlust using

   (use-package wanderlust
     ...)
   

   or

   (with-eval-after-load 'wanderlust
     ...)
   

   which would fail, because no such library exists.

I hope you resolve the issue within your package
system. If you won't accept semi without semi.el, for example,
how about changing package name to existing file name?

Changing the name of packages is problematic.

Potential future users may for example have heard about wanderlust before and would have difficulties finding it if it were suddenly named wl instead, in the package list provided by package.el.

Renaming a package that isn't directly used by end-users, but is depended upon by other packages, is also problematic. If, for example, we renamed semi to something else, then several other packages which used to specify the dependency on that by mentioning semi in Package-Requires, would have to be adjusted to refer to it under the new name.

This would only work for the "snapshot" version of these packages. Releases (i.e., what is distributed on MELPA Stable) would be broken until both the package-formerly-known-as-semi and the dependant package had new releases.

Thanks for giving my proposal a second change!
Jonas

[ikazuhiro]

Thank you for the information and sorry for the late response.
I think I've understood PACKAGENAME-pkg.el in source repository has been not good.

It is IMO more likely that it just did not occur to them that this is something that would have to be mentioned, since they had never seen a package that did not do so.

IMO, they may simply think PACKAGENAME.el is not required.

This is going to change soon, at which point all ELPA will require the PACKAGE.el file.

In other words, this is very much not "MELPA's own requirement".

As far as I read elpa-admin.el, it seems to take care of the case PACKAGENAME.el is missing. elpa-packages file has :main-file property to specify the file that provides metadata. PACKAGENAME.el is not mandatory.

So, I think PACKAGENAME.el is MELPA's own requirement and I hope MELPA's recipe provides :main-file support.

If you won't accept semi without semi.el, for example, how about changing package name to existing file name?
Changing the name of packages is problematic.

Of course, I know it has some problems. But I would choice renaming package rather than adding PACKAGENAME.el into the repository.

Potential future users may for example have heard about wanderlust before and would have difficulties finding it if it were suddenly named wl instead, in the package list provided by package.el.

I think there is little problem if we add real name into description.

(with-eval-after-load 'wanderlust
  ...)

which would fail, because no such library exists.

When is wanderlust.el is loaded? I think it will never be loaded in starting wanderlust (normally by M-x wl).

This would only work for the "snapshot" version of these packages. Releases (i.e., what is distributed on MELPA Stable) would be broken until both the package-formerly-known-as-semi and the dependant package had new releases.

Wanderlust and related or dependant packages have only snapshot versions al least in MELPA.

As previous post, I consider renaming packages if MELPA requires PACAKGENAME.el.

[tarsius]

The situation now seems to be:

• I want you to add <name.el because this has been a policy of Melpa since the beginning (it just wasn't strictly enforced) and because, going forward, strictly enforcing it is becoming necessary. It is a prerequisite for a major improvements to Melpa; better version strings. But you absolutely don't want to do it because it is not "necessary".

  [Actually, I don't quite understand why you insist on not adding them. The files are not necessary in your opinion, I understood that, but would it be so horrible to add them anyway, even just to comply with Melpa's requirements and to make its maintainers happy. As a gesture of appreciation among foss maintainers, for our years of work, distributing everybody's packages?.]

• You suggest instead renaming the packages so that the new package names matches an existing library. But I think that is a really really bad idea, because it would negatively affect everybody who has already installed these packages under their current names.

That's fine. We don't have to agree on this. Sometimes well meaning people cannot agree on something, and if that happens parting ways may be the best we can do.

I see a few ways we can deal with this:

1. In the short run we can just leave things as they are. I can delay strictly enforcing this rule until I introduce the new version scheme, which will make doing so necessary. At that time I would remove these packages from Melpa, and you can delay until then deciding how to deal with it.

2. You could rename these packages as you suggested. I want to make it very clear that if you do that, I accept zero responsibility for the consequences. If you do this then this is going to cause issues for many users, and I am warning you about it now. If you go through with it, you will have to own the decision.

3. You distribute these packages on NonGNU Elpa. As you said, they support :main-file, so they might accept these packages, without asking for <name>.el to be added.

4. Melpa could patch your packages to add the <name>.el on our side. In fact the Emacsmirror (which I also maintain) has been adding these files for years. So all we would have to do is change the Melpa recipes to get these packages from, e.g., https://github.com/emacsmirror/wanderlust instead of from the real upstream repository. The repositories on the mirror are updated about once a weak and the additional "patch" commits on the mirror are automatically rebased in the process.

I want to make it clear that I understand that you are under absolutely no obligation to accommodate us. Melpa is just an unofficial distribution channel, and if you do not want to do something, that we ask you to do, that is perfectly within your rights.

But the same holds true in the other direction. Melpa is not obligated to distribute your packages. We have been distributing your package for years, despite them not satisfying our requirements, but now that continuing to do so is becoming more of a problem, we want to stop doing that.

We ask you to adjust your packages to satisfy our requirements, a request you are free to decline. Likewise we are under no obligation to continue to distribute your package. I do not mean this as a threat ("do as we say, or we kick you out"), instead what I am trying to say is that we have a fundamental disagreement and that it might be best to just part ways.

[egh]

I wrote the MELPA packages for WL. Can we just do this? MELPA is an amazing resource. 35,000 installs of Wanderlust have happened via MELPA. Before that we frequently had questions on how to install Wanderlust. I'm happy to write the pull requests to add these simple files.