Doc: parent-child convention for installed packages

[Julow]

This convention is already in use by Odig, Voodoo and Dune. It needs to be specified as it needs to change in order to support assets and complex hierarchies.

[Julow]

@dbuenzli We wrote this in response to #1002. Our plan is to document the state before the asset PR and then to modify the documentation in the asset PR.

Does the convention match the one currently in use ?

[dbuenzli]

I think it does except for the index.mld renaming bit.

[dbuenzli]

The renaming bit doesn't work, it prevents from making links from other pages into the index page (example).

[Julow]

The index of package page is not generated using Odoc ? It can't make references to {!page-index}, as that would be very ambiguous.

Though I believe we need references through the page hierarchy like {!page-packages.odoc.index} but even then, the index part isn't wanted.

[Julow]

Also, we'll want to allow deeper hierarchies in the future and our first though is to generalize the index.mld renaming:

If you have foo/bar/index.mld, it makes a page named foo/bar.html and have foo/bar/*.mld as children.

[jonludlam]

@dbuenzli you're right, linking to the index is currently broken. We can fix this by having links to page-index resolve to the parent, but it all feels a little hacky. If anyone has any other bright ideas I'll be very happy to hear suggestions.

The index of package page is not generated using Odoc ? It can't make references to {!page-index}, as that would be very ambiguous.

It's only ambiguous knowing how the various drivers work - it's entirely unambiguous in the context of having a flat hierarchy of mld pages per package - ie, the view that the package author has. There can be only 1 index.mld file, as they're all stored in the same directory (and dune at least enforces this - I presume other packaging tools do similarly.) When we have support for user-written hierarchical docs, we can be more explicit in what to name your mlds, how the hierarchy works, how to make links between pages correctly and whatnot.

[dbuenzli]

The index of package page is not generated using Odoc ?

I'm not sure what you are asking here. The index of the package page is generated by processing the index.mld file provided by the package or one generated by the driver if none is provided.

Note that currently odig is actually using the legacy --pkg option (I'm planning to rewrite the driver any time soon). So I'm wondering whether currently odig is not maybe making something that is not described by this convention in this PR and we are missing a bit.

It can't make references to {!page-index}, as that would be very ambiguous.

I notice for the example I mention that while the link gets correctly rendered in the brzo driver which I use for docsets on erratique.ch, it's neither rendered by odig nor by ocaml.org.

Though I believe we need references through the page hierarchy like {!page-packages.odoc.index} but even then, the index part isn't wanted.

For cross packages links I don't really care if there's an additional .index needed. However one thing that is important is not to have to hard code your package/parent name everywhere. Even for your example of deeper hierarchies you want to be able to refer to your parent's index page without having to name it.

Besides I guess that if we want these cross packages links to work. Then we need to go a bit beyond what is described here, on how things need to be compiled. (In the legacy pipeline the --pkg option made all that clear).

If you have foo/bar/index.mld, it makes a page named foo/bar.html and have foo/bar/*.mld as children.

I don't really care how the page end up being named but we need a reliable way to be able to refer to these in the sources. However in general the idea of solving these problems by renaming .mld by the driver doesn't seem very sound. It's rather something you should specify at the cli level when you compile the source.

[dbuenzli]

(The slighly annoying bit is that package cross references go through index then e.g.
{!pkg.index.mypage})

[dbuenzli]

Ah but no in fact we simply do:

pkg <-- empty, non materialized page created by the driver
├── Mymodule
├── index.mld
├── myimage.gif
└── page.mld

and simply decrete that index.mld is the landing page shown to users.

Note that this essentially comes back to the old --pkg option.

[Julow]

I think we should focus here on what currently works (or should work!)

I think it's useful to discuss more because we'll want deep hierarchy soon and it would be a shame to create conflicts now.

furthermore it is entirely impossible to link to that page because you've no idea what the driver might have named it

Maybe it's reasonable to say that we never want to reference a parent page by its name as it was said above ?
Though, the convention should tell how it should be renamed.

In my opinion these kind of shortcuts tend to bring in more problems than they solve because now you need to say something about what happens when you have both foo.mld and foo/index.mld.

Defining what happens in this case does not seem worse than having undefined references exist.

I think good looking URLs, then good looking references come before work in the driver.

It seems assets references proposed in #1002 have some kind of search procedure (not sure it's a good idea, but why not) what if we start treating pages references like assets ?

Yes I think we want that! It's currently impossible to solve a page name conflict without limiting what can be linked to (by limiting the search path, as it's done currently).
It's also a reason why we need dot references, a page outside of this hierarchy could do {!page-pkg.index.mypage}.

We also want to do:

pkg <-- an mld page created by the driver but compiled like any other pages.
├── Mymodule
├── index.mld <-- Has a reference to 'mydir/index.mld'
├── myimage.gif
├── page.mld
└── mydir/
    ├── index.mld <-- Has a reference to 'index.mld' in the parent directory
    └── myimage.gif

[dbuenzli]

Defining what happens in this case does not seem worse than having undefined references exist.

Well it goes without saying that if you use the reference it warns you for the fact it doesn't exist.

Yes I think we want that!

Actually I'm not sure. I think it would be clearer to have explicit parent notation, that is the moral equivalent of .. which we all know perfectly is sadly missing from ocaml's module system :-)

Maybe it's reasonable to say that we never want to reference a parent page by its name as it was said above ?
Though, the convention should tell how it should be renamed.

We definitively need that if we want to enable packages to escape their little island to link into other ones (in fact AFAIR that worked at some point in odoc on the --pkg option path).

pkg <-- an mld page created by the driver but compiled like any other pages.
[…]
└── mydir/
    ├── index.mld <-- Has a reference to 'index.mld' in the parent directory

But what do you put on this pkg page ? how does it materialize ?

Basically I think the idea that the parent-child relationship occurs through concrete content (i.e. a .mld page) is not necessarily a good one and we should let it go.

I think we rather want to think in terms of nested bundles: understood as a directory with .mld files, assets and possibly module files. The name of the directory defines the reference path/url prefix and the default page is index.mld so a hierarchy like:

pkg/ 
├── Mymodule
├── index.mld <-- Has a reference to 'mydir/index.mld'
├── myimage.gif
├── page.mld
└── mydir/
    ├── index.mld <-- Has a reference to 'index.mld' in the parent directory
    └── myimage.gif

Generates basically:

pkg/ 
├── Mymodule.html
├── index.html
├── myimage.gif
├── page.html
└── mydir/
    ├── index.html
    └── myimage.gif

To which you can refer to, as far as the pages are concerned and according to where you are as:

1. In another package.

{!page-pkg.index}       # pkg/index.mld
{!page-pkg.page}        # pkg/page.mld
{!page-pkg.mydir.index} # pkg/mydir/index.mld

2. When you are in the pkg level

{!page.index}           # pkg/index.mld
{!page.page}            # pkg/page.mld
{!page.mydir.index}.    # pkg/mydir/index.mld

3. When ou are in the mydir level

{!page-^index}          # pkg/index.mld
{!page-^page}           # pkg/page.mld
{!page-index}           # pkg/mydir/index.mld

That way we can actually already have structured documents now. The convention for a package pkg is simply as follows:

1. A "directory" pkg is created
2. The content of odoc-pages is rerooted to this directory
3. If there is no index.mld at that point, generate one for the package in pkg
4. The modules are added to this directory.

Directory is methaphorical here this can happen through --parent options.

This is a system that is extremely easy to use for end users as it naturally maps on the file system, no crazy renamings going on behind the scene or arbitrary attachements. The position in the file system mostly defines how your reference looks like. The only thing the user needs to be made aware of is the pkg directory creation – and only if it needs to make references into other packages.

[dbuenzli]

1. In another package.

{!page-pkg.index}       # pkg/index.mld
{!page-pkg.page}        # pkg/page.mld
{!page-pkg.mydir.index} # pkg/mydir/index.mld

Unless you have a pkg hierarchy in your bundle in which case you need to write:

{!page-^pkg.index}       # pkg/index.mld
{!page-^pkg.page}        # pkg/page.mld
{!page-^pkg.mydir.index} # pkg/mydir/index.mld

[dbuenzli]

Thanks to support from OCSF I can allocate a bit of time now to migrate the odig driver which is still using the old command set to the new command set. Could we maybe try to agree on the convention so that I can implement it directly and remove the one in odig to point to the one suggested by odoc ?

Personally I'm really not fond of the renaming business that it seems other drivers are doing. It puts more burden on document writers which must be made aware of it to make links on the index.mld page of their package and I find it rather unnatural to make links on an unmaterialized {!page-pkg} rather than on the {!page-index} that lives in my code base (which @jonludlam seemed to agree with).

I am also unconvinced by the current semantics of the --parent option which requires a concrete .mld document to exist for it to be effective (IIUC).

I think we should simply let the --parent option materialize reference path segments to which the ressource basename of pages and assets, as derived from their filename basename, are attached (vs. using --parent to attach to concrete .mld pages). So for example compiling:

--parent hey.ho index.mld
--parent hey.ho myimage.png

Materializes the references {!page-hey.ho.index} and {!assets-hey.ho."myimage.png"}. If one still want the notion of a "main page" for a sub-hierarchy that can simply be, by convention, index.mld.

If we have that then, what do you think about the scheme I proposed in this comment ? This is compatible with the current state affairs, allows reference on a package's index.mld regardless of the context, enables fully hierarchical documents with a straightforward file system name to reference name mapping (and derived URL structure), and enables inter-package page and assets links.

We would need to formalize the name resolution procedure a bit more but the idea is to make it mostly as the OCaml one in the module system with the addition of the moral equivalent of .. for which I used ^ in the proposal above; maybe @lpw25 would have opinions on that.

[panglesd]

Sorry for the late answer.

I gave some thinking on this, and I think you are right that the good way to do it is by having pages without content. I'd like to propose a slightly different way to implement that though.

Also, I'm not going to speak much about the reference resolution/syntax, since I think it is a sufficiently disjoint topic! (not completely orthogonal, I agree)

Context

I wrote some details about the current state, mostly for myself, but I think it is worth sharing.

What we currently have is:

• A page needs to have a content
• A page can have children
• If a page a has some a child, its content is put in directory a/index.html and its children live in a/.
• If a page b has no children, it's content is put in b.html.

As a consequence of this, if a page a has a child named index, there will be a clash in the content (all of them want to be rendered in a/index.html), and the last one which is html-generated will stay. This is the source of the "renaming stuff" problem: the convention is precisely to define an index.mld file, but it then cannot be added as a child.

I think we need cannot change this, to avoid generating many dead links.

New feature

However, as @dbuenzli suggests, we can simply add pages without content! And, in this case, I think we should just "unify" the two following scenario:

• A page without content, but a with a child named index, itself without children
• A page with content

The first option would be "the right way to do it", the second option "kept for compatibility purpose".

In the case of:

• A page without content, without (leaf) child named index

we could either say that it's forbidden, or say that it's allowed but cannot be referenced, or say that it's allowed, can be reference, and generate it's content from the list of children.

This would allow to map exactly a file hierarchy to a page hierarchy, with references working naturally.

Concretely, the compile command could take a --name argument instead of a mld file: odoc compile --child ... --parent ... --name "dir_name". This would produce a .odoc file used for reference resolution, but also to generate its asset children with odoc html-generate --asset ... page-dir_name.odoc.

The driver convention

As an example, the commands to compile the following hierarchy:

pkg/ 
├── Mymodule
├── index.mld
├── myimage.gif
├── page.mld
└── mydir/
    ├── index.mld
    └── myimage.gif

would be:

odoc compile --name "page-pkg" --child module-Mymodule --child page-index --child asset-myimage.gif --child page-page --child page-mydir
odoc compile --name "page-mydir" --parent pkg --child page-index --child asset-myimage.gif
odoc compile --parent pkg index.mld
odoc compile --parent pkg page.mld
odoc compile --parent mydir mydir/index.mld

So:

• A directory is turned into a compile command without content, but a "name" argument to name the page
• Pages, assets and modules have their "directory page" as parent

Comparison with @dbuenzli proposal

I think it basically achieve the same result, but is more clearly retrocompatible and closer to what we have now (so hopefully easier to implement).

In particular, with @dbuenzli's proposal:

• Since children of "directory pages" are not explicitely mentioned, we either cannot add asset children to directory, or we have to implement a "asset compiling" as suggested by --parent hey.ho myimage.png
• The coherence check (parent exists, children exists, they know each other) seems harder to define
• Some edge cases seems also hard to reason about (what if page exists, and we define a page to have parent page.dir1.dir2?)

Does that make sense?

Convention until the above is accepted and implemented

I think that we can already achieve the same result without implementing my proposal, by having the following convention:

• A directory is turned into a compile command with empty mld, named after the directory name
• Pages, assets and modules have their "directory page" as parent
• Pages named "index" are html-generated after their directory pages. This ensures that the right index.html remains!

So, taking the same example as above, the commands would be:

odoc compile --child module-Mymodule --child page-index --child asset-myimage.gif --child page-page --child page-mydir pkg.mld
odoc compile --parent pkg --child page-index --child asset-myimage.gif mydir.mld
odoc compile --parent pkg index.mld
odoc compile --parent pkg page.mld
odoc compile --parent mydir mydir/index.mld

odoc link ...

odoc html-generate --asset ... page-pkg.odocl
odoc html-generate page-mydir.odocl
odoc html-generate ...
odoc html-generate page-index.odocl
odoc html-generate mydir/page-index.odocl

Thanks for reading until here!

[dbuenzli]

Thanks @panglesd !

Convention until the above is accepted and implemented

That's useful thank you. I will try to go with that for now in my odig migration to the new command set.

This is the source of the "renaming stuff" problem: the convention is precisely to define an index.mld file, but it then cannot be added as a child.

Did you mean to define an index.html file ?

I think we need cannot change this, to avoid generating many dead links.

For my own understanding, could you be more specific about this ? Since it seems there was no support so far for hierarchical documents in any build system. It seems we just have the toplevel renaming business done by some of them, which leads to an index.html or simple FILE.mld files which leads to FILE.html so I think we should keep ourselves leeway to change things.

New feature

It seems to me that it captures the essence of what I wanted¹:

Here are few comments:

1. Maybe we should try to fine another name for --name there are already too many names out there. --node perhaps ?

2. How is the node name defined (the one you later specify in --parent) I can't see that here:

   odoc compile --name "page-pkg" --child module-Mymodule --child page-index --child asset-myimage.gif --child page-page --child page-mydir

   Is it by munging --name ?

3. Can't the --child options take files directly and produce the references itself basically take the basename, if the extension is .mld that's a page-$(basename) otherwise that's an asset-$(basename).$(ext).

I think these comments could perhaps be solved by having a separate command for nodes:

odoc compile-node --name pkg --child …

Which would produce an pkg.node.odoc file. Also I think that the --parent files could take the node file directly. This avoids looks up procedures and makes it easier for incremental build system that is:

odoc compile --parent pkg.node.odoc index.mld

Footnotes

1. My proposal didn't have the whole compilation specifics so I'm not sure I agree with your points about mine which could have been solved the way you do in yours but that's besides the point. ↩

[panglesd]

This is the source of the "renaming stuff" problem: the convention is precisely to define an index.mld file, but it then cannot be added as a child.

Did you mean to define an index.html file ?

I meant that the convention for the user is to write an index.mld file as "documentation entry page", but at the same time we want a page to contain all documentation page. We use the content of the "index" page as parent page, but we want another name for the parent page, so we might have to rename it. Not very clear, sorry :/

For my own understanding, could you be more specific about this ?

Suppose odoc.2.4.0 introduces the "empty pages feature" we are discussing and odig.0.10.0 introduces the "not always flat hierarchy" convention.
One of the compatibility property that is nice to have is that all the following (flat) hierarchies are the same:

• odig.0.9.0+odoc.2.3.0,
• odig.0.9.0+odoc.2.4.0,
• odig.0.10.0+odoc.2.4.0.

For instance, this implies that we cannot in odoc.2.4.0 remove the notion of leaf pages/page with children. (if we rendered leaf pages the same way as pages with children, that is page_name/index.html, there won't be the problem that having a leaf page named index make the two pages fight for parent_name/index.html).

But yes, I agree, we have should not restrict ourself too much!

Maybe we should try to fine another name for --name there are already too many names out there. --node perhaps ?

Sure, I just came up with that for the example. Node seems a good pick, let's see in the PR later :)

How is the node name defined (the one you later specify in --parent)

Yes, its using --name (or --node) as shown in:

odoc compile --name "page-mydir" --parent pkg --child page-index --child asset-myimage.gif

Can't the --child options take files directly and produce the references itself basically take the basename, if the extension is .mld that's a page-$(basename) otherwise that's an asset-$(basename).$(ext)

That's a possibility! I guess we should not introduce breaking changes, so maybe in another --child-file argument? Also, we should keep the possibility to add an mld file as asset, which is not possible with --child-file!

I think these comments could perhaps be solved by having a separate command for nodes

Yes, why not!

Also I think that the --parent files could take the node file directly.

+1!

Thanks for all these comments, it will definitely help if/when I implement the feature.

I'm not sure I agree with your points about mine

Sorry if I misunderstood your proposal, and made invalid points! If you think it has some advantages, feel free to defend it.

---

I'm curious about the opinion of @jonludlam and Julow, especially on the "Convention until the above is accepted and implemented" since it is what this PR was about initially!

(Of course, I'm also curious about their opinion on the rest of the discussion!)

[dbuenzli]

Yes, its using --name (or --node) as shown in:

odoc compile --name "page-mydir" --parent pkg --child page-index --child asset-myimage.gif

Well I'm not very fond of your idea to encode little languages into the argument values. You can use as much options as you want. They are cheap to define, there's plenty of characters to choose from them and it avoids having to munge stuff in drivers and unmunge them in odoc, you have a nice cli library that does that for you. I mean the above invocation could simply be (again using files directly):

odoc compile --node mydir --parent pkg.node.odoc --child-page mydir/index.mld --child-asset mydir/myimage.gif

It's much less obscure and actually shows directly in the invocation itself on which files the results depends.

That's a possibility! I guess we should not introduce breaking changes, so maybe in another --child-file argument? Also, we should keep the possibility to add an mld file
as asset, which is not possible with --child-file!

See above. Solved by having both --child-asset and --child-page.

Suppose odoc.2.4.0 introduces the "empty pages feature" we are discussing and odig.0.10.0 introduces the "not always flat hierarchy" convention.

I'm not sure I understand your points here. In any case new odig releases tend to constraint odoc anyways because there are usually changes in the generated markup which means that the stylesheets do not or can't work well accross odoc versions.

I think you should not take compatibility too much into account, it seems we have only three or four odoc drivers out there. It shouldn't be too hard to coordinate a release and constraint odoc appropriately in those packages that depend on it. Moving the odoc lower bound is never a problem AFAIK.

If you think it has some advantages, feel free to defend it.

I didn't think deeply about it but I can't really make a difference with what you are proposing. What's the difference ? You can still attach stuff to pages ? Maybe we can simply say no page can be a --node (which as I already mentioned would like not break anything) and then we get to the same thing no ?

[panglesd]

Well I'm not very fond of your idea to encode little languages into the argument values.

That's not really my idea, I'm just using what is already in odoc... But I agree that --child-<kind> <file> seems better.

I'm not sure I understand your points here

Instead of using odig as an example, I should have used voodoos, the ocaml.org driver. I would prefer if the links were unchanged, between voodoos+odoc before the new feature, and voodoos+odoc after the new feature, for packages that do not use the feature (so have flat hierarchy). But anyway, that's already the case with this proposal.

I think you should not take compatibility too much into account

Thanks, good to know!

I didn't think deeply about it but I can't really make a difference with what you are proposing

What I understood from your proposal was that nodes did not have a corresponding odoc file, but were guessed from the --parent argument of pages. For instance in --parent hey.ho page.mld I though hey and ho could be "nodes" without odoc file. That's what I was discussing in the "comparison with your proposal", but maybe it was made from a wrong interpretation of your examples!
Other than that, I think it's identical, which is good!

[dbuenzli]

For instance in --parent hey.ho page.mld I though hey and ho could be "nodes" without odoc file.

Ah yes maybe. But I'm not against having to devise empty node .odoc files for them. For me the gist of the idea was rather to stop having .mld documents as attachement points and to have a convention in which odoc reference names can be trivially inferred from the structure of the file system.

We will however still need a discussion about scoping. Basically in your example above if in pkg/page.mld I use {!page-index} this should refer to pkg/index.mld. I think the least surprising is to do it like OCaml does in the module system (you can see nodes as defining a struct with its children therein) – after that maybe we can consider adding a ../.

[Julow]

I believe this documents the current state. I would like to merge this now and to continue the discussion in other issues.

#991
#1037
#1038

[jonludlam]

I don't think any driver that follows this spec renders source today. If we're just documenting how things work right now we should omit this section.

[Julow]

doc/driver.mld does that. This document doesn't describe what an existing driver does, rather what can be done by a driver today.

[dbuenzli]

I think it's better if we document what driver should do, so that there are no useless discrepancies between them.

[Julow]

I think this paragraph should be kept then.

[jonludlam]

doc/driver.mld is a bad example for this - it follows literally none of the other conventions in the paragraph, as the intent back when it was first written was to show how odoc can be used more generally than just for making docs from installed opam packages.

We haven't quite figured out a convention for the sources, and it's not at all clear to me that the hierarchy will share the same namespace as the other package pages. As a concrete example, on ocaml.org cmdliner's pages currently appear under the hierarchy https://ocaml.org/p/cmdliner/latest/doc/ - e.g. tutorial.mld is https://ocaml.org/p/cmdliner/latest/doc/tutorial.html . I can easily imagine that there would be a desire for the source to appear under https://ocaml.org/p/cmdliner/latest/src/ in which case there's no namespace clash.

I'd quite like in this PR to just focus on what's actually implemented 'in the wild' and then we can have a separate discussion on these sorts of topics to bash out the wider question about what we should do to support the new features.

[Julow]

I don't agree that what exists in the wild is in the scope of this PR but I removed the paragraph to not interferes with the future changes.

[jonludlam]

Thanks!

[panglesd]

Thanks to support from OCSF I can allocate a bit of time now to migrate the odig driver which is still using the old command set to the new command set. Could we maybe try to agree on the convention so that I can implement it directly and remove the one in odig to point to the one suggested by odoc ?

@dbuenzli We are currently working on a better CLI and convention for building hierarchical docs. I remembered your message quoted above, so: If you did not work too much on porting odig to the new command set, you might want to wait a little bit to jump directly to the last command set! (the design will be posted for discussion as soon as it is ready).

[dbuenzli]

I kept on postponing that for now. I wanted to squeeze that soon (next week) but it may fall through again. When do you expect to post the design ?

[panglesd]

(Sorry for the delay, but I'll be able to answer more accurately this evening!)

[panglesd]

The document we have is clearly not finished enough and there would be no interest in sharing it now. It's hard to give an estimate on when it will be in a shareable state, but for sure that won't happen this week unfortunately.

We do think it would be better to wait for the new design to be published and validated ; and maybe even implemented in odoc, instead of porting odig to --parentnow.

[dbuenzli]

Is there any reason why odoc is being increasingly developed behind closed doors ?

[panglesd]

Let me give more details, since re-reading my post I agree it gives more an impression of being developed "closed doors" than I think is the case. In particular the use of "we" without being explicit who it is referring to.

I'm a Tarides employee and work on odoc as part of this job. For this particular task (improving the CLI to enable hierarchical documentations, assets, render source code and a CLI friendlier to an incremental build system) I started working with @Julow on a proposal.
We (@Julow and I) often ask for feedback to @jonludlam and @trefis. This is what happened yesterday and there was quite a lot of things to change in the proposal. We (this time @Julow @jonludlam @trefis and I) decided that the document required such significant changes that opening it up to the public now would be a waste of time for the public (since it will change anyway).

Of course, this first stage of "in private" writing of the proposal will not replace a public discussion here.

Many of this was discussed in odoc dev meetings. I don't know if those meetings are public or not? (I'm participating also to some ppxlib dev meeting, which are public, with public notes, which is working quite well. If odoc dev meetings are not public, I would be in favor of having them public, especially public notes.)

I cannot compare to how odoc was before, since I was not there at the time. What has changed to give this impression?

[dbuenzli]

Thanks for you clarification @panglesd. My impression also stems from this (which could perfectly have been a public issue).

[panglesd]

I see! We use a gh board to organize ourselves, usually with issues that more or less just link to a public one. This time it contained a bit more than that, and indeed it should have been a public issue. I added the content of the "issue" that was referenced in the PR.