Skip to content

Commit

Permalink
Revise TOML format
Browse files Browse the repository at this point in the history
Instead of an `[Inventory]` section, the TOML format now has to start
with a comment line that declares the format version. The header fields
`project` and `version` are now top-level keys.

This looks cleaner, saves a few bytes, and introduces unique "magic
bytes" at the beginning of the file so that an `inventory.toml` file
could be recognized as a distinct fromat by FileIO.jl.

Also, `version` is now optional.

Most previous (experimental) `inventory.toml` files can still be read
without error.
  • Loading branch information
goerz committed Jan 27, 2024
1 parent 64f349a commit e23ec34
Show file tree
Hide file tree
Showing 7 changed files with 196 additions and 43 deletions.
9 changes: 9 additions & 0 deletions .typos.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
[default]
extend-ignore-identifiers-re = [
"[A-Z]T",
"[0-9abcdef]{8}",
"[A-Z][a-z]+[A-Z]{2,5}[0-9]{4}",
]

[files]
extend-exclude = ["*.bib", "*.toml", "test_inventory.jl"]
17 changes: 9 additions & 8 deletions docs/src/formats.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,23 +35,24 @@ Note that `DocInventories` internally uses the `text/x-intersphinx` MIME type fo

## TOML Format

The TOML format is another text output format that is optimized for human readability. It starts with a header section of the form
The TOML format is a text output format that is optimized for human readability. It starts with a header section of the form

```
[Inventory]
format = "DocInventories v0"
```toml
# DocInventory version 0
project = "<project>"
version = "<version>"
```

The `format` line is mandatory and identifies the file as containing inventory data in the format described here.
The comment in the first line is mandatory and identifies the file as containing inventory data in the format described here.

!!! warning
As indicated by the `v0` in the `format` line, the format described here is currently experimental and may change without notice
As indicated by the "version 0" in the header comment line, the format described here is currently experimental and may change without notice

The `project` line must specify the name of the project described by the inventory. It is mandatory. The `version` line may specify the version of the project. It is optional, but recommended.

After that, each [`InventoryItem`](@ref) is represented by a multi-line block of the form

```
```toml
[[<domain>.<role>]]
name = "<name>"
uri = "<uri>"
Expand All @@ -61,7 +62,7 @@ priority = <priority>

The four lines for `name`, `uri`, `dispname`, and `priority` may occur in any order. Also, for items with the default priority (`-1` for the `std` domain, `1` otherwise), the `priority` line may be omitted. If `dispname` is equal to `name` (usually indicated by `dispname="-"`), the `dispname` line may also be omitted.

The item-blocks may be grouped/separated by blank lines. In `.toml` file generated by `DocInventories.save("inventory.toml", inventory)` items will be grouped into blocks with the same `[[<domain>.<role>]]` with a blank line between each block.
The item-blocks may be grouped/separated by blank lines. In `.toml` files generated by `DocInventories.save("inventory.toml", inventory)` items will be grouped into blocks with the same `[[<domain>.<role>]]` with a blank line between each block.

Any TOML parser should read a `.toml` file with the above structure into a nested dictionary, so that `item_dict = toml_data[domain][role][i]` corresponds to the `i`'th inventory item with the given `domain` and `role`. That `item_dict` will then map `"name"`, `"uri"`, and potentially `"dispname"` and `"priority"` to their respective values.

Expand Down
3 changes: 1 addition & 2 deletions docs/src/inventories/Documenter.toml
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
[Inventory]
format = "DocInventories v0"
# DocInventory version 0
project = "Documenter.jl"
version = "1.2.1"

Expand Down
4 changes: 2 additions & 2 deletions docs/src/inventories/Julia.toml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
[Inventory]
format = "DocInventories v0"
# DocInventory version 0

project = "The Julia Language"
version = "1.10.0"

Expand Down
4 changes: 1 addition & 3 deletions docs/src/inventories/JuliaDocs.toml
Original file line number Diff line number Diff line change
@@ -1,7 +1,5 @@
[Inventory]
format = "DocInventories v0"
# DocInventory version 0
project = "JuliaDocs"
version = ""

[[std.doc]]
name = "DocumenterInterLinks"
Expand Down
59 changes: 31 additions & 28 deletions src/toml_format.jl
Original file line number Diff line number Diff line change
Expand Up @@ -25,47 +25,50 @@ function Base.show(io::IO, ::MIME"application/toml", inventory::Inventory)
push!(domains[item.domain][item.role], item_data)
end

toml_dict = Dict(
"Inventory" => Dict(
"format" => "DocInventories v0",
"project" => inventory.project,
"version" => inventory.version,
),
domains...
)
toml_dict = Dict("project" => inventory.project, domains...)
if !isempty(inventory.version)
toml_dict["version"] = inventory.version
end

TOML.println(io, "# DocInventory version 0")
TOML.print(io, toml_dict; sorted=true)

end


function read_inventory(io::IO, ::MIME"application/toml")
format_header = readline(io)
if !startswith(format_header, "#")
@warn "Invalid format_header: $format_header"
# TODO: verify format_header more strictly once the format has settled.
end
data = TOML.parse(io)
try
inventory_format = data["Inventory"]["format"]
if inventory_format != "DocInventories v0"
msg = "Invalid inventory format: $(repr(inventory_format))"
throw(InventoryFormatError(msg))
end
project = data["Inventory"]["project"]
version = data["Inventory"]["version"]
project = string(pop!(data, "project")) # mandatory
version = string(pop!(data, "version", "")) # optional
items = InventoryItem[]
for (domain, domain_data) in data
(domain == "Inventory") && continue # that's the header, not a domain
for (role, role_data) in domain_data
for item_data in role_data
push!(
items,
InventoryItem(
item_data["name"],
domain,
role,
get(item_data, "priority", (domain == "std") ? -1 : 1),
item_data["uri"],
get(item_data, "dispname", "-")
if domain_data isa Dict
for (role, role_data) in domain_data
for item_data in role_data
push!(
items,
InventoryItem(
item_data["name"],
domain,
role,
get(item_data, "priority", (domain == "std") ? -1 : 1),
item_data["uri"],
get(item_data, "dispname", "-")
)
)
)
end
end
elseif domain == "format"
# For backward-compatibility, remove this `elseif` in v1.0
@warn "Unexpected key: $domain"
else
throw(InventoryFormatError("Unexpected key: $domain"))
end
end
return project, version, items
Expand Down
143 changes: 143 additions & 0 deletions test/test_inventory.jl
Original file line number Diff line number Diff line change
Expand Up @@ -181,6 +181,118 @@ end
@test c.value isa InventoryFormatError
@test contains(c.output, "Unexpected line")

filename = joinpath(tempdir, "missing_project.toml")
#!format: off
write(filename, """
# DocInventory version 0
[[std.doc]]
name = "DocumenterInterLinks"
uri = "DocumenterInterLinks.jl#readme"
""")
#!format: on
c = IOCapture.capture(rethrow=Union{}) do
Inventory(filename; root_url="")
end
@test c.value isa InventoryFormatError
@test contains(c.output, "key \"project\" not found")

filename = joinpath(tempdir, "old_format.toml")
#!format: off
write(filename, """
[Inventory]
format = "DocInventories v0"
project = "Test"
version = "0.1.0"
[[std.doc]]
name = "DocumenterInterLinks"
uri = "DocumenterInterLinks.jl#readme"
""")
#!format: on
c = IOCapture.capture(rethrow=Union{}) do
Inventory(filename; root_url="")
end
@test_broken c.value isa InventoryFormatError
@test contains(c.output, "Unexpected key: format")
@test contains(c.output, "Invalid format_header: [Inventory]")

filename = joinpath(tempdir, "old_format2.toml")
#!format: off
write(filename, """
# This is an old inventory file, and because of this comment, the
# `[Inventory]` won't be stripped as a format_header
[Inventory]
format = "DocInventories v0"
project = "Test"
version = "0.1.0"
[[std.doc]]
name = "DocumenterInterLinks"
uri = "DocumenterInterLinks.jl#readme"
""")
#!format: on
c = IOCapture.capture(rethrow=Union{}) do
Inventory(filename; root_url="")
end
@test c.value isa InventoryFormatError
@test contains(c.output, "key \"project\" not found")

filename = joinpath(tempdir, "no_header_line.toml")
#!format: off
write(filename, """
project = "Test"
version = "0.1.0"
[[std.doc]]
name = "DocumenterInterLinks"
uri = "DocumenterInterLinks.jl#readme"
""")
#!format: on
c = IOCapture.capture(rethrow=Union{}) do
Inventory(filename; root_url="")
end
@test c.value isa InventoryFormatError
@test contains(c.output, "Invalid format_header: project = \"Test\"")
@test contains(c.output, "key \"project\" not found")

filename = joinpath(tempdir, "typo1.toml")
#!format: off
write(filename, """
# DocInventory version 0
project = "Test"
verison = "0.1.0"
[[std.doc]]
name = "DocumenterInterLinks"
uri = "DocumenterInterLinks.jl#readme"
""")
#!format: on
c = IOCapture.capture(rethrow=Union{}) do
Inventory(filename; root_url="")
end
@test c.value isa InventoryFormatError
@test contains(c.output, "Unexpected key: verison")

filename = joinpath(tempdir, "typo2.toml")
#!format: off
write(filename, """
# DocInventory version 0
project = "Test"
version = "0.1.0"
[[std_doc]]
name = "DocumenterInterLinks"
uri = "DocumenterInterLinks.jl#readme"
""")
#!format: on
c = IOCapture.capture(rethrow=Union{}) do
Inventory(filename; root_url="")
end
@test c.value isa InventoryFormatError
@test contains(c.output, "Unexpected key: std_doc")

end

end
Expand Down Expand Up @@ -484,6 +596,37 @@ end
@test contains(c.output, "Only v2 objects.inv files currently supported")
@test c.value isa InventoryFormatError

filename = joinpath(tempdir, "wrong_version_type.toml")
#!format: off
write(filename, """
# DocInventory version 0
project = "Test"
version = 1.0
[[std.doc]]
name = "DocumenterInterLinks"
uri = "DocumenterInterLinks.jl#readme"
""")
#!format: on
readinv = Inventory(filename; root_url="")
# this still works because the float "1.0" is converted to string
@test readinv.version == "1.0"

filename = joinpath(tempdir, "future_header.toml")
#!format: off
write(filename, """
# Documenter Inventory version 1
project = "Test"
version = "1.0"
[[std.doc]]
name = "DocumenterInterLinks"
uri = "DocumenterInterLinks.jl#readme"
""")
#!format: on
readinv = Inventory(filename; root_url="")
@test readinv.project == "Test"

end

end

0 comments on commit e23ec34

Please sign in to comment.