Add templating to publishing functionality #515

wojciechczerniak · 2021-06-13T18:19:45Z

Hello,

This time a feature request.

Problem

I've seen that there are few requests/issues with customization of the Markdown output:

And I would have few others requirements:

Start from 2nd lvl header
Format links as list
Add custom class to up/down links
Insert table of contents
Various smaller formatting adjustments

But IMO a change from string building to one of the templating languages may solve all of the above in a one go

Solution idea

I think Jinja2 is the number one for Python libraries? With a template and an ability to use our own fixing all of the above issues would be just a matter of customization with --template my-markdown.jinja2

Remove hard-coded string building in HTML publisher:

doorstop/doorstop/core/publisher.py

Lines 165 to 167 in a770dc4

    
           yield '<!DOCTYPE html>' 
        
           yield '<head>' 
        
           yield (

and Markdown markdown publisher:

doorstop/doorstop/core/publisher.py

Lines 399 to 409 in a770dc4

    
           heading = '#' * item.depth 
        
           level = _format_level(item.level) 
        
           if item.heading: 
        
               text_lines = item.text.splitlines() 
        
               # Level and Text 
        
               if settings.PUBLISH_HEADING_LEVELS: 
        
                   standard = "{h} {lev} {t}".format( 
        
                       h=heading, lev=level, t=text_lines[0] if text_lines else '' 
        
                   ) 
        
               else:

With Jinja2 templates defaulting to whatever is in those methods:

# {{ document.header }}

{% for item in document.items %}
  ## {{ item.header }} <small>{{ item.id }}</small> { #{{ item.id }} }
  
  {{ item.text }}
  
  {% if item.PUBLISH_CHILD_LINKS %}  
  Links:
    {% for link in links %}
        [{{ link.header }}]({{ link.file }}#{{ link.id }})
    {% endfor %}
  {% endif %}
  
  {% if item.document.publish %}  
        | Attribute | Value |
        | --------- | ----- |
    {% for attr in item.document.publish %}
        | {{ attr }} | {{ item.attribute(attr) }} |
    {% endfor %}
  {% endif %}
{% endfor %}

AND allow the --template arg for all output formats.

doorstop publish -m all docs/requirements/ --template my-template.jinja2

Use cases

You want Frontmatter/YAML headers in Markdown #295, just add them to the template:

---
title: {{ page.header | title() }}
metadata: {{ page.description | truncate(240) }}
---
{% block content %}
.... content

Footer maybe as in #166? Add it at the end:

.... content
{% endblock  %}

Copyright Year, Restrictions

The text was updated successfully, but these errors were encountered:

eterX · 2021-06-25T19:55:43Z

Hi Wojciech

I realize that it would be a very useful feature. I've been using something like that, for communication documents (but without Jinja, your approach looks far more sound)

As soon as a I get #507 done, count on some help on this :)

Leandro

eterX · 2021-07-20T19:58:42Z

Hi!

As promissed, started exploring this feature along with @lucasd92, now that #507 is finally waiting to be merged, (maybe @jacebrowning ?) :)
@lucasd92 managed to replace Bottle's default template engine with Jinja2. Please, see this fork (branch feature/jinja) that can give us a glipse of the feature's potential, by:

issuing the command doorstop publish all /mydir -H --template test_jinja2
tweaking views/test_jinja2_index.tpl
modifying the test at test_all.py

Looks promissing!

First pitfall we came across: Jinja2's output can be consumed as an iterator, and Doorstop expects that. Thusly allowing for iterator chains and coroutines. But it looks like iterators composition is lost in Bottle.Jinja2Template. In spite of the description saying "Get a rendered template as a string iterator" it calls render() instead of generate() with no altarnetive. Workarounded here, somewhat piercing Bottle's abraction.
IMHO, it might not be an important difference in terms for small/big projects, but for future API endpoints. Maybe more if the project moves to FastAPI and async methods?

Best regards,

monch1962 · 2021-10-15T10:42:18Z

I'm also looking into options to use customised reporting templates with Doorstop.

Given the challenges @eterX outlines above, would it make more sense to build a separate, template-based Doorstop reporter and leave the current Doorstop reporting as is? Doorstop's YAML data format is elegant and extensible, and I think the extensibility is where the magic is.

For example, I'm currently using Doorstop with custom requirements fields that I'll want to use every time (e.g. classifying each requirement using FURPS+), but also with fields that will change from project to project. These could include e.g.

requirement-source-url field to link to external requirements documents,
requirement-source-category field to describe whether individual requirements are regulatory, project-specific, enterprise-standard, industry best practice, etc.
a jira-url array that link to any Jira stories that describe how the requirement is implemented
a terraform-url array that link to any Terraforms that can be checked to validate that specific requirements are met

Given how well Doorstop's extensibility supports this approach, it makes sense to have reporting templates be similarly customisable to specific use cases. There's also Doorstop APIs that give access to all the custom fields, which make it easier still to implement.

I'll knock something simple together and post it here as an example

monch1962 · 2021-10-18T12:05:09Z

Jinja is a pretty handy templating engine for Python. This approach lets you use Jinja templates with doorstop:

#!/usr/bin/env python
import argparse
import doorstop
import sys
from jinja2 import Template, Environment, nodes
from jinja_markdown import MarkdownExtension

with open("./templates/report.html.jinja2", "r") as f:
    # This would work if you didn't have to load a Jinja extension i.e. MarkdownExtension
    #t = Template(f.read())

    # This is the approach to consume a template that includes {% markdown %}..{% endmarkdown %} tags
    s = f.read()
    t = Environment(extensions=[MarkdownExtension]).from_string(s)
output = t.render(dependency_tree=repr(tree))

with open("./dist/rendered.html", "w") as f:
    f.write(output)

Inside your Jinja template, you can then use something like

<h2>Dependency tree: {{dependency_tree|e}}</h2>

{% markdown %}
---
Start of markdown content

# H1 heading
## H2 heading

- list item
- another list item

End of markdown content
---

{% endmarkdown %}

to present your doorstop data. Note that you might need to escape your doorstop data if it contains certain characters, which is why I've got {{ dependency_tree | e }}. Note also that adding the Markdown extension to Jinja lets Jinja work with Markdown content, which you might be using inside your doorstop YAMLs.

Hopefully I haven't cut down this code too much to make it understandable.

Now I just need to work out how to walk the tree of linked documents using the doorstop scripting interface...

yoyodyn · 2023-07-23T11:45:53Z

Would it be possible to specify the publishing template within the .doorstop.yml file for each document/type?

neerdoc · 2024-01-26T08:26:37Z

@wojciechczerniak Could you have a look at version v.3.0b12 and see if you think this version solves this issue?

jacebrowning added feature help wanted labels Jun 13, 2021

neerdoc mentioned this issue Mar 2, 2022

LaTeX publisher #545

Merged

wojciechczerniak mentioned this issue Mar 15, 2022

Jinja2 / Mustache syntax incompatibility #513

Open

yoyodyn mentioned this issue Jul 21, 2023

Overriding publishers #597

Closed

neerdoc mentioned this issue Jan 22, 2024

Rework of HTML publisher. #622

Merged

neerdoc closed this as completed in #622 Jan 26, 2024

neerdoc reopened this Jan 26, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add templating to publishing functionality #515

Add templating to publishing functionality #515

wojciechczerniak commented Jun 13, 2021 •

edited

Loading

eterX commented Jun 25, 2021

eterX commented Jul 20, 2021

monch1962 commented Oct 15, 2021

monch1962 commented Oct 18, 2021

yoyodyn commented Jul 23, 2023

neerdoc commented Jan 26, 2024

Add templating to publishing functionality #515

Add templating to publishing functionality #515

Comments

wojciechczerniak commented Jun 13, 2021 • edited Loading

Problem

Solution idea

Use cases

eterX commented Jun 25, 2021

eterX commented Jul 20, 2021

monch1962 commented Oct 15, 2021

monch1962 commented Oct 18, 2021

yoyodyn commented Jul 23, 2023

neerdoc commented Jan 26, 2024

wojciechczerniak commented Jun 13, 2021 •

edited

Loading