🎧 Kaizen 8 - 24 improvements & a lot more - Nov. 23, 2022

[gerhard]

This is now available as shipit.show/80! See all improvements in this 🐙 GitHub repository: Kaizen 7 ... 8

---

For the last Kaizen episode of 2022 - https://shipit.show/80 🎧 Nov. 23, 2022 - I want us to improve the conversation with our listeners. Rather than reference some PRs & talk about the changes that we did in the 10 weeks since the previous Kaizen, I want us to create an open-ended discussion (like this one) which gives everyone the opportunity to participate as the improvements are happening. Instead of waiting to hear what happened, if you are interested and are able to make the time, you can be part of it. You can start by leaving a comment below, or simply adding a reaction to this post 👍 👎 😄 🎉 😕 ❤️ 🚀 👀

On top of allowing others to participate, we (+@jerodsanto +@adamstac) can experiment with keeping the episode notes in public (yes, directly in this discussion), rather than in a private doc. By the way, we have been using HackMD for this purpose for all previous Kaizens. An outline from 📻 previous Kaizen 7 👉

Before I wrap up this intro, I want to mention the Arbitrary deadlines are actually awesome article that @jerodsanto posted a month ago. As it turns out, that was a follow-up to a blog post by @lucasfcosta. This is the paragraph that seeded the idea of these Kaizen discussions:

Every tenth episode of Ship It! we talk about the work we’re doing here at Changelog to make things better. That means I know that every two and a half months, I’m gonna go on a podcast and talk about what I accomplished. So when that sucker is coming up my brain is like, “We’re shipping stuff!”

Jerod Santo - Oct. 19, 2022

The last thing which I want to mention is that, while I have a good feeling about this initiative and want to try it out, as with any experiment, this might be just a one off. I don't have any expectations beyond giving it a go and seeing what happens - sound familiar? I am especially curious how easy & straightforward this will be in practice for @jerodsanto & @adamstac. If you are a committed Kaizen fan and want to share your perspective, I look forward to reading your comment(s) below.

What are the 🎧 Kaizen 8 highlights?

Before we unpack the highlights, I want to start off with something that I have not done since 2019: a diagram of our setup. I suspect that @jerodsanto & @adamstac will appreciate being able to visualise all the components of our current setup 😎

graph TB
    classDef link stroke:#59b287,stroke-width:3px;
    %% Code & assets
    subgraph GitHub
        repo{{ fab:fa-github thechangelog/changelog.com }}:::link
        click repo "https://github.com/thechangelog/changelog.com"
        cicd[/ fa:fa-circle-check GitHub Action - Ship It \]:::link
        click cicd "https://github.com/thechangelog/changelog.com/actions/workflows/ship_it.yml"
        automation[\ fa:fa-space-shuttle dagger v0.1 /]:::link
        click automation "https://github.com/thechangelog/changelog.com/blob/master/2021/dagger/prod_image/main.cue"
    end
    
    registry(( fab:fa-docker Docker Hub )):::link
    click registry "https://hub.docker.com/r/thechangelog/changelog.com/tags"
    chat(( fab:fa-slack Slack )):::link
    click chat "https://changelog.slack.com/archives/C03SA8VE2"

    repo -.- |.github/workflows/ship_it.yml| cicd
    repo -.- |2021/dagger/prod_image/main.cue| automation
    
    cicd --> |runs| automation ---> |wireguard| container --> |app image| registry -.-> |app image| app
    cicd ---> |flyctl deploy| app
    cicd ----> |success #dev| chat

    repo -.- |2022.fly config| app
    repo -.- |2022.fly/docker config| container

    repo -.-> |make runtime-image| registry -.-> |runtime image| automation

    %% PaaS - https://fly.io/dashboard/changelog
    subgraph Fly.io
        proxy{fa:fa-globe Proxy}
        proxy ====> |https| app

        subgraph IAD
            container([ fab:fa-docker Docker Engine 2022-03-13 ]):::link
            click container "https://fly.io/apps/docker-2022-06-13"
            app(( fab:fa-phoenix-framework App changelog-2022-03-13.fly.dev )):::link
            style app fill:#488969;
            click app "https://fly.io/apps/changelog-2022-03-13"
            dbw([ fa:fa-database PostgreSQL Leader 2022-03-12 ]):::link
            click dbw "https://fly.io/apps/postgres-2022-03-12"
            dbr1([ fa:fa-database PostgreSQL Replica 2022-03-12 ])

            app <==> |pgsql| dbw
            dbw -.-> |replication| dbr1
        end

        metricsdb([ fa:fa-chart-line Prometheus ])
        metrics[ fa:fa-columns Grafana fly-metrics.net ]:::link
        click metrics "https://fly-metrics.net"
        metrics --- |promql| metricsdb
        metricsdb -..- |metrics| app
        metricsdb -..- |metrics| dbw
        metricsdb -..- |metrics| container
    end

    secrets(( fa:fa-key 1Password )):::link
    click secrets "https://changelog.1password.com/"
    secrets -.-> |secrets| app
        secrets -.-> |secrets| cicd

        
    %% Exceptions
    exceptions(( fa:fa-car-crash Sentry )):::link
    click exceptions "https://sentry.io/organizations/changelog-media/issues/?project=5668962"
    app -..-> |exceptions| exceptions

    %% CDN - https://manage.fastly.com/configure/services/7gKbcKSKGDyqU7IuDr43eG
    subgraph Fastly
        apex[ changelog.com ]:::link
        click apex "https://changelog.com"
        
        subgraph Ashburn
            cdn[ cdn.changelog.com ]
        end
    end
    observability(( fa:fa-bug Honeycomb )):::link
    click observability "https://ui.honeycomb.io/changelog/boards"
    apex -.-> |logs| observability
    apex ==> |https| proxy
    subgraph AWS.S3
        subgraph us-east-1
            assets[ fab:fa-aws changelog-assets.s3.amazonaws.com ]
        end
    end
    cdn ==> |https| assets

    %% Monitoring
    monitoring(( fa:fa-table-tennis Pingdom )):::link
    click monitoring "https://my.pingdom.com/app/newchecks/checks"
    monitoring -.-> |monitors| apex
    monitoring -.-> |monitors| cdn
    monitoring -.-> |monitors| app

Loading

👆 All those 🟩 green elements are links - click them and see what happens!

Ever since I spoke with @kelseyhightower on 🔈 shipit.show/44, I really wanted to document how everything works. While this is just a mermaid flowchart, it is a step in that direction (and hopefully worth a thousand words). Click on the ↔ to expand the diagram for the best viewing experience - it looks the part on 2880 x 1620.

Before we move on, take a moment to consider just how better having these diagrams is for visualising improvements. Did I mention that they are all code? If you take one thing away from the new Kaizen experiment, this would be it: Put some 🧜‍♀️ https://mermaid.live in your GitHub markdown files

OK, so what are the other highlights?

1/3. Kaizen tags

I thought that it would be a good idea to tag all commits in this repository which matched the publish date of every previous Kaizen episode. You will find them all here: https://github.com/thechangelog/changelog.com/tags

My favourite part is that now we can all see what changed in our changelog.com setup between the Kaizens:

• Kaizen 1 ... 2
• Kaizen 2 ... 3
• Kaizen 3 ... 4
• Kaizen 4 ... 5
• Kaizen 5 ... 6
• Kaizen 6 ... 7

Which brings us to today, a few weeks before Kaizen 8: Kaizen 7 ... today, Nov. 13. As you can see, @jerodsanto was rather busy 🐰 . I know what you're thinking: could we have a higher-level view of what actually changed? Do you mean something like a... changelog 😁

2/3. Issues & PRs

These are the issues that we have closed since Kaizen 7:

• Link to Scribe URLs instead of Medium in News? #392
• Can't upgrade Elixir/Erlang following instructions in README #425
• Upgrade Elixir from 1.13 to 1.14 #427
• Certain characters in RSS feeds aren't being HTML-escaped and are causing browser RSS/XML parsers to throw errors #429
• 503 Error when hitting a Post link #432

And here are all the improvements that shipped as PRs since Kaizen 7:

• enable syntax highlight docker #415 🙏 Thank you @gilacost
• How to container images with Docker on Fly.io #426
• Bump from Elixir 1.13 to 1.14 (and OTP 24 to 25) #428 🙏 Thank you @Nezteb
  • two follow-ups for @jerodsanto Bump from Elixir 1.13 to 1.14 (and OTP 24 to 25) #428 (comment)
• Increase connections limit for our Fly.io app instance #431
• chore: New contributing guide #434
• Prepare for 2023 setup #436

There is also a bunch of changes that we - it's the royal we, this is all @jerodsanto - did as small improvements shipped straight into production (no PR):

1. [admin] Queue a AudioUpdater after episodes get updated
2. Replace Google reCAPTCHA with Cloudflare Turnstile
3. Add a field to distinguish news publications from platforms + ee974d4
4. Redirect Medium posts to Scribe instead
5. [admin] Add a button for copying chapter data to ++ file
6. Ensure chapter links and images have full URL + 5cf8cea
7. Only show author avatar on news items if one has been uploaded
8. [admin] Add image URL fields to chapters form
9. Update mix deps
10. Improve person-related data deletion rules
11. Improve how we track visited news links
12. We're down with OP3
13. Fix a few outdated redirects
14. New feed keywords for Ship It!
15. Remove obsolete <itunes:owner> tag from feeds
16. Delete people from CM lists when deleting locally
17. Only Weekly subscribers can submit news items
18. Adjust source title for Changelog News episodes
19. Fix bug in news item controller's new action
20. Follow redirects when fetching remote temp files
21. Differentiate between audio public URL and direct URL
22. First draft of chapter support in the onsite player
23. Don't autoplay after restoring onsite player state
24. Close the onsite player when audio reaches its end

I, @gerhard, am officially astonished 😲 with all the improvements that @jerodsanto shipped since Kaizen 7. If there is anyone that deserves to wear a Kaizen t-shirt, that would be him! 👕

As I was going through / working on the above, I started thinking that maybe we should:

• sign & verify our commits - starting with @jerodsanto, continuing with @adamstac
• improve our CI setup - PRs currently don't run tests 🤔
• start creating actual releases from these tags, starting with Kaizen 8
• double-check that our list of contributors is up-to-date

3/3. An open-ended discussion

The last improvement worth mentioning is this discussion. It was already mentioned at the top, but it's worth mentioning it again. If you want to ask questions before or after the 🎧 Kaizen 8 episode, contribute to episode references, or simply dig into more details, it's all here, in this GitHub discussions. The best part is that it's an open-ended discussion because... the show must go on 🎸

---

https://changelog.com/topic/kaizen

[adamstac]

Comment during the recording. Loving this!

[jerodsanto]

A few links for the show notes:

• op3.dev
• Backstage with John Spurlock from OP3
• Kaizen t-shirts
• Jerod's Fly/Mastodon work