Nx 19.7.0+: Custom remote caches aren't respected with relevant NX_*DB* flags enabled

[eliellis]

Current Behavior

With NX_DISABLE_DB=false NX_DB_CACHE=true, a user's own RemoteCache (or RemoteCacheV2) implementation is not respected.

Expected Behavior

With NX_DISABLE_DB=false NX_DB_CACHE=true, a user's own RemoteCache (or RemoteCacheV2) implementation is respected.

GitHub Repo

https://github.com/eliellis/nx-remote-cache-with-db-cache-repro

Steps to Reproduce

DB not enabled

1. Clone linked repo
2. Run npm ci
3. Run npm run build:without-db
4. Observe that the DummyRemoteCache prints the expected highlighted text to the terminal

DB enabled

1. Clone linked repo
2. Run npm ci
3. Run npm run build:with-db
4. DummyRemoteCache is not used, and therefore no highlighted text is printed to the terminal

Nx Report

NX   Report complete - copy this into the issue template

Node           : 21.1.0
OS             : darwin-arm64
Native Target  : aarch64-macos
npm            : 10.2.0

nx                 : 19.8.2
@nx/js             : 19.8.2
@nx/jest           : 19.8.2
@nx/linter         : 19.8.2
@nx/eslint         : 19.8.2
@nx/workspace      : 19.8.2
@nx/devkit         : 19.8.2
@nx/eslint-plugin  : 19.8.2
@nx/plugin         : 19.8.2
@nrwl/tao          : 19.8.2
typescript         : 5.5.4
---------------------------------------
Registered Plugins:
@nx/eslint/plugin

Failure Logs

N/A

Package Manager Version

NPM 10.2.0

Operating System

• macOS
• Linux
• Windows
• Other (Please specify)

Additional Information

Ultimately, I would like some guidance here about what to do and what to expect in the future.

I have been using Nx since the early days (since v8). I love how Nx has grown and all the bells and whistles that have been added since then. Project Crystal has been huge for DX. I'd like to consider myself both an early adopter and advocate for Nx to anyone who has the problems something like Nx can solve. Obviously I am also the character that likes to dig around and understand the systems I'm using a little more, hacking on things when they're there to be hacked on. In that spirit, I would love to assume good faith on the part of the Nx team. As I previously commented here, it seems like this breakage may be intentional and points in the direction that the future of Nx development is headed.

With the recent changes to the relevant code, especially with the introduction of Powerpacks, I am assuming the days of custom runners and providing your own remote cache implementations are going to be behind us at some point as Nx hackers.

On that note, if I may, I would implore the Nx team to reconsider closing off these extension points. There are lots of us out there who either cannot afford Powerpacks, are not comfortable with Nx Cloud, cannot afford Nx Cloud at the relative scale we are operating at, just plain want control over our own build processes to whatever extent we can, or some variadic combination of all of these. Many build systems adjacent to Nx, like Rush, Pants, Bazel, TurboRepo, and (soon) Moonrepo offer the ability to implement your own remote cache extension points to those who are so inclined and in whatever capacity makes the most sense for the tooling. The great thing about Nx is that we've already enjoyed this offering up until now! I genuinely hope that this can continue.

With all that said:

1. Is this breakage intentional? If not, will there be any plans to continue to support this capability? Are PRs welcome to add support for this once more?
2. Will custom runners also be sun-downed at some later point?
3. If the answer to 1. is wholly "Yes, and we will not consider PRs to support this", is it possible for there to be comms on versions of Nx where we should expect things to break completely so that we can begin planning on our own about what to do next?

Thank you for all your hard work. I appreciate everything that Nx has enabled for me personally and the JS community. All the best.

[garydex]

Check the latest blog posts from Jeff and Juri.

https://nx.dev/blog/evolving-nx
https://nx.dev/blog/introducing-nx-powerpack

Sadly your hunch is right, custom runners are going the way of the dodo in order to push people towards paying for a commercial license for Nx Powerpack. The custom caching is one of the new paid-for features.

To answer your third question, they're deprecated in v20 and removed in v21:
https://nx.dev/deprecated/custom-task-runners

I understand the reasons, but this is going to cause a lot of headaches for a lot of enterprise users.

[Jordan-Hall]

I'm going to weigh in on this. I love NX, but I think it's an absolutely terrible decision to force people to pay for something that was already possible. I'm not against the NX Power Pack—it's a great idea—but I believe users should be able to pay for just the module they need.

The pricing might suit enterprises, but it's expensive for small to medium-sized businesses that are losing functionality just so you can rightfully make money.

Victor mentioned that custom task runners are not feasible because of a shift towards Rust. However, these packages should still be accessible through nx.json. It should then be up to independent developers to figure out how to make a WASM or JS solution work with NX. Again, I'm fine with a paywall, but removing existing features only to add them behind a paywall is unacceptable.

PR for this #28170

Also for now i released a patch file https://gist.github.com/Jordan-Hall/5743969938cefc4fa462c86c96b5c3d0

Edit:
I understand there could be a concern about someone creating a competing paid service or even a sponsored solution that goes against NX Cloud and, in some future case, may even provide competition to Powerpack. I believe this could be resolved with a change in the licence, which I don't see being an issue to protect services.

I believe this is on the same level as Wordpress VS WP Engine. Removing functionality unless you pay up.

[Jordan-Hall]

@eliellis i think title should change to "Reintroduce support for community remote cache" or something like this

[eliellis]

I have had some time to read through the blog posts now, and I am having a heartbreaking time trying to craft any response that is understanding to the good of the Nx team and their livelihood and understanding of the community that has brought this level of success to our favorite build tool. In particular these two bullet-points stand in stark contrast and are difficult to take at face value given that previously-free features do seem to be getting repealed:

I do believe that without generous reframing, things are certainly being put behind a paywall that were previously free. Well-known, and free, implementations of remote caching for Nx have existed in the open for a while now, so much so, that despite the first bullet point, there is an implicit acknowledgement that existing custom remote caches will cease to work as of Nx 21:

In conjunction with the second written bullet point, I am not sure how to interpret any of this.

Victor mentioned that custom task runners are not feasible because of a shift towards Rust. However, these packages should still be accessible through nx.json. It should then be up to independent developers to figure out how to make a WASM or JS solution work with NX. Again, I'm fine with a paywall, but removing existing features only to add them behind a paywall is unacceptable.

On this point, I agree. The community has invested time and energy into building on top of Nx thus far, and if these extension points are not taken away, but merely transitioned to a state to support more performant implementations, I don't doubt the community will gladly jump in to find the right solution, being able to continue to channel their enthusiasm and support for Nx, championing the point being made in the second bullet point from above.

On a more technical note, I do find it somewhat difficult to understand how both generators and, in particular, executors can continue to exist in their current capacity if the internals of Nx transition more completely to Rust. If they are intended to continue functioning similarly (e.g. in the form of Typescript / Javascript), then I do not understand exactly how Rust plays a role in this story, but again, I do not have access or understanding of Nx's own internal roadmap, so there may be more yet to come on that front.

As it stands however, even if nothing is done, there is actually a bug in the current "transitional" implementation. The moment this PR lands, counter to the language in blog posts, remote caches will cease to function. Because of the way that this check is written, remote caches passed through to the defaultTaskRunner will not work unless you are using Nx Cloud, which as far as I can tell, is counter to the deprecation strategy outlined in the related links.

In any event, I again would implore the Nx team to re-consider some components of this transition, trying to work with the community on this–be it a license change, a head nod and some holes poked through in whatever capacity makes the most sense for performance reasons, or an outright slamming of the gavel in plain language so that we (and future readers) have crystal (no pun intended) clarity about what to expect at this time, and moving into the future with Nx.

[Kepro]

so this will be not true anymore... https://nx.dev/concepts/turbo-and-nx#nx-and-turborepo

That said, you don’t have to use Nx Cloud to get features such as remote caching and distributed task execution. We provide public APIs so you can build your own, if you'd prefer not to use Nx Cloud.

[o629625]

The change in strategy and contradiction to what was previously stated is quite unfortunate and disappointing. I would think that if the Powerpack feature set has value to NX users, they will purchase it. Regardless of whether there is an alternative way to set up a custom cache.

The Nx team should reconsider removing existing open source functionality that many teams depend on.

[Jordan-Hall]

i guess your forcing a fork of NX core and keep 1 to 1 but with the API for all. this is not good for the community

[psalv]

It feels like Nx is trying to get the advantages of having an open-source software (i.e. community development) while simultaneously building a walled garden (i.e. pay us or lose critical functionality). This feels very against the ethos of a open source project and I'm surprised someone like @JamesHenry is ok with this given his passion for the open-source community.

If Powerpack adds substantial value then people will pay for it on it's own merit, forcing people onto paid solutions is the wrong approach. Cypress took a similar approach and many people are migrating away from it as a result.

If a product doesn't provide a potential offramp it makes it very difficult to justify investing into it and potentially putting oneself into a state of vendor lock-in. If the product is good enough people will stay, and I do believe Nx has a great product but they have introduced a pretty substantial amount of risk to partnering with them with this change.

[Jesse1989pp]

Frankly been looking for any response by the team on Git and Discord, is there any so far? Timely response would really help address the community’s concerns and potentially improve perceptions imo. Coming from an announcement two weeks ago and this quite respectful thread, a sign of acknowledgment would be appreciated.

[Jordan-Hall]

Frankly been looking for any response by the team on Git and Discord, is there any so far? Timely response would really help address the community’s concerns and potentially improve perceptions imo. Coming from an announcement two weeks ago and this quite respectful thread, a sign of acknowledgment would be appreciated.

I've stayed quiet regarding the limited conversations I've had with Victor, but out of respect for the open-source community and the NX community, which has helped us all thrive, I'll share the discussion here as it relates to this issue. This is due to the lack of any 'public' response, and these matters should not be handled behind closed doors. The conversation is from my X inbox:

Victor:

Hey Jordan!

Sorry to bug you.

Could we have a quick video call to talk Nx and Powerpack?

I'd like to make sure I understand your use case and accommodate you.

Response: (yes i use chat GPT to fix my grammar and spelling hence the mistake in reponse.

Here’s a refined version of your text in British English:

Hi Victor,

Unfortunately, I won’t be available for a call as I’ve lost my voice. I’ve been a proud user of NX for years, having implemented it in government, insurance, and small businesses. Task runners allowed me to create NX Bun. I’m happy for you to charge for NX modules and would tell big companies to pay up. However, restricting a widely-used feature for custom caching and then putting it behind a paywall for NX-only is, in my personal view, wrong.

Three lines of code could solve this issue, and the community would support you. PowerPack is a great idea, but please reconsider. My thoughts are public for all to see in the PR and issue: #28170 (comment).

Regarding the move to Rust for the task runner, I think it’s a great decision. However, providing an API for a custom Rust task runner would be beneficial. There’s no need to provide anything beyond that—it’s straightforward to figure out. This would allow us to utilise runtimes that aren’t currently supported.

Many thanks
Jordan

VIctor:

This is the rationale:

As you know, we have been migrating Nx core into Rust one piece at a time, so Nx keeps getting faster with every release. The most recent piece is the cache, which will be the default in Nx 20. Based on this benchmark https://github.com/vsavkin/large-monorepo Nx 20 will about 9-19% faster.

I believe we achieved the peak performance of the NodeJS + Rust mode. The Nx overhead for a sizeable workspace is ~150-200ms. I want it to be under 50ms. The only way to achieve it is by migrating the rest of Nx core to Rust. This means we cannot load a Node.js env for core operations (including remote cache). We will still support using TypeScript for writing plugins cause those execute in the daemon, so the overhead is less important.

Customs runners are about to get deprecated but not removed. We are essentially saying that in the future, in some version of Nx, they will stop working. Tbh, the custom runners API was a bad extension api. That's why we didn't generate any runners config for years. The mechanism is too generic and doesn't let us make any assumptions about what is executed. Also, it won't work in any ergonomic way in the future Rust version. None of the competing tools had a similar extension mechanism and folks do use those tools effectively.

Currently, we load remote cache impl in JS, but it will be done differently soon (cause we won't load node.js for core operatoins). So providing a temporary public API that we are sure will deprecate doesn't feel wise. We talked about it and decided to instead be more lenient in issuing powerpack licenses to existing users who cannot afford to get one. We do this for Cloud.

One sentiment was always and still is: if you cannot afford Cloud or Powerpack, we will help you. If you are an org worth billions of dollars, affordability isn't an issue.

It goes without saying OSS projects get both Cloud and Powerpack for free.

To summarize:

• Custom runners are still around and all the ways to customize them are still around.
• If you want to use the new version of the cache, with all the advantages, and you need to share the cache via s3, you will need powerpack. If you need a license, we will provide on for you.

If you want to chat next week, I'm happy to jump on a call.

Response:

Why can't you provide an API in Rust? I'm not asking for a Node runtime; I'm happy to work with Rust. I believe the move to Rust is crucial. For instance, Moon is working on Bazel API I'm rust https://github.com/amkartashov/bazel-remote-apis-rust disclaimer after your change I started moving a project over to it.

I would personally love to see the plugin API also developed in Rust. I think it’s a bit short-sighted not to consider this.

Victor:

We will provide some APIs in Rust, but not right now. The API we will provide, however, won't let you replace the whole runner.

Replacing the whole runner is simply a bad extension API. It binds us where we can't make any changes regarding execution without breaking someone.

We put it in place as an easy fix early on. It was easier to implement "you do it yourself" API rather than think through the use cases. We did regret it soon after that. That's why we de facto removed it several years ago, where we no longer generate any runners configuration or mention runners at all.

I'm not aware of any successful project that lets you swap the task running entirely.

Response:

If you're willing to provide an API for Rust, I’d be happy to assist with it. There’s a balance to be struck; NX needs the funds, and rightly so. However, I believe that removing features just to place them behind a paywall is ethically questionable. While there is currently no Rust API, would you be open to deprecating the new hybrid approach and making it a toggle in nx.json until it's ready?

Honestly, please consider a licence change rather than placing existing features behind a paywall. I will continue to support NX in that case and promote it. No one should clone NX Cloud or similar to create a competitor.

Victor:

I genuinely appreciate your feedback. We did consider a license change but it has a huge negative impact on a lot of organizations, so we decided against it.

response:

Any chance you can leave it possible until a API is developed. Like I said I'm happy to help with making rust extendable

Victor:

The API still works, right? It's deprecated meaning we are planning to remove it, but it's currently there, so no one should be affected today or tomorrow or in the near future.

It will be hard for us to atomically land all of it. It will be a lot of changes in the period of many months.

Most importantly, Nx is a published package using semver. One can always depend on its older version while a new version is being updated/developed. Nx won't update itself. Nx isn't a REST API, for instance, that can break you without your choosing to update. Most organizations don't update right away anyway.

I thought i make this public its not something that should be handle behind close doors. It appears their wont change course its pay up or stick on a older version. API will eventually come but how long a piece of string :/ even offered to help implement anything. Im sure their plenty in here that would do the same thing!

[pete-mcwilliams]

cannot afford Nx Cloud at the relative scale we are operating at

This is the boat we are in, there seems to be a missing tier between hobby and pro. We would be happy to spend a little money each year, but we are a small organisation with 4 devs and < 100k operations... we do not need or can not justify the cost of $250/month for something scaled for 3 times our size.

[Jesse1989pp]

Appreciate the insights @Jordan-Hall, got my personal views on some of the remarks. Mainly on deprecation vs the communicated removal of Custom Task Runners on V21 without an OSS alternative. I will refrain from joining this private discussion until the community is properly addressed. Baseline, I am all for supporting the team and product of NX. Clearly it deserves to be a healthy product for all people involved. Frankly everything is salvable pretty easily, but confidence falls and stands with timely and transparent communication.

[vsavkin]

Hi folks,

I'm sorry for the delayed response. Many people on the core team have been off due to our annual conference and today's Canadian public holiday. Having said that, I should have written something last week. I apologize.

This is the relevant information from me and the Core team:
#28434

I'm going to close this issue.

Thank you!

[github-actions]

This issue has been closed for more than 30 days. If this issue is still occuring, please open a new issue with more recent context.