New package: TechnicalCompute v1.0.0

[JuliaRegistrator]

• Registering package: TechnicalCompute
• Repository: https://github.com/dgleich/TechnicalCompute.jl
• Created by: @dgleich
• Version: v1.0.0
• Commit: 0a19c4accb56fce081cc0678672c2cdabed1baea
• Reviewed by: @dgleich
• Reference: dgleich/TechnicalCompute.jl@0a19c4a#commitcomment-149043307
• Description: A simple metapackage for Julia to be used for technical computing

[gdalle]

Hi, congrats on this cool initiative. However, I'm not sure whether this is suited to a wide variety of users.

My first issue is that most people will only need to install an infinitesimal fraction of the packages you import, which means they would pay a disproportionate loading time. Additionally, using TechnicalCompute would prevent them from knowing where all of this imported stuff actually comes from. I happen to think that adding the dependencies ourselves is a good way to keep project complexity reasonable and understand the moving pieces.

My second issue is that this package looks a lot like a personal utility. It has a lot of personal overrides and options, and even contains type piracy. So I would not recommend registering it, even though it may be useful to your personal workflows. See the package registration guidelines for details.

I'm not well-versed in other languages, but I assume the goal might have been to provide a turnkey environment to students, similar to what might be found in more integrated languages?

[github-actions]

Hello, I am an automated registration bot. I help manage the registration process by checking your registration against a set of AutoMerge guidelines. If all these guidelines are met, this pull request will be merged automatically, completing your registration. It is strongly recommended to follow the guidelines, since otherwise the pull request needs to be manually reviewed and merged by a human.

1. New package registration

Please make sure that you have read the package naming guidelines.

2. AutoMerge Guidelines which are not met ❌

• No licenses detected in the package's top-level folder. An OSI-approved license is required.

3. Needs action: here's what to do next

1. Please try to update your package to conform to these guidelines. The General registry's README has an FAQ that can help figure out how to do so.
2. After you have fixed the AutoMerge issues, simply retrigger Registrator, the same way you did in the initial registration. This will automatically update this pull request. You do not need to change the version number in your Project.toml file (unless the AutoMerge issue is that you skipped a version number).

If you need help fixing the AutoMerge issues, or want your pull request to be manually merged instead, please post a comment explaining what you need help with or why you would like this pull request to be manually merged. Then, send a message to the #pkg-registration channel in the public Julia Slack for better visibility.

4. To pause or stop registration

If you want to prevent this pull request from being auto-merged, simply leave a comment. If you want to post a comment without blocking auto-merging, you must include the text [noblock] in your comment.

Tip: You can edit blocking comments to add [noblock] in order to unblock auto-merging.

[goerz]

even contains type piracy

I would absolutely veto the registration of any package that contains type piracy, i.e., defining methods for functions you don't own for types you don't own. Unfortunately, "own" doesn't always map cleanly to package boundaries, so this can only be detected by manual code review, like in this case. But in my view, it makes that package ineligible for registration (and is a major foot-gun even for "private" use).

Type piracy is how you get the kind of bugs where you get plausible but wrong results and end up having to retract papers, or worse. Avoid it at all costs.

[goerz]

I haven't looked at how "serious" any potential type piracy in this package might be, but even beyond that, I'm inclined to agree with @gdalle that this type of meta-package is more useful in a closed group or context. It is better to not have meta-packages registered, and pull them in manually or via a local registry. After all, this kind of curation is extremely subjective. It would hardly be the catch-all curated collection of packages that would work for everyone doing technical computing in Julia (which is every Julia user)

[dgleich]

The origin of this package was after I talked/emailed with Viral Shah, Alan Edelman, and Chris Rackauckas about 5 years ago about how Julia has been moving so much stuff out of the core language and into packages (which I agree with for the core language), and how this made it harder to teach. The vision for the package emerged out of those discussions is indeed to have something turnkey that supports much of what you'd probably see / need in undergrad / early grad classes. (And yes, the commits really do go that far back; might have done this earlier but, pandemic, etc.)

So this isn't designed as a personal package. I really really hope it'd be useful to a wider group of people -- especially educators! Although not having it registered as package would make that more challenging. As far as I know, there aren't other huge meta packages like it. I'm hoping to put together a suite of demos that really show off functionality across the ecosystem.

The "overrides" are mostly just to do method integration among multiple packages (update! is frequently used, but easy to merge...) or to pick one method from a package in the case that there's a clear "winner". (i.e. JuMP and Symbolics both export @variables, but JuMP never uses it in all the documentation, whereas Symbolics uses it extensively... hence that one went to Symbolics; also JuMP.@variables isn't that bad as a clarifier for that situation.) I can only think of a few methods where I couldn't get a successful merge that made me unhappy (Makie.Axis and AxisArrays.Axis was the biggest...).

I'm hoping we got the mix of packages roughly right to start getting feedback and working to improve. My hope is to see if we can do 'semester-aligned' releases to support teaching and bring new stuff in.

I'm actually amazed this many packages can work together without broader / more fundamental method clashes!

I can look at fixing the type piracy if this is truly a deal breaker -- it's largely just omissions/bugs in other packages. (e.g. JuliaMath/DoubleFloats.jl#208, ITensor/ITensors.jl#1581).

[goerz]

Okay… this seems like something that should be manually merged by someone like @ViralBShah or @ChrisRackauckas then, if they agree with the reasoning.

Of course, the license issue has to be fixed as well.

Personally, I'm still a little skeptical that these types of curated meta-packages are a good idea to have in the registry. But if they're sufficiently well-maintained, and have buy-in from some of the core devs or other big players, I could change my mind on that. It might also be worth to "pre-announce" this on Discourse, to have a discussion with the wider community.

[ViralBShah]

Yes, @dgleich and I spoke about this a while ago! Since then, the package ecosystem has matured tremendously. I don't personally see a downside to such a package - but perhaps we can change the name to something a bit more instructive. TechnicalComputeEdu or something (not very inspired on my part).

@dgleich Do you think the package benefits from being in General, or would students be ok directly adding it by the full URL?

@mgkuhn brought up a similar sentiment here: JuliaLang/juliaup#1083, and I think this would address many of the things he brought up.

[gdalle]

I think renaming this to something like TechnicalComputeForStudents would alleviate most of my concerns indeed.

[ChrisRackauckas]

The "overrides" are mostly just to do method integration among multiple packages (update! is frequently used, but easy to merge...) or to pick one method from a package in the case that there's a clear "winner". (i.e. JuMP and Symbolics both export @variables, but JuMP never uses it in all the documentation, whereas Symbolics uses it extensively... hence that one went to Symbolics; also JuMP.@variables isn't that bad as a clarifier for that situation.) I can only think of a few methods where I couldn't get a successful merge that made me unhappy (Makie.Axis and AxisArrays.Axis was the biggest...).

I really think this should be worked out with more ecosystem coordination, not a package on top. For example, we have CommonSolve.jl that defines the behavior of solve in a reusable fashion: those other packages should really coordinate to extend the same function under the same meaning given it's such a common verb. It seems most of the type piracy issues are fixed when brought up (JuliaMath/DoubleFloats.jl#208, ITensor/ITensors.jl#1581 seem to be complete?), so I think most of what is solved in the package src is best solved via ecosystem coordination.

The origin of this package was after I talked/emailed with Viral Shah, Alan Edelman, and Chris Rackauckas about 5 years ago about how Julia has been moving so much stuff out of the core language and into packages (which I agree with for the core language), and how this made it harder to teach. The vision for the package emerged out of those discussions is indeed to have something turnkey that supports much of what you'd probably see / need in undergrad / early grad classes. (And yes, the commits really do go that far back; might have done this earlier but, pandemic, etc.)

Multiple things here. I think ecosystem coordination issues should be solved directly in the packages, as mentioned above. But what I was saying is valuable is cross-package documentation. The reason is because for example a neural ODE training is not only using DiffEqFlux.jl, but it's a combination of SciMLSensitivity.jl, DifferentialEquations.jl, Optimization.jl, etc. Thus packages being diligent in documenting only their own functionality does not hit real-world workflows which are generally a multi-package endeavor. Someone needs to package up and document this higher level.

SciML now has a documentation which merges all of the docs together and has a highest level set of examples which covers such workflows https://docs.sciml.ai/Overview/stable/ . To me, this is the valuable part of bringing stuff together, but it is a massive undertaking in terms of maintenance. You'd likely need to have a few people really dedicated to the project and buy-in from major maintainers to keep such a thing alive with the scale of packages you're setup for. Right now, TechnicalCompute.jl does not have a documentation, and I think that's most of the work and most of the value. So I'm not sure of its current value with this piece missing, and one of the major changes has been for these types of high level pieces to now somewhat exist.

As for a core batteries included experience, I think many things have shifted around due to package images. There is still a major need for version coordination, but that needs to be done with package maintainers involved otherwise it's going to be a nightmare to keep up. But I think the real solution here is the features feature JuliaLang/julia#55516 which should be coming in the near future. With this kind of thing, we plan to re-bundle SciML packages back down to a smaller set and use features as a way to separate compile and allow for quicker load times by default. With these kinds of changes already being lined up, I think TechnicalCompute.jl is likely to not be so useful, since for example SciML.jl should be almost the same thing except setup with separate procompilation, pooled issues, etc.

Now last thing is I think such a pooled system would be very useful if it had some automation in terms of version bumping and binary building. If this was "Scientific Julia" system images with a website that was auto-building an updating system image that has a clean set of all major packages built in, I would be on-board for that. That is something that I have been considering maintaining with SciML since it would be easier to globally document. though I am mostly waiting on the features feature now which would effectively make that a simpler design.

So all this is to say, I'm not sure of the value of the current form in 2024. To me, it's missing the hard parts that I think would be required for adoption:

1. A high-level comprehensive documentation of cross-package workflows
2. Automated system image building and binary distribution as a weekly-updatable batteries included Julia download (which would be nice to setup to juliaup)

What I think it currently has a use for is as a test of ecosystem coordination. But I think it needs a bit more CI involved in order to really fill this purpose. It would be nice if it auto-ran daily tests, highlighted all cross-package inconsistencies, and pinged the right people to start discussions. But it's missing the devops component to this.

Overall, the current state of the package I think is very useful for highlighting things that need to be changed in the ecosystem today, but I don't see the devops elements for it to really sustain itself as a multi-dev tool, nor do I see the elements for full adoption.

That said, I'm very lenient on the registration process. So I'd say, if you pick a more specific name for it TechnicalComputeForStudents then I'd say, let's see what happens.

[dgleich]

Thanks for all the feedback!! (@ChrisRackauckas -- lots of good ideas in that note, really appreciate it; will keep feedback for another channel and/or when things come up.)

So it sounds like this is 'probably okay' but we'll need to change the name.

I'm trying to avoid names that put down students or 'special case' things for them, and something that might have some appeal to a broader group as well.

Would

StarterPackages.jl

be acceptable?

(I have a longer list of possibilities ... just trying to focus discussion.)

[gdalle]

I would also avoid that because it sounds like the set of starter packages everyone should have. I'd like to add one word which makes its opinionated aspect more salient. Maybe TechnicalComputeBundle.jl? TechnicalComputeSelection.jl?

[ViralBShah]

I like TechnicalComputingBundle.jl

[dgleich]

Okay, I'll go with TechnicalComputeBundle.jl, I liked that one too! Just to be clear on next steps. I rename Repo and then send a new register request (after fixing the license, Project, Readme, etc.)?