scimax 3.0?

[jkitchin]

This is not so much an issue as a place to spark some discussion. I am thinking about making a new version of scimax with a few goals designed to make it easier for new people to try it out. This is a somewhat long-term plan, my aim is to have this ready for deployment in Aug 2020 so I can use it for teaching in Fall 2020.

1. Make cross-platform installation simpler

The main idea here is to create a scimax python package that installs scimax with pip install --user scimax. This will figure out what platform the user is on, and install scimax in the user site-packages directory, and create some simple commands to start scimax. I am not committed to this approach but it seems like the simplest from a user point of view. Experts will still be able to install scimax any way/where they want.

2. Create a scimax github institution

This is mostly to separate the scimax repositories into one place for organization. It might make it more modular. For example, I would probably feel better about making a repo for latex packages for scientific publishing.

3. update to Emacs 26+

I think I want to migrate away from ob-ipython towards emacs-jupyter. This will require emacs 26+. I need to make sure this works smoothly on Windows though, as it requires building an emacs module.

4. Provide a curated initial set of packages

The initial installation of scimax, especially on windows, is often problematic with issues on Melpa, or gnu elpa. This would make the initial installation simpler and more robust.

Any other ideas? Post them as comments below, and I will add to/edit this issue to integrate them. Thanks!

[domingoUnican]

Inside the new latex repo, it would be nice to have some examples for ox-manuscript, at least
for some of the templates, to see how they look.

There are two ways of starting these templates: ox-manuscript-new-ivy and ox-manuscript-new-helm. Is there a prefered way?

[jkitchin]

You might be interested in http://catalysis-preprint-archive.github.io/. It is an experiment I worked on many years ago. Most of the templates came from articles listed at this site, and in theory most of them should have something called manuscript.org in them that became the paper. Links to these could at least be documented in the new repo.

I prefer the ivy way of starting new manuscripts. The helm one came first, and later I mostly transitioned everything to ivy.

[singletS]

I'm excited to hear of these plans! Modularizing emacs packages and integrating emacs-jupyter are particular interesting to me for my use cases (mostly just running code and authoring jupyter notebooks in emacs).

One minor suggestion would be to consider conda packaging in addition to pip. I cannot make claims about emacs enviroments, but conda has been a huge help in getting my students up and running complicated modeling codes on different OSs. This includes compiled binaries not related to python.

[jkitchin]

I thought about conda too, especially since I prefer the Anaconda python distribution. There is an emacs conda package at https://anaconda.org/conda-forge/emacs for Mac and Linux. If it is as easy as conda install -c conda-forge emacs for Windows too, I would be all in!

[tgbugs]

Big thumbs up. Happy to contribute to the python module and packaging in any way.

[singletS]

Another idea/suggestion: Have ox-ipynb be a derived export backend so it understands more of the org-mode built-in structures. Because of ipynb limitations there will always be differences, but several of the export-related issues have involved a missing org mode feature or unexpected output for an ipynb compared to HTML or LaTeX output.

[jkitchin]

It would be helpful to post these as issues on the ox-ipynb repo. It is such a different exporter, I am not sure there is a way to make it like html or latex.

[shrysr]

One difference in philosophy I've seen with starter kits like Prelude versus Scimax is that they tend to offer options that the user can customise, and do not 'impose' much on the user. Like the leuven theme being default with Scimax, and customisations of the source block colors. It would be convenient if user.el example had almost all the scimax specific customisations with options of loading, or not at all. i.e default scimax is as close to default emacs and everything else is an option.

[jkitchin]

That is a fair observation. Scimax started as a way to get my students quickly up to speed with a tool they can do their research and work with, with scientific publishing at the core of that, in a way that is compatible with what they need and I need from them. The large majority of these students do not know what they need to accomplish that, and I don't want them to spend the years I have spent figuring it out. Thus, the imposition of my opinions on what they need so they can learn from me how to do it. With this, they can on day 1 use scimax the way I do. I think of scimax as more like an app in this sense.

For those who do know what they want, it is not necessary to load all of scimax. You can start with vanilla emacs, and just add scimax to a load-path, not load init.el, and load the parts you want. That is mostly true, although there are a lot of things in scimax-org.el that might feel like impositions to just load them all. Some of these could be split out into smaller files to limit that. If there were strong feelings (which mostly have not been voiced so far), I would be happy to talk about what to move out, and make optional. This would make sense especially for the monkey-patched org-code that does things like color the src-blocks.

In its current setup, preload.el is the place to put options for loading, since this would be run before anything else is loaded. I am not too excited by this idea, as it means defining variables for each feature, then checking those variables before loading them. This would be ok for the features that are the biggest impositions, but tedious for all of them. Maybe an expert-mode-variable could be defined that doesn't load any packages and goes straight to user/user.el?

Alternatively, packages.el could be renamed to user/user.example where a user would have to rename it to user.el to get any customization. That seems like one step more than needed to me, but the install script could do this automatically. The downside of this, to me, is that this file (user/user.el) is not part of the scimax repo so it is not possible for me to update their installations. I see that this too might be unwelcome by some, but it is a necessary feature for me to support my students.

Thanks for the observation, it got me to think a lot about what is in scimax and why, and how it might be reorganized in v3 to accommodate different use cases.

[shrysr]

I see your point about treating it as an app, and it makes sense. No doubt - even as it is - Scimax rocks, and it's unlikely I can accurately estimate the time it has saved me since several years. Thank you! I do live in Scimax more or less. I can't think of other things I find imposing, other than the theme per se - but I guess for me Scimax is Emacs, and I would not be able to judge very well. I guess the comment also stemmed from a sense that native Emacs and Prelude (based on brief explorations) feel snappy(er) in terms of loading as well as response, as well a desire to have a lean and mean config, i.e only the packages I need.

Ha! It's been some years since I was a student - but w.r.t Emacs - I'm not very different from those students who don't completely know what they need either, and it has been a journey. I still don't know or leverage all of scimax's nifty customisations. In fact - I probably did not find the Scimax help file till after using Scimax for quite some time, and started using even the Scimax hydra rather late (which I'm yet to fully explore).

I can understand the tedious nature of defining variables for each feature. It doesn't seem to be the optimal long term solution either, though I guess the other starter kits appear to follow this. For eg I liked the 'doom doctor' feature during a brief exploration of Doom Emacs.

The expert-mode-variable actually sounds good. I think the expert-mode-variable makes sense in one additional way - I get to actually learn the features of scimax step by step and systematically understand what I need and do not, and not to mention what exists. I was wondering whether an Org mode based config could help here. Sort of expand scimax.org into a user-example.org, and use it to provide groups of features, or even config snippets that can optionally tangled?

FWIW: I've been meaning to convey since some time that I'm happy to help in any way I can, especially documentation and config examples. If there are specific areas you have in mind - it would guide me if you tag them 'good first issue' or something suitable and I could work on them.

[shrysr]

conda install -c conda-forge emacs

I actually use this as a shortcut to install newer versions of Emacs on Linux, but I guess one issue is that when the virtual env is switched - Emacs is no longer available to be called from shell?

[malcook]

FWIW: apparently there is a new (still undocumented) conda option to [support stacking environments](conda/conda#5159) that might solve this conundrum. I have not tried it.

[jkitchin]

I see your point about treating it as an app, and it makes sense. No doubt - even as it is - Scimax rocks, and it's unlikely I can accurately estimate the time it has saved me since several years. Thank you! I do live in Scimax more or less. I can't think of other things I find imposing, other than the theme per se - but I guess for me Scimax is Emacs, and I would not be able to judge very well. I guess the comment also stemmed from a sense that native Emacs and Prelude (based on brief explorations) feel snappy(er) in terms of loading as well as response, as well a desire to have a lean and mean config, i.e only the packages I need.

The theme is now configurable. It is still leuven by default. You can define a theme in user/preload.el with (setq scimax-theme 'some-theme-symbol), or set it to nil to have no theme. (c5cec09).

It is almost certain they load faster. org-ref is the main culprit here, and it is mostly because of pdf-tools. I leave Emacs open for so long, I don't worry about the startup time, but it would be great if we found a way to make it load faster.

Ha! It's been some years since I was a student - but w.r.t Emacs - I'm not very different from those students who don't completely know what they need either, and it has been a journey. I still don't know or leverage all of scimax's nifty customisations. In fact - I probably did not find the Scimax help file till after using Scimax for quite some time, and started using even the Scimax hydra rather late (which I'm yet to fully explore).

You aren't the only one, I have occasionally found myself reinventing things I already wrote in scimax and forgot about! Some things are just ideas to try out.

I can understand the tedious nature of defining variables for each feature. It doesn't seem to be the optimal long term solution either, though I guess the other starter kits appear to follow this. For eg I liked the 'doom doctor' feature during a brief exploration of Doom Emacs.

The expert-mode-variable actually sounds good. I think the expert-mode-variable makes sense in one additional way - I get to actually learn the features of scimax step by step and systematically understand what I need and do not, and not to mention what exists. I was wondering whether an Org mode based config could help here. Sort of expand scimax.org into a user-example.org, and use it to provide groups of features, or even config snippets that can optionally tangled?

I think this wouldn't require an expert-mode variable. It sounds like more of an org-based config that starts in init.el and goes through it.

FWIW: I've been meaning to convey since some time that I'm happy to help in any way I can, especially documentation and config examples. If there are specific areas you have in mind - it would guide me if you tag them 'good first issue' or something suitable and I could work on them.

I will give this some thought.

[shrysr]

I think this wouldn't require an expert-mode variable. It sounds like more of an org-based config that starts in init.el and goes through it.

Yes. Org mode based config would not need the expert-variable. I guess there are advantages with separate .el files - but somehow using an Org mode based config has been a lot easier to manage for my custom config, and write notes in it. With a tangle on save snippet - I don't have an additional wait time for a tangle during start-up.

update to Emacs 26+

Wait - am I supposed to be using Scimax with an older version of Emacs??

[jkitchin]

no. 26+ is nearly required these days! But the older Windows version installs with an Emacs25 I think.

[hummuscience]

1. Making scimax installable with pip is a great advantage. I recently introduced some of my colleagues to emacs and scimax and it is still a relatively steep learning curve. Anything that would make that experience easier is welcome.

2. I like the idea of making scimax more modular.

I would like to pitch something. It has to do with the thought that scimax can be used to create self-contained publications (or lab notebooks/thesis).

I wonder how one could turn a .org file that contains everything for a paper into one neat package that can be installed on another researchers computer and easily executable. Maybe using docker or singularity? I wonder what is your approach to this and whether it would be something to keep in mind, specially when competing with current solutions for jupyter notebooks etc.

[tgbugs]

In theory you could just write all the instructions and code blocks for bootstrapping the environment for the org file in the file itself that process could use containers, or any number of other possible mechanisms for replicating an environment. I use a brute for approach in setup.org to get a working scimax install. It is not a good approach because it is error prone and depends too much on the user knowing how to read and run code blocks. However, it works on windows, linux, and osx -- if they follow the instructions. In another case I have a self publishing org file that tangles files into the git commit hooks folder along with a publishing config so that pdf publishing runs every time I create a commit. I don't have that one in a public git repo yet (it is for my thesis), but have been working on a stripped down version that captures the essence of the workflow.

Portable environment bootstrapping is quite difficult if you really want it to be reproducible, not just containerized. But a source block that is a dockerfile for a minimal environment is a reasonable start.

[malcook]

@tgbugs - how can I stay tuned to your efforts on the "stripped down version" - sounds great!

[tgbugs]

@malcook I will drop a note in this thread when I create the repo.

[tgbugs]

@malcook https://github.com/tgbugs/orgstrap It only took about 10 months, but I think it is more or less ready, and hopefully will be on melpa within the next week. Also potentially of interest in this thread is the bootstrapping to emacs section given the continued struggle to simplify the process of getting colleagues up and running with org-mode.

[federkamm]

A note on Emacs on Windows:

I started with native Emacs on Windows (Emax64 worked best for me) but there are lots of minor issues that made me change to Emacs in WLS (Windows Subsystem for Linux, just enable "Windows Subsystem for Linux" in "Control Panel -> Programs and Features -> Turn Windows Features on or off" and pick your favorite Linux distribution in the Microsoft Store, mine is Debian). I use MobaXTerm as Terminal and X11-Server. This setup works like charm for me, already for a couple of years. Today, I built Emacs with support for modules (Debian doesn't have it by default) to run emacs-jupyter and successfully connected to Wolfram Language. See https://gist.github.com/federkamm/c268c7268e4a1e9d85ef4866a6afc6ea for compiling instructions.

The downside from WSL is that the GPU is not available to Linux, making Machine Learning no fun (this will most likely change in the next couple of years, but that's how it is for now). However, I didn't try yet, but I suppose it is possible to run native Python and Jupyter on Windows and use WSL Emacs to connect to Windows Jupyter and use the GPU.

EDIT: Just noticed, WSL Emacs can simply run Windows binaries, like "jupyter.exe" (e.g. from M-!, when in path). This makes it probably even easier to link WSL Emacs to Windows Python/Jupyter.

[jkitchin]

@federkamm thanks for the note. That sounds pretty promising. I think packaging is probably critical to the future success of scimax, it seems to be a stumbling block with new people, especially given the ease of Google docs and colab.

[shervinsafavi]

Having a proper ipython/jupyter-shell will help quite a lot for data analysis workflows and debugging. I've tried various solution by connecting to jupyter kernel being used in a given org file with term/shell/eshell, but all have some issues. I guess migrating away from ob-ipython towards emacs-jupyter might provide a solution already (because emacs-jupyter give you an interactive shell, when you initialize a kernel), but not sure.

If you decided to incorporate an ipython/jupyter-shell in scimax 3.0, I'd be happy to share my experience (and help).

[jkitchin]

In the current scimax you can get to an interactive shell already, with the existing ob-ipython setup. You use the command org-babel-switch-to-session or C-c C-v C-z or from the menu, Org -> ob-ipython -> Switch to repl. Does this shell not do something you want?

I do plan to eventually switch over to emacs-jupyter, hopefully this summer, at least for myself. The pandemic has really cut down on development time though, so who knows.

[shervinsafavi]

Actually I was not aware of org-babel-switch-to-session! I just tried it, and it works quite well :) The only thing I noticed is different from emacs-jupyter shell is you could have inline images (the figure output of an ipython cell) in the shell (although I admit is not something essential at all - it's just nice to have it sometimes).

I was about to push some minor suggestions for the documentation of the org-show, I just thought it is also worth mentioning org-babel-switch-to-session in the manual, for the ipython part. I'll add it somewhere appropriate and feel free to keep it if you also find it helpful.

By the way, thanks for all the effort you have put in creating scimax! I start working with emacs relatively recently and scimax saved a lot of time and gave a lot of joy :)

[jkitchin]

inline images in the shell are fancy. when i make images from org-babel-switch-to-session they pop up externally. I don't know what magic emacs-jupyter does there.

the repl switch is mentioned a few times in https://raw.githubusercontent.com/jkitchin/scimax/master/scimax-ipython.org. it would be easy to miss there though.

Glad to hear you are enjoying scimax!

[dcstang]

Hi Kitchin, just found your package when looking up how to write-up scientific text for my grad course. Tutorial videos were really inspiring :)

Just an open question here to anyone: can I install this on top of doom-emacs?
I'm new to emacs but as I understand it... doom and scimax are curated 'distributions' of emacs. Scimax more so suited for the whole process of scientific writing.
I'm slightly worried if I will break anything in doom if I install scimax now.

[tgbugs]

If you are using doom you would need to port a number of the packages from scimax to work with Doom's package manager (straight) compile and configuration system. scimax includes a number of packages that are not required for the scientific writing portion, configuring the relevant set to work on doom might be tractable since I think it is most org-ref and dependencies.

[jkitchin]

you can also just clone scimax, add it to your load-path, and then just manually load/require what you want from it.

[mricke]

I have two questions.

On the topic of making scimax more modular, would it make sense to have org-db as a separate package? Lots of existing org extensions need to be reworked to work with either scimax's org-db and org-roam's database.

Also, when it comes to modifying files outside of emacs, could detection of file changes be (optionally) managed by git or something similar?

[jkitchin]

I am not aware of any extensions that are either using org-db, or incompatible with it. Do you have some examples?

I don't think there is a reliable solution to modifying files outside of emacs. The org-files could belong to many different git repos. Practically, this hasn't been a real issue for me.

one could run a function on a timer that checks all the entries of the database for consistency, at some time it is not inconvenient, or maybe it could run asynchrounously to not block emacs.

You could try using filewatch (https://www.gnu.org/software/emacs/manual/html_node/elisp/File-Notifications.html). This probably works when Emacs is open, but not when closed. I am not sure how this would scale to thousands of files though.

You could add an external watcher that makes a changed file list perhaps. I also don't know how well this would scale, or how to separate external changes from internal changes.

[mricke]

I am not aware of any extensions that are either using org-db, or incompatible with it.

Neither am I but it seems like there is a need for a central database for indexing large sets of org files. It just seems like a feature that should be built into org itself and the name kind of implies that it is the database for org, except not really because it comes as part of a larger package. Would you mind if org-db was repurposed into a separate package by the same name?

You could add an external watcher that makes a changed file list perhaps. I also don't know how well this would scale, or how to separate external changes from internal changes.

The easiest way I can think of separating internal from external changes is by creating a hash of the file after saving from within emacs and adding that hash to the database, consult filewatcher when synchronizing files, do stuff.

I'd still prefer git though, if I'm completely honest. No background service required, I've tested it with 10000 org files and after modifying ~9700 of them (by substituting a word), git status returns a list of modified files (tracked and unstaged, in this instance) in around 0.3s. Not the most exhaustive test, I know. Git commit hashes (or maybe tree hashes too?) can be cached to verify commits made by org-db. The biggest downside to this approach is that being managed by git will increase the directory size but that can be somewhat negated by flattening the git tree. Maybe that option is too storage heavy...

[jkitchin]

I am not interested at this time in separating org-db out to a separate package. It works well enough for me, and these days is stable enough, but I don't have time to support it differently than it is. It has all the problems I mentioned though, so I don't think this is the right solution for everyone else. I don't think this would ever get integrated into org.

Maybe soon there will be integrated sqlite in emacs, and that will be an opportunity to do this very differently.

An md5 hash and last-updated is already stored in org-db.

As I said org-db works across many git repos (for me at least). There is no single git repo, and not every org file is in a git repo either. Storage is not the issue imo.

[wickedjargon]

It'd be nice if this starter kit included a dark theme version leuven (an exact inversion of its colors would be great).
using:

(setq scimax-theme 'some-dark-theme)

causes the codeblocks in an .org file to remain light.

[jkitchin]

I don't know a way to couple the code block colors to the themes.

Those colors are set here:

(setq  org-src-block-faces '(("emacs-lisp" (:background "LightCyan1" :extend t))
			     ("sh" (:background "gray90" :extend t))
			     ("python" (:background "DarkSeaGreen1" :extend t))
			     ("ipython" (:background "thistle1" :extend t))
			     ("jupyter-python" (:background "thistle1" :extend t))))

I guess if you don't want special colors so they match your theme, you could (setq org-src-block-faces nil)

[wickedjargon]

thanks.

Shall I make a pull request to include a function that toggles between leuven and leuven-dark in scimax?

[jkitchin]

Sure, I would take a look at that. Maybe it could go in scimax-utils.el for now?