using org-ql to edit org document?

[pcompassion]

I am trying to carry-over todo items, (similar to what org-journal carry over does)

I used org-ql to do query what I want to carry over.

Here's the code.

I had some perplexing bug on putting

            (message (format "title: %s" (org-element-property :title (org-element-at-point))))

only when cursor at source buffer is at the end of the end.
Caused my code to not correctly carry-over.

I was suspecting many things, but it turned out

(org-element-at-point) is buggy with non-ascii character (in my case Korean, I guess I'll have to put some encoding hint at the top of file or somewhere)

Since I didn't know what the cause was, I also suspected it might be related to org-ql (specially how it caches..)
But it was not.

@alphapapa Thanks for saying to ask for help. While I prepared the question, I found the bug.
I ask a general question, since org-ql seems to be supporting "query" not "replace or edit" I was wondering if I was misusing library or if there are things I need to know when using it for editing. (maybe too general question.. but I 'll be glad if I can get any hints)

(cl-defun ek/ob-collect-nodes-test
    (&key (source-query (error "source-query is mandatory"))
          source-buffer
          (source-spec #'ek/ob-get-sources)
          target-buffer
          )

  (let (
        move-data-list
        (source-files (if source-buffer
                          (list source-buffer)
                        (if (functionp source-spec)
                            (funcall source-spec)
                          source-spec)))
        )
    (org-ql-select source-files source-query
      :action (lambda ()

                ;; (save-excursion
                ;;   (message (format "cursor1: %s" (point)))
                ;;   (message (format "title: %s" (org-element-property :title (org-element-at-point))))
                ;;   (message (format "cursor2: %s" (point))))
                (message (format "title: %s" (org-element-property :title (org-element-at-point))))
                (let ((start-point (point))
                      (file-name (buffer-file-name))
                      (element (org-element-at-point))
                      end-point
                      move-data
                      )
                  (org-end-of-subtree t t)
                  (backward-char 1)
                  (move-end-of-line nil)
                  (setq end-point (point))
                  (setq move-data
                        (list :start-point start-point
                              :file-name file-name
                              :element element
                              :end-point end-point
                              )
                        )
                  (push move-data move-data-list)
                  )
                )
      )

    (let ((above))
      (dolist (move-data move-data-list)

        (let ((source-buffer (get-file-buffer (plist-get move-data :file-name))))
          (ek/ob-move-node-test
           :source-buffer source-buffer
           :source-element (plist-get move-data :element)
           :source-start (plist-get move-data :start-point)
           :source-end (plist-get move-data :end-point)
           :target-buffer target-buffer
           :above above
           ))
        (setq above t)
        ))
    ))


(defun ek/org-make-top-level (content)
  (with-temp-buffer

    (org-mode)
    (erase-buffer)
    (insert content)
    (unless (org-before-first-heading-p)
      (org-back-to-heading))
    
    (while (and (org-at-heading-p) (> (org-current-level) 1))
      (org-promote-subtree)
      )
    (buffer-substring-no-properties (point-min) (point-max))
    )
  )

(cl-defun ek/ob-move-node-test
    (&key
     source-buffer
     source-element
     source-start
     source-end
     target-buffer
     above
     )

  (let*
      ((source-content (ek/org-make-top-level
                        (with-current-buffer source-buffer
                          (buffer-substring-no-properties source-start source-end)))))
    (with-current-buffer target-buffer
      (setq target-marker (point-marker)))

    (when (and target-marker (not (eq (marker-buffer target-marker) source-buffer)))
      (with-current-buffer (marker-buffer target-marker)
        (goto-char (marker-position target-marker))
        (cond
         ((org-before-first-heading-p) nil)
         (t
          (org-back-to-heading)
          ))
        (when above
          (unless (org-before-first-heading-p)
            (org-backward-heading-same-level 1))
          )
        (org-insert-todo-heading nil t)

        (kill-whole-line 0)
        (move-beginning-of-line nil)
        (insert source-content)

        ;; Now remove the node from the source buffer
        (with-current-buffer source-buffer
          (delete-region source-start source-end)))))

  )

(defun ek/ob-collect-nodes-to-buffer (&optional buffer)

  (interactive)

  (let (
        (source-query '(and
                        (or (todo "NEXT")
                            (and
                             (todo "TODO")
                             (deadline)
                             )
                            )
                        (not (done))

                        ))
        )

    (ek/ob-collect-nodes-test
     :source-spec "~/Dropbox/notes/roam/daily/test.org"
     :source-query source-query
     :target-buffer (or buffer (current-buffer))
     )

    )
  )

and the test org doc is

#+title: Test

* TODO finance :@a_trading:


** TODO question

*** TODO what is 한글 ?
DEADLINE: <2023-09-19 Tue 04:00>

*** TODO why is saving
DEADLINE: <2023-09-19 Tue 12:48>

* TODO emacs
DEADLINE: <2023-09-20 Wed 15:30>
- [X] scratch capture template

[alphapapa]

As you said, org-ql is a query/search library. What your code does with the results is up to you.

If you need a library to help with editing, maybe this can help: https://github.com/ndwarshuis/org-ml

(org-element-at-point) is buggy with non-ascii character (in my case Korean, I guess I'll have to put some encoding hint at the top of file or somewhere)

If you're sure that's the problem, please report a bug to the Org mailing list so it can be fixed.

[pcompassion]

@alphapapa thanks for response. I was not seeing org-ml-update , so i can use it as to edit the node.

If you're sure that's the problem, please report a bug to the Org mailing list so it can be fixed.

I'll do if I'm convinced, since encoding needs to be told explicitly, i might need to set somewhere default encoding for org file is utf-8 or something. (so it might be my lack of system setup)

Thank you for support

I can see using org-ql to selecting the nodes to edit, and modify using org-ml on selected nodes.

just in my short experience,

I think org-ql can be useful when updating because it provides tool to select the nodes to edit.
and here are some difficulties I had

• i'll have to first collect the nodes to modify then do update separately.
  (Especially in reverse order because modifying above content changes the below content position)
• I need to store (node or position, buffer-file-name), since buffer itself might get destroyed after org-ql call
• apply the changes in reverse order

If I happen to have more needs to update using org-ql (and org-ml)
I'd probabily make utility function that automates the above pattern.

• function to store (pos, buffer-file-name)
• function to sort by (pos) (which will be given to :sort, i guess org-ql does forward searching only, so sorting might not be neccessary, just reverse would be fine)

If I may propose,:action-reversewould be useful, (it would do the forward search to build up the nodes, and call action-reverse for each match in reverse order (in buffer position)
then, one could just perform modification work in :action-reverse function

[alphapapa]

IIUC, you needn't bother about those complicated patterns. Just perform the edit in the org-ql-select's ACTION function. When your function returns, org-ql will continue and find the next match, skipping to the next heading before searching again.

Alternatively, you could return a marker rather than an integer position, collecting a list of markers. Then use org-with-point-at with the marker as the first argument.

since buffer itself might get destroyed after org-ql call

It's good to keep that in mind, but since this is your own code, what would be killing the buffer too soon?

[pcompassion]

wow, thanks for the response.

So it was the org-element-at-point all along.

I thought cutting element from source buffer somehow disrupts org-ql-select , selecting next node.

But it was org-element-at-point (with non-ascii character) playing against me.

About buffer, I provided files and since org-ql-select opened the buffer,
I thought I had no reason to expect the buffers to be alive after org-ql-select returned
(Because at that time, I thought I couldn't modify buffer in :action and stored buffer so that I can work on later)

Now, looking at org-ql-select code, it advances with outline-next-heading,
so, cutting an element wouldn't matter, as long as following two cases are covered

1. my cursor falls into next-heading, then this heading would be considered as (current-heading) by org-ql and it will be skipped. (and it happend because (org-end-of-subtree t t) placed cursor after current element against my assumption)

2. my cursor somehow falls into nowhere
   (I'm not sure if any cursor position after the first heading belongs to a heading in org document,
   specifically I'm not sure if org considers in-between-lines as part of previous heading,
   which I suspect so)

Thank you for taking time to point it out.

[alphapapa]

I can't say I understand exactly what you mean. But:

my cursor falls into next-heading, then this heading would be considered as (current-heading) by org-ql and it will be skipped.

Unless your action function actually deletes the heading, I don't think this will be a problem.

my cursor somehow falls into nowhere (I'm not sure if any cursor position after the first heading belongs to a heading in org document, specifically I'm not sure if org considers in-between-lines as part of previous heading, which I suspect so)

Yes, every position after the first heading is part of an "entry" under a heading. There is no non-entry content after the first heading position, nor any between-entry content.

[pcompassion]

Unless your action function actually deletes the heading, I don't think this will be a problem.

Yes that's exactly what I'm doing, I'm doing similar thing as org-journal carries over items
Essentially, it's moving left over items from previous days to current buffer.
(I'm doing two moving operations, the one described above (cut-and-paste)..
and another one is.. creating a copy of items into this buffer,

With two operationsm I'm trying to collect items into a buffer and sends it back to where it came
Then I can have editable agenda buffer where I don't need to visit the original buffer to edit items
)

So, this kind of operation is not typical I guess, org-ml also doesn't seem to expect this kind of operation (I can't cut and paste somewhere because org-ml doesn't expect someone to change :begin)

Yes, every position after the first heading is part of an "entry" under a heading. There is no non-entry content after the first heading position, nor any between-entry content.

Thank you for letting me know!