How to access stored documents on the server?

[PhilippCh]

First of all, thank you so much for the work you've done. I love this project, it's been so easy to add offline-first synchronization to my web app and C# backend using Yjs and this package for the server.

I'm currently struggling with how to modify documents on the server-side. I'm using the MongoDb Server and want to create a separate REST endpoint that my client apps can call (while online) to sort the objects stored in the document (using SyncedStore in the client projects). For this, I'd like to retrieve all objects stored in MongoDb and convert them to class instances that I have in the backend, modify them and then write them back to the store to be propagated to all clients.

Maybe I'm not searching right, but I haven't seen any methods to retrieve a stored document on the server and cast it's contents to my class types. Should I be using the IDocumentManager interface and if so, is there an example on what exactly you need to do?

[Edit]
In re-reading the example provided, I think it became more clear how the synchronization is supposed to work, but I'm still unsure as to how the connection is made from the server acting as a client and server for the documents at the same time. How would I retrieve a remote document? On the web clients I open my connections to the websocket API provided by the server, but I don't see how this would be done from the server directly.

[SebastianStehle]

Hi,

the demo somehow shows that: https://github.com/y-crdt/ydotnet/blob/main/Demo/Callback.cs

You can implement a callback and then listen to changes on the client. This would be one way.

Another option could be the following method:
https://github.com/y-crdt/ydotnet/blob/main/YDotNet.Server/IDocumentManager.cs#L20

But you are right, there is no option to get all documents right now. I am not sure if this is really needed, as I would only make the change if a user is actually connected.

[PhilippCh]

Thanks a lot for your input! In my use case, what I want to achieve is to have an API endpoint that iterates over todo items and assigns them labels created by an algorithm. I want to trigger this via an API endpoint since finding which labels to use requires an LLM call, which I don't want to perform every time each todo item changes.

With your hint on IDocumentManager.UpdateDocAsync, I managed to cobble together the following method for getting all tasks in the DB:

public async Task<IEnumerable<TaskModel>> GetAllTasksAsync(CancellationToken cancellationToken = default)
{
	var tasks = new List<TaskModel>();
	var context = new DocumentContext("shoppr", 0);

	await documentManager.UpdateDocAsync(
		context, doc =>
		{
			var docTasks = doc.Array("tasks");
			using var transaction = docTasks.ReadTransaction();
			tasks.AddRange(docTasks.Iterate(transaction).Select(output => output.To<TaskModel>(transaction)));
		}, cancellationToken);

	return tasks;
}

Do you recognize any obvious issues with this approach? I'd then modify the tasks and push them back in using another UpdateDocAsync call.

[SebastianStehle]

No, this looks good to me.

[PhilippCh]

Then thanks for your help, I appreciate it!

[SebastianStehle]

If you need an endpoint to fetch all documents, a PR would be welcome :)

[PhilippCh]

I'll try to add one to IDocumentManager with the logic described above. Would it be find with you to also add the possibility of passing JsonSerializerOptions to the To<T> extension method of Output in the same PR as well or should I open a separate one for that change?

Currently, I need to specify a JsonParameterName attribute for each field of my model classes because I want to name them in PascalCase in C#, but they're actually stored in camelCase coming from the web clients. Allowing passing of serializer options I would be able to use the PropertyNameCaseInsensitive field or some other serialization logic.

[SebastianStehle]

Separate PRs are always welcome.

[PhilippCh]

Alright, will do :)

Another question I came across while trying to update the tasks in the document: Is there a way to modify an element in a doc array instead of removing and then adding it back in? Like I mentioned, I'm converting my Outputs to TaskModel instances, which I can then modify, but I don't see a straightforward way to insert the changes back into the Array.

[Edit]
After some fiddling, I came up with this a little clunky method for updating multiple elements in an array. The first call to RemoveRange seems to work fine, but the subsequent call to add the updated element back in throws the following error:

System.Runtime.InteropServices.SEHException (0x80004005): External component has thrown an exception.
   at YDotNet.Native.Types.ArrayChannel.InsertRange(IntPtr array, IntPtr transaction, UInt32 index, IntPtr items, UInt32 itemsLength)
   at YDotNet.Document.Types.Arrays.Array.InsertRange(Transaction transaction, UInt32 index, Input[] inputs)
   at Shoppr.Api.Domain.Database.YDbContext.<>c__DisplayClass4_0`1.<UpdateArrayAsync>b__0(Doc doc) in C:\repos\shoppr\backend\shoppr-backend\Shoppr.Api\Domain\Database\YDbContext.cs:line 42
   at YDotNet.Server.DefaultDocumentManager.<>c__DisplayClass10_0.<<UpdateDocAsync>b__0>d.MoveNext()

The code is as follows (pretty similar to querying all elements):

public async Task UpdateArrayAsync<T>(string collectionName, IEnumerable<ArrayElement<T>> items, CancellationToken cancellationToken = default)
{
	await documentManager.UpdateDocAsync(
		_documentContext, doc =>
		{
			var docArray = doc.Array(collectionName);
			foreach (var item in items)
			{
				using var transaction = docArray.WriteTransaction();
				docArray.RemoveRange(transaction, item.Index, 1);
				docArray.InsertRange(transaction, item.Index, [item.Item.ToInput()]);
				transaction.Commit();
			}
		}, cancellationToken);
}

Not sure what's going on here tbh, but a little guidance would be greatly appreciated :)
Also, when throwing the unhandled exception, I noticed that the array element stayed deleted. Is this intentional? I read that the transaction is committed on Dispose(), but is there a way to cancel a transaction with e.g. a Try/Catch?

[Edit 2]
It seems that there's been a fix for this issue with #99. Do you have an ETA for when you want to release v0.4.1?

[SebastianStehle]

If you have an array of objects you can just modify the objects. There might be exceptions like these, because it is a very new library but I hope we can always fix them asap.

Perhaps we should also add a method to replace an element.

[PhilippCh]

I don't know how much work it is, but would it be possible to release a 0.4.1 patch including the fix you provided in #99? I tried setting up the dev environment, built the y-crdt libraries as specified and also the YDotNet.Native.Win32 dll. When referencing the built dll in my project, I keep running into the following error:

System.EntryPointNotFoundException: Unable to find an entry point named 'ybranch_write_transaction' in DLL 'yrs'.
   at YDotNet.Native.Types.Branches.BranchChannel.WriteTransaction(IntPtr branch)
   at YDotNet.Document.Types.Branches.Branch.WriteTransaction()
   at Shoppr.Api.Domain.Database.YDbContext.<>c__DisplayClass9_0`1.<PerformWithTransaction>b__0(Doc doc) in C:\repos\shoppr\backend\shoppr-backend\Shoppr.Api\Domain\Database\YDbContext.cs:line 115
   at YDotNet.Server.DefaultDocumentManager.<>c__DisplayClass10_0.<<UpdateDocAsync>b__0>d.MoveNext()

Any idea what I'm missing? As far as I can see in the y-crdt repo, there's only ydoc_write_transaction entry points, never ybranch_write_transaction.

[PhilippCh]

Thanks for updating to 0.4.1 :) Appreciate it!

After some fiddling and still receiving the SEHException, it seems that I found at least a workaround. In InputFactory.FromJson, the first line works while the second (returning Input.Object instead of Input.Map) results in the exception.

Input ConvertObject(JsonElement element)
{
	// Works
	return Input.Map(element.EnumerateObject().ToDictionary(x => useCamelCaseNames ? x.Name.ToCamelCase() : x.Name, x => ConvertValue(x.Value), StringComparer.Ordinal));
	// Results in Unrecognized YVal value tag. as mentioned in #99.
	return Input.Object(element.EnumerateObject().ToDictionary(x => useCamelCaseNames ? x.Name.ToCamelCase() : x.Name, x => ConvertValue(x.Value), StringComparer.Ordinal));
}

[Edit]
Okay, it seems that storing a map in the doc instead of a JSON object does make a difference when reading the values in the frontend. It seems that I do need to store my objects as Input.Object in the backend.

I've attempted to construct an Input.Object manually as follows, that also throws the error. Do you have any idea what could be amiss here?

var task2 = Input.Object(new Dictionary<string, Input>
{
	{"uuid", Guid.NewGuid().ToString().ToInput()},
	{"tagUuids", new List<Guid>().ToInput()}
});

// The following line throws. When changing above to Input.Map, it works.
array.InsertRange(transaction, 0, task2);

[SebastianStehle]

No idea at the moment. I will have to dig into that. Is there no test that covers that?

[PhilippCh]

I have no idea tbh. I'd appreciate you looking into the issue as being able to write JsonArrays and JsonObjects would simplify my code immensely. There's currently a workaround by storing all objects as YArray and YMap in the frontend (you'll also need to convert all children of the objects if they're objects or arrays). Then you're able to write to them in the backend.

When I get back from vacation, I'll try to dig a bit deeper as well.