Persist the data and retrieve it...

[aboimpinto]

Hello,

I did a small example where I have a Key-Value-Pair of string/string and I want to persist it in the disk and next time I open the app, I want to retrieve the data.

Found this example
https://github.com/microsoft/FASTER/blob/ebe4c31594a61aa9a88e8480122fe1128fd37aa5/docs/_docs/25-fasterkv-recovery.md
that could fit my needs but I'm getting errors and I have some questions:

1. In the example the ObjectLogDevice is never set and without I'm getting an error. Is the example not aligned with the last version of FASTER?
2. When I read the data, why I need to provide the key name as ref? Why I could not just provide a normal string?
3. Why Read cannot just return the output instead of return the value as ref? This makes everything a little bit complicated to understand.
4. Is there a way to persist the date after every Upsert? Tried to understand the WaitForCommitAsync with no success.
5. Why the TakeFullCheckpoint return a guid? I notice the guid is used to persist the data. How I should use it? Found this tests
   https://github.com/microsoft/FASTER/blob/a31b52b29e952827b4ad464674df57d9e230020a/cs/test/ObjectRecoveryTest.cs
   but when I use the token to recover, I get an error
   
6. How can we know if there is already persisted data to recover instead of creating?

I'm using Ubuntu Linux and .NET 5.0 and this the code

using System;
using FASTER.core;

namespace FASTERTestBed
{
    class Program
    {
        private static IDevice log, objLog;
        private static FasterKV<string, string> store;
        private static Guid token;

        static void Main(string[] args)
        {
            log = Devices.CreateLogDevice("Channels.db", deleteOnClose: false);
            objLog = Devices.CreateLogDevice("Channels.obj.db", deleteOnClose: false);
            store = new FasterKV<string, string>(
                1L << 20,
                new LogSettings { LogDevice = log, ObjectLogDevice = objLog }
            );

            Run();
            Continue();
        }

        private static void Run()
        {
            var session = store.NewSession(new SimpleFunctions<string, string>(), "s1");

            session.Upsert("Key1", "Value1");
            session.Upsert("Key2", "Value2");

            var key = "Key2";
            var output = string.Empty;
            session.Read(ref key, ref output);

            Console.WriteLine($"{key} | {output}");

            store.TakeFullCheckpoint(out token);     // why the guid?
            Console.WriteLine(token);

        }

        private static void Continue()
        {
            store.Recover(token, token);
            var session = store.ResumeSession(new SimpleFunctions<string, string>(), "s1", out CommitPoint cp);

            var key = "Key2";
            var output = string.Empty;
            session.Read(ref key, ref output);

            Console.WriteLine($"{key} | {output}");

            // Is this save return a differnt guid?
            store.TakeFullCheckpoint(out token);
            Console.WriteLine(token);

        }
    }
}

[badrishc]

In the example the ObjectLogDevice is never set and without I'm getting an error. Is the example not aligned with the last version of FASTER?

The example uses value types and therefore did not require an object log. If you use reference types, you need to define an object log.

When I read the data, why I need to provide the key name as ref? Why I could not just provide a normal string?

Ref instances are passed by ref, similar to pointers in C++. This allows us to avoid the overhead of copying value-types on the stack, and lets us work seamlessly with variable-sized value objects (such as SpanByte). But we offer overloads that are NOT ref based, you are welcome to use them.

Why Read cannot just return the output instead of return the value as ref? This makes everything a little bit complicated to understand.

Reads can go pending due to I/O, so it returns a status. Also, a return of Output means users cannot keep a fixed location in memory and read into that, for performance reasons (also see explanation of ref above). For the simpler use cases, ReadAsync indeed has an overload that returns (Status, Output) exactly as you describe.

Is there a way to persist the date after every Upsert? Tried to understand the WaitForCommitAsync with no success.

Yes, there is. After your Upsert, just call checkpoint as await store.TakeHybridLogCheckpointAsync(CheckpointType.FoldOver). See here for details. WaitForCommitAsync is orthogonal, and useful if you have a separate thread/task doing the checkpoints asynchronously for you.

Why the TakeFullCheckpoint return a guid?

Some applications take checkpoints every 10 minutes, and want to recover to a specific checkpoint in the past, e.g., recover to the checkpoint from one hour back. Tokens help identify the specific checkpoint to recover to. They are completely optional -- you can ignore them and Checkpoint/Recover without a token in which can we will simply recover to the latest valid checkpoint:

store.TakeFullCheckpointAsync(CheckpointType.FoldOver).GetAwaiter().GetResult(); 
...
store.Recover()

You were having bugs in your sample above. Here is the fixed version, just run it multiple times, first time it will upsert and take checkpoint, subsequent runs will recover and read (no upsert):

        static void Main(string[] args)
        {
            log = Devices.CreateLogDevice("Channels.db", deleteOnClose: false);
            objLog = Devices.CreateLogDevice("Channels.obj.db", deleteOnClose: false);
            store = new FasterKV<string, string>( 1L << 20, new LogSettings { LogDevice = log, ObjectLogDevice = objLog });

            bool recovered = true;
            try
            {
                store.Recover();
            }
            catch {
                recovered = false;
            }

            if (recovered)
                Continue();
            else
                Run();
        }

        private static void Run()
        {
            Console.WriteLine("Run()");

            var session = store.NewSession(new SimpleFunctions<string, string>(), "s1");

            Console.WriteLine("Upserting");
            session.Upsert("Key1", "Value1");
            session.Upsert("Key2", "Value2");

            Console.WriteLine("Reading");
            var key = "Key2";
            var output = string.Empty;
            session.Read(ref key, ref output);

            Console.WriteLine($"{key} | {output}");

            store.TakeFullCheckpointAsync(CheckpointType.FoldOver).GetAwaiter().GetResult();     // why the guid?
        }

        private static void Continue()
        {
            Console.WriteLine("Continue()");

            var session = store.ResumeSession(new SimpleFunctions<string, string>(), "s1", out CommitPoint cp);

            Console.WriteLine("Reading");
            var key = "Key2";
            var output = string.Empty;
            session.Read(ref key, ref output);

            Console.WriteLine($"{key} | {output}");
        }

How can we know if there is already persisted data to recover instead of creating?

Just call recover after creating the store, if there is nothing to recover it will throw an exception that you can ignore.