Chained reads w/ SpanByte

[adam-dot-cohen]

Is there an example somewhere in the codebase of chained reads using SpanByte similar to the VersionReadApp sample? Prior to version 2.3 the code below worked seamlessly. In the latest version of FASTER it appears to cause memory leaks - and chained reads with the new push iterator take up to 5 minutes for a single key ( log file w/ ~5K keys and ~20M chained entries).

public class KvStoreChainedFunctions : SpanByteFunctions<KvStoreCallbackContext>
{
	private protected readonly MemoryPool<byte> _memoryPool;
	public KvStoreChainedFunctions(bool locking = false) { this._memoryPool = MemoryPool<byte>.Shared; }

	/// <inheritdoc/>
	public override bool InitialUpdater(ref SpanByte key, ref SpanByte input, ref SpanByte value, ref SpanByteAndMemory output, ref RMWInfo rmwInfo)
	=> return false;


	public override bool ConcurrentWriter(ref SpanByte key, ref SpanByte input, ref SpanByte src, ref SpanByte dst, ref SpanByteAndMemory output, ref UpsertInfo upsertInfo)
	{
		// We can write the source (src) data to the existing destination (dst) in-place, 
		// only if there is sufficient space
		if (dst.Length < src.Length) return false;

		// Note: Do not copy SRC to DST when chaining
		src.CopyTo(ref output, this._memoryPool);

		// We can adjust the length header on the serialized log, if we wish.
		// This method will also zero out the extra space to retain log scan correctness.
		dst.ShrinkSerializedLength(src.Length);

		return false;
	}

	/// <inheritdoc/>
	public override bool SingleWriter(ref SpanByte key, ref SpanByte input, ref SpanByte src, ref SpanByte dst, ref SpanByteAndMemory output, ref UpsertInfo upsertInfo, WriteReason reason)
	{
		src.CopyTo(ref dst);
		src.CopyTo(ref output, this._memoryPool);
		return true;
	}

	/// <inheritdoc/>
	public override bool CopyUpdater(ref SpanByte key, ref SpanByte input, ref SpanByte oldValue, ref SpanByte newValue, ref SpanByteAndMemory output, ref RMWInfo rmwInfo)
	{
		input.CopyTo(ref newValue);
		input.CopyTo(ref output, this._memoryPool);
		return true;
	}

	/// <inheritdoc/>
	public override bool InPlaceUpdater(ref SpanByte key, ref SpanByte input, ref SpanByte value, ref SpanByteAndMemory output, ref RMWInfo rmwInfo)
	{
		// The default implementation of IPU simply writes input to destination, if there is space
		base.InPlaceUpdater(ref key, ref input, ref value, ref output, ref rmwInfo);
		input.CopyTo(ref output, this._memoryPool);
		return true;
	}
	/// <inheritdoc />
	public override void ReadCompletionCallback(ref SpanByte key, ref SpanByte input, ref SpanByteAndMemory output, KvStoreCallbackContext ctx, Status status, RecordMetadata recordMetadata)
	{
		if (ctx != null)
		{
			ctx.recordInfo = recordMetadata;
			ctx.status = status;
		}
	}

[TedHartMS]

is the calling code calling output.Memory.Dispose()?

[adam-dot-cohen]

Yes, via IMemoryOwner. Do you see any missing or problematic with the SpanByte function overloads to force chaining? Maybe an issue with InPlaceUpdater or ConcurrentWriter returning false? Concurrent writer intermittently gets called multiple times for a single write operation . It’s seems like it might be an issue with the function callbacks Thanks for the quick response

[TedHartMS]

With ConcurrentWriter returning false, SingleWriter should be called--and this may be overwriting output.Memory.