store: new API ApplyStagedLayer

[giuseppe]

Add a race-condition-free alternative to using CreateLayer and ApplyDiffFromStagingDirectory, ensuring the store is locked for the entire duration while the layer is being created and populated.

Signed-off-by: Giuseppe Scrivano [email protected]

The relative patch for c/image is:

diff --git a/storage/storage_dest.go b/storage/storage_dest.go
index 88e492b7..51b30a64 100644
--- a/storage/storage_dest.go
+++ b/storage/storage_dest.go
@@ -149,7 +149,7 @@ func (s *storageImageDestination) Close() error {
 	}
 	for _, v := range s.diffOutputs {
 		if v.Target != "" {
-			_ = s.imageRef.transport.store.CleanupStagingDirectory(v.Target)
+			_ = s.imageRef.transport.store.CleanupStagedLayer(v)
 		}
 	}
 	return os.RemoveAll(s.directory)
@@ -669,11 +669,6 @@ func (s *storageImageDestination) commitLayer(index int, info addedLayerInfo, si
 			return false, fmt.Errorf("index %d out of range for configOCI.RootFS.DiffIDs", index)
 		}
 
-		layer, err := s.imageRef.transport.store.CreateLayer(id, parentLayer, nil, "", false, nil)
-		if err != nil {
-			return false, err
-		}
-
 		// let the storage layer know what was the original uncompressed layer.
 		flags := make(map[string]interface{})
 		flags[expectedLayerDiffIDFlag] = configOCI.RootFS.DiffIDs[index]
@@ -682,8 +677,15 @@ func (s *storageImageDestination) commitLayer(index int, info addedLayerInfo, si
 			Flags: flags,
 		}
 
-		if err := s.imageRef.transport.store.ApplyDiffFromStagingDirectory(layer.ID, diffOutput.Target, diffOutput, options); err != nil {
-			_ = s.imageRef.transport.store.Delete(layer.ID)
+		args := storage.ApplyStagedLayerOptions{
+			ID:          id,
+			ParentLayer: parentLayer,
+
+			DiffOutput:       diffOutput,
+			DiffOptions:      options,
+		}
+		layer, err := s.imageRef.transport.store.ApplyStagedLayer(args)
+		if err != nil {
 			return false, err
 		}

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: giuseppe

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [giuseppe]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[mtrmac]

Another opportunity to clean up that ApplyDiffWithDiffer and ApplyDiffFromStagingDirectory should have separate options types, so that callers aren’t tempted to set completely ignored options.

I guess that’s non-blocking…

[giuseppe]

I agree the API can be improved, and this is a good chance. More in details, how would you improve this part?

[mtrmac]

I’m not sure I have the whole picture … All I really wanted here was to have separate ApplyDiffWithDifferOptions and ApplyDiffFromStagingDirectoryOptions, and to drop fields that are not relevant to either operation.

But looking at ApplyDiffOptions below, even that seems fairly invasive to do to the maximum extent.

---

Looking a bit further… A lot of this might be better as separate PRs, and probably not right now?

• Is there any caller of ApplyDiffWithDiffer writing to a pre-existing layer? If not, maybe that can be just dropped. In the driver, the usingComposefs paths have already diverged, which seem like a good reason to either share more code (not in this PR!), or to remove the redundant code path entirely.
• From a c/image perspective, we now have two “apply” functions which are ~opposites of each other, and it always takes me a bit to tell which one is “stage” and which one is “commit”. So something like s/ApplyDiffWithDiffer/StageChangesWithDiffer/ would be nice — but that also somewhat depends on the above.
• Actually having ApplyDiffOptions inside ApplyDiffWithDifferOptions is not a good semantic match at all — ApplyDiffOptions all revolve around the ApplyDiffOptions.Diff stream, and that is completely ignored in those paths.
• … and using the graphdriver.Differ interface to connect the overlay driver and the zstd deduplicator code seems like an imprecise fit as well; maybe that should be just a c-storage-private interface with no exposed methods, so that c/storage can change the mechanics over time.
• as mentioned elsewhere, it would be convenient for CleanupStagingDirectory to consume DriverWithDifferOutput so that c/image doesn’t need to care about .Target at all

[mtrmac]

A much more general, and much more vague, thought is that the “stage”+“commit” model might be interesting for non-chunked layers as well.

Right now, on the PutBlob path, c/image:

• putBlobToPendingFile stores the input stream into a file, fully consuming it. (That must happen to validate digests.) That is fully parallel.
• commitLayer extracts the file into a the graph driver’s layer. On some graph drivers (DM, VFS), that is inherently serial; in overlay, that is just a tar extraction which could, in principle, be fully parallel
• creates the layer record, with parent links, etc. That is inherently serial, or maybe it could be parallel but it is anyway cheap enough not to worry about.

It seems potentially interesting to see whether the “extract tar” part could be parallelized — and whether it would be better. I can imagine that this part is I/O heavy enough that doing two of them would just slow things down; that probably needs building and measuring.

The "store stream” + “extract" + “commit" parts seem very similar to what the chunked path does in the “convert” case. OTOH “true chunked” input does’t have a stream in the first place…

---

But then, worrying about the traditional tar layers is backwards-looking. What would an ideal API for creating a composefs layer look like? I didn’t look into that and I have no idea. Would that be relevant for building the chunked one?

[mtrmac]

… I think Dan would bite our head off if we worked on changing the traditional-tar API right now :)

[mtrmac]

Thanks for working on this!

[giuseppe]

@mtrmac thanks for the review.

I've addressed your comments, except one pending question: #1826 (comment)

[mtrmac]

Changing the API, after looking a bit more, seems to to require a bunch of fairly intrusive changes all at once… so I don’t think it is worth blocking this PR on them.

Or maybe you can see some way to make all of that simple.

[mtrmac]

I’m not sure I have the whole picture … All I really wanted here was to have separate ApplyDiffWithDifferOptions and ApplyDiffFromStagingDirectoryOptions, and to drop fields that are not relevant to either operation.

But looking at ApplyDiffOptions below, even that seems fairly invasive to do to the maximum extent.

---

Looking a bit further… A lot of this might be better as separate PRs, and probably not right now?

• Is there any caller of ApplyDiffWithDiffer writing to a pre-existing layer? If not, maybe that can be just dropped. In the driver, the usingComposefs paths have already diverged, which seem like a good reason to either share more code (not in this PR!), or to remove the redundant code path entirely.
• From a c/image perspective, we now have two “apply” functions which are ~opposites of each other, and it always takes me a bit to tell which one is “stage” and which one is “commit”. So something like s/ApplyDiffWithDiffer/StageChangesWithDiffer/ would be nice — but that also somewhat depends on the above.
• Actually having ApplyDiffOptions inside ApplyDiffWithDifferOptions is not a good semantic match at all — ApplyDiffOptions all revolve around the ApplyDiffOptions.Diff stream, and that is completely ignored in those paths.
• … and using the graphdriver.Differ interface to connect the overlay driver and the zstd deduplicator code seems like an imprecise fit as well; maybe that should be just a c-storage-private interface with no exposed methods, so that c/storage can change the mechanics over time.
• as mentioned elsewhere, it would be convenient for CleanupStagingDirectory to consume DriverWithDifferOutput so that c/image doesn’t need to care about .Target at all

[mtrmac]

A much more general, and much more vague, thought is that the “stage”+“commit” model might be interesting for non-chunked layers as well.

Right now, on the PutBlob path, c/image:

• putBlobToPendingFile stores the input stream into a file, fully consuming it. (That must happen to validate digests.) That is fully parallel.
• commitLayer extracts the file into a the graph driver’s layer. On some graph drivers (DM, VFS), that is inherently serial; in overlay, that is just a tar extraction which could, in principle, be fully parallel
• creates the layer record, with parent links, etc. That is inherently serial, or maybe it could be parallel but it is anyway cheap enough not to worry about.

It seems potentially interesting to see whether the “extract tar” part could be parallelized — and whether it would be better. I can imagine that this part is I/O heavy enough that doing two of them would just slow things down; that probably needs building and measuring.

The "store stream” + “extract" + “commit" parts seem very similar to what the chunked path does in the “convert” case. OTOH “true chunked” input does’t have a stream in the first place…

---

But then, worrying about the traditional tar layers is backwards-looking. What would an ideal API for creating a composefs layer look like? I didn’t look into that and I have no idea. Would that be relevant for building the chunked one?

[mtrmac]

… I think Dan would bite our head off if we worked on changing the traditional-tar API right now :)

[rhatdan]

LGTM

[mtrmac]

On second thought, I’m afraid this isn’t sufficient.

It is fine for a running system: it prevents other processes from observing the WIP layer.

But it doesn’t handle crashes sufficiently. For that, the layer metadata needs to be saved to disk with incompleteFlag (so that if we are recovering from a crash, we delete everything), and after the contents are set up, the flag is removed again.

(Or, hypothetically, we could first write the on-disk contents and only afterwards write the layer metadata?? But that would be a new unproven code path, and we would have to worry about re-creating a layer on top of previously partially-created files. Seems risky, when the other path is well-understood.)

Very roughly speaking, I think this can be done by applying the staged data from inside layerStore.create, around the place where applyDiffWithOptions is called for non-chunked layers.

[mtrmac]

The API design LGTM.

For the record:

The relative patch for c/image is:

diff --git a/storage/storage_dest.go b/storage/storage_dest.go
index 88e492b7..51b30a64 100644
--- a/storage/storage_dest.go
+++ b/storage/storage_dest.go

+		layer, err := s.imageRef.transport.store.ApplyStagedLayer(args)
+		if err != nil {

This path in c/image also needs to check for, and succeed with, ErrDuplicateID. And it would be convenient, and consistent with PutLayer, for the new function to return a layer value with the duplicate object in this case.

[giuseppe]

On second thought, I’m afraid this isn’t sufficient.

It is fine for a running system: it prevents other processes from observing the WIP layer.

But it doesn’t handle crashes sufficiently. For that, the layer metadata needs to be saved to disk with incompleteFlag (so that if we are recovering from a crash, we delete everything), and after the contents are set up, the flag is removed again.

(Or, hypothetically, we could first write the on-disk contents and only afterwards write the layer metadata?? But that would be a new unproven code path, and we would have to worry about re-creating a layer on top of previously partially-created files. Seems risky, when the other path is well-understood.)

Very roughly speaking, I think this can be done by applying the staged data from inside layerStore.create, around the place where applyDiffWithOptions is called for non-chunked layers.

you are right, we need to use the incompleteFlag. I've pushed a new version where I set the incompleteFlag, barely tested. I'll work more on it through the day

[giuseppe]

@mtrmac what do you think of the last version?

[mtrmac]

I’d rather prefer if the incompleteFlag remained an internal implementation detail of layers.go.

[giuseppe]

I’d rather prefer if the incompleteFlag remained an internal implementation detail of layers.go.

moved the incompleteFlag usage to layers.go

[mtrmac]

This is a half-way step, but the partialoption is still basically “leave the layer incomplete” and/or “I promise to callapplyDiffFromStagingDirectorylater” (with neither effect documented), withstore.go` being responsible for that.

It seems to me

	if diff != nil {
		if size, err = r.applyDiffWithOptions …
+	else if staged != nil {
+		if size, err = r.applyDiffFromStagingDirectory …
	else { …

should be possible and and not too invasive a change.

[giuseppe]

pushed a new version, I moved the applyDiffFromStagingDirectory() call inside create()

[mtrmac]

ACK.

[mtrmac]

/lgtm

Thanks!

[giuseppe]

@mtrmac the related change in c/image: containers/image#2301