Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix versioning. Remove objMap. #243

Open
wants to merge 94 commits into
base: master
Choose a base branch
from

Conversation

brucen1030
Copy link

Tested with test/versioning.py
Objects in versioned buckets are not appendable.

Versioning is handled as follows.

  1. In PUT (new) object, version in db table is set as follows in meta/types/object.go.
func (o *Object) GetCreateSql() (string, []interface{}, uint64) {
	version := math.MaxUint64 - uint64(o.LastModifiedTime.UnixNano())

And responsed to client with converted versionId in meta/client/tidbclient/object.go

func (t *TidbClient) PutObject(object *Object, tx DB) (err error) {
	if tx == nil {
		tx, err = t.Client.Begin()
		if err != nil {
			return err
		}
		defer func() {
			if err == nil {
				err = tx.(*sql.Tx).Commit()
			}
			if err != nil {
				tx.(*sql.Tx).Rollback()
			}
		}()
	}
	sql, args, iversion := object.GetCreateSql()
	object.VersionId = ConvertRawVersionToS3Version(iversion)
  1. In ListObjectVersions, object versions will be returned in meta/client/tidbclient/bucket.go and meta/client/tidbclient/object.go.
func (t *TidbClient) GetObject(bucketName, objectName, version string) (object *Object, err error) {
	var ibucketname, iname, customattributes, acl, lastModifiedTime string
	var iversion uint64

	var row *sql.Row
	sqltext := "select bucketname,name,version,location,pool,ownerid,size,objectid,lastmodifiedtime,etag,contenttype," +
		"customattributes,acl,nullversion,deletemarker,ssetype,encryptionkey,initializationvector,type,storageclass from objects where bucketname=? and name=? "
	if version == "" {
		sqltext += "order by bucketname,name,version limit 1;"
		row = t.Client.QueryRow(sqltext, bucketName, objectName)
	} else if version == ObjectNullVersion {
		sqltext += "and nullversion=1 limit 1;" // There should be only one NullVersion object.
		row = t.Client.QueryRow(sqltext, bucketName, objectName)
	} else {
		sqltext += "and version=?;"
		internalVersion, err := ConvertS3VersionToRawVersion(version)
		if err != nil {
			return nil, ErrInternalError
		}
		row = t.Client.QueryRow(sqltext, bucketName, objectName, internalVersion)
	}
...
	object.VersionId = ConvertRawVersionToS3Version(iversion)

	return
}

  1. In GET object, objectid for ceph will be get with client specified version Id.
  2. In PUT objects in versioned buckets in storage/object.go, old objects will be handled in storage/object.go.
func (yig *YigStorage) checkOldObject(bucketName, objectName, versioning string) (err error) {
	switch versioning {
	case meta.VersionDisabled:
		// Delete all the bucketName + objectName.
		return yig.removeAllObjectsEntryByName(bucketName, objectName) // TODO: not deleted.

	case meta.VersionEnabled:
		// Do nothing.

	case meta.VersionSuspended:
		// Delete null version object.
		var object *meta.Object
		if object, err = yig.MetaStorage.GetObjectVersion(bucketName, objectName, meta.ObjectNullVersion, false); err != nil {
			if err == ErrNoSuchKey {
				return nil
			}
			return err
		}

		return yig.MetaStorage.DeleteObject(object, object.DeleteMarker)

	default:
		return errors.New("No Such versioning status!")
	}

	return nil
}

TODO work

  1. Lifecycle and versioning adaption.
    https://docs.aws.amazon.com/AmazonS3/latest/dev/intro-lifecycle-rules.html#intro-lifecycle-rules-actions

sunfengyun and others added 25 commits October 24, 2019 19:10
Change some names, import and log style.
Improve performace of listing objects
Signed-off-by: sunfch <[email protected]>
Improve performace of listing object
Set the number of query result 10000
fix the problem of start job to sync bucket usage.
fix the problem of removing wrong key when perform bucket delete.
Signed-off-by: sunfch <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants