Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: HTTP client interface #1776

Merged
merged 58 commits into from
Sep 8, 2023
Merged

refactor: HTTP client interface #1776

Show file tree
Hide file tree
Changes from 56 commits
Commits
Show all changes
58 commits
Select commit Hold shift + click to select a range
a8d6791
initial version of http server and client
nasdf Aug 11, 2023
25d2250
update lens client http base url
nasdf Aug 11, 2023
1e70947
integration tests working with http client
nasdf Aug 11, 2023
466fe78
fix linter errors
nasdf Aug 11, 2023
d86ee90
fixes to make more tests pass
nasdf Aug 11, 2023
ef58ecd
add http subscription logic
nasdf Aug 14, 2023
da866b5
implement more http collection methods
nasdf Aug 14, 2023
1c3c828
implement lens http server and client
nasdf Aug 14, 2023
6a8e798
refactor http client and server for better use in testing. move clien…
nasdf Aug 15, 2023
0162b1c
implement http transaction api
nasdf Aug 16, 2023
38d0f0e
http client fixes
nasdf Aug 16, 2023
7f00cba
clean document after save and update
nasdf Aug 19, 2023
9962325
more http client test fixes
nasdf Aug 22, 2023
f5c29b3
almost all tests passing
nasdf Aug 22, 2023
6a72bb6
fix lint errors
nasdf Aug 22, 2023
ce0054a
Merge branch 'develop' into nasdf/refactor/api-client-interface
nasdf Aug 22, 2023
508cd83
enable all tests
nasdf Aug 22, 2023
0ebf11c
fix http lens client with incorrect method path
nasdf Aug 22, 2023
677f18f
fix http transaction middleware
nasdf Aug 22, 2023
48a4103
handle transaction conflict errors in http client
nasdf Aug 23, 2023
41ce1cd
update lens getMigrations test logic
nasdf Aug 23, 2023
edd246e
fix linter warnings
nasdf Aug 23, 2023
181d6c3
fix race in event publish
nasdf Aug 24, 2023
680efcd
add wait group sync to publisher
nasdf Aug 24, 2023
b6f7d4a
fix bug in memory datastore
nasdf Aug 24, 2023
166080e
fix memory transaction deadlock
nasdf Aug 24, 2023
509cf00
use transaction close lock in memory datastore
nasdf Aug 24, 2023
add612f
revert publisher changes
nasdf Aug 24, 2023
b8614a3
disable lens transactions in http middleware
nasdf Aug 24, 2023
efbc15c
enable transaction middleware. make http client WithTxn public
nasdf Aug 24, 2023
dc4bbdd
change http server implementation to chi
nasdf Aug 25, 2023
b0eda37
increase test timeout
nasdf Aug 25, 2023
b9dfeb1
move compare logic for integration tests. implement AnyOf comparison …
nasdf Aug 25, 2023
7a2ec0e
more test cleanup
nasdf Aug 25, 2023
0854393
cleanup
nasdf Aug 25, 2023
e89d045
add logger middleware
nasdf Aug 25, 2023
51590d3
add generic functions for array test helpers. rename compare to results
nasdf Aug 28, 2023
cfd66a4
add generic helper for trimExplainAttributes
nasdf Aug 28, 2023
dc859a9
Merge branch 'develop' into nasdf/refactor/api-client-interface
nasdf Aug 28, 2023
13c98fc
more cleanup. add PrintDump to http client
nasdf Aug 28, 2023
2de06d1
Merge branch 'develop' into nasdf/refactor/api-client-interface
nasdf Aug 31, 2023
8146640
update http client names to better match interfaces
nasdf Aug 31, 2023
fd20479
improve http error parsing. fix bug in collection get http handler
nasdf Aug 31, 2023
17fa8be
revert test case changes that are fixed now
nasdf Aug 31, 2023
5f4e043
add generic resultOptionsAreEqual test function
nasdf Sep 1, 2023
9f88fef
test utils cleanup. fix change detector defaults
nasdf Sep 1, 2023
7b12e04
add comments to ignored close errors in http client
nasdf Sep 1, 2023
7f7f747
add http middleware context documentation
nasdf Sep 1, 2023
5f96d78
relax comparisons when testing non go clients
nasdf Sep 1, 2023
31f9b98
add error documentation. restore lens test comment
nasdf Sep 5, 2023
cba08de
only test go client by default. allow change detector with any client…
nasdf Sep 5, 2023
4dded7d
clean up generic function calls. remove comment notes. adjust assertR…
nasdf Sep 5, 2023
df79ef4
fix lens http client parsing bug
nasdf Sep 5, 2023
05f6f80
fix bug with body parsing logic in lens migrate handlers
nasdf Sep 6, 2023
97efa00
fix bug in http lens client migrate
nasdf Sep 6, 2023
35d0040
Merge branch 'develop' into nasdf/refactor/api-client-interface
nasdf Sep 7, 2023
cc773c7
fixes for failing http tests
nasdf Sep 8, 2023
170d07a
update result test helper names
nasdf Sep 8, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 15 additions & 5 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ ifdef BUILD_TAGS
BUILD_FLAGS+=-tags $(BUILD_TAGS)
endif

TEST_FLAGS=-race -shuffle=on -timeout 210s
TEST_FLAGS=-race -shuffle=on -timeout 300s

PLAYGROUND_DIRECTORY=playground
LENS_TEST_DIRECTORY=tests/integration/schema/migrations
Expand Down Expand Up @@ -187,11 +187,15 @@ test\:build:

.PHONY: test\:ci
test\:ci:
DEFRA_BADGER_MEMORY=true DEFRA_BADGER_FILE=true $(MAKE) test:all
DEFRA_BADGER_MEMORY=true DEFRA_BADGER_FILE=true \
DEFRA_CLIENT_GO=true DEFRA_CLIENT_HTTP=true \
$(MAKE) test:all

.PHONY: test\:ci-gql-mutations
test\:ci-gql-mutations:
DEFRA_MUTATION_TYPE=gql DEFRA_BADGER_MEMORY=true $(MAKE) test:all
DEFRA_MUTATION_TYPE=gql DEFRA_BADGER_MEMORY=true \
DEFRA_CLIENT_GO=true DEFRA_CLIENT_HTTP=true \
$(MAKE) test:all

.PHONY: test\:gql-mutations
test\:gql-mutations:
Expand All @@ -204,7 +208,9 @@ test\:gql-mutations:
# UpdateDoc will call [Collection.Update].
.PHONY: test\:ci-col-named-mutations
test\:ci-col-named-mutations:
DEFRA_MUTATION_TYPE=collection-named DEFRA_BADGER_MEMORY=true $(MAKE) test:all
DEFRA_MUTATION_TYPE=collection-named DEFRA_BADGER_MEMORY=true \
DEFRA_CLIENT_GO=true DEFRA_CLIENT_HTTP=true \
$(MAKE) test:all

.PHONY: test\:col-named-mutations
test\:col-named-mutations:
Expand All @@ -214,6 +220,10 @@ test\:col-named-mutations:
test\:go:
go test $(DEFAULT_TEST_DIRECTORIES) $(TEST_FLAGS)

.PHONY: test\:http
test\:http:
DEFRA_CLIENT_HTTP=true go test $(DEFAULT_TEST_DIRECTORIES) $(TEST_FLAGS)

.PHONY: test\:names
test\:names:
gotestsum --format testname -- $(DEFAULT_TEST_DIRECTORIES) $(TEST_FLAGS)
Expand Down Expand Up @@ -285,7 +295,7 @@ test\:coverage-html:
.PHONY: test\:changes
test\:changes:
@$(MAKE) deps:lens
env DEFRA_DETECT_DATABASE_CHANGES=true gotestsum -- ./... -shuffle=on -p 1
env DEFRA_DETECT_DATABASE_CHANGES=true DEFRA_CLIENT_GO=true gotestsum -- ./... -shuffle=on -p 1

.PHONY: validate\:codecov
validate\:codecov:
Expand Down
8 changes: 4 additions & 4 deletions datastore/memory/txn.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -123,9 +123,9 @@ func (t *basicTxn) GetSize(ctx context.Context, key ds.Key) (size int, err error

// Has implements ds.Has.
func (t *basicTxn) Has(ctx context.Context, key ds.Key) (exists bool, err error) {
t.ds.closeLk.RLock()
defer t.ds.closeLk.RUnlock()
if t.ds.closed {
t.closeLk.RLock()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For reference, for myself and other reviewers, the rest of the funcs in basicTxn do this (t.closeLk) and this appears to be a bug fix.

defer t.closeLk.RUnlock()
if t.closed {
return false, ErrClosed
}

Expand Down Expand Up @@ -162,7 +162,7 @@ func (t *basicTxn) Put(ctx context.Context, key ds.Key, value []byte) error {
func (t *basicTxn) Query(ctx context.Context, q dsq.Query) (dsq.Results, error) {
t.closeLk.RLock()
defer t.closeLk.RUnlock()
if t.ds.closed {
if t.closed {
return nil, ErrClosed
}

Expand Down
1 change: 1 addition & 0 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,7 @@ require (
github.com/tidwall/btree v1.6.0
github.com/ugorji/go/codec v1.2.11
github.com/valyala/fastjson v1.6.4
github.com/vito/go-sse v1.0.0
go.opentelemetry.io/otel/metric v1.17.0
go.opentelemetry.io/otel/sdk/metric v0.40.0
go.uber.org/zap v1.25.0
Expand Down
5 changes: 5 additions & 0 deletions go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1035,6 +1035,7 @@ github.com/nats-io/nuid v1.0.1/go.mod h1:19wcPz3Ph3q0Jbyiqsd0kePYG7A95tJPxeL+1OS
github.com/neelance/astrewrite v0.0.0-20160511093645-99348263ae86/go.mod h1:kHJEU3ofeGjhHklVoIGuVj85JJwZ6kWPaJwCIxgnFmo=
github.com/neelance/sourcemap v0.0.0-20151028013722-8c68805598ab/go.mod h1:Qr6/a/Q4r9LP1IltGz7tA7iOK1WonHEYhu1HRBA7ZiM=
github.com/nxadm/tail v1.4.4/go.mod h1:kenIhsEOeOJmVchQTgglprH7qJGnHDVpk1VPCcaMI8A=
github.com/nxadm/tail v1.4.8 h1:nPr65rt6Y5JFSKQO7qToXr7pePgD6Gwiw05lkbyAQTE=
github.com/oklog/oklog v0.3.2/go.mod h1:FCV+B7mhrz4o+ueLpx+KqkyXRGMWOYEvfiXtdGtbWGs=
github.com/oklog/run v1.0.0/go.mod h1:dlhp/R75TPv97u0XWUtDeV/lRKWPKSdTuV0TZvrmrQA=
github.com/olekukonko/tablewriter v0.0.0-20170122224234-a0225b3f23b5/go.mod h1:vsDQFd/mU46D+Z4whnwzcISnGGzXWMclvtLoiIKAKIo=
Expand All @@ -1044,6 +1045,7 @@ github.com/onsi/ginkgo v1.8.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+W
github.com/onsi/ginkgo v1.12.0/go.mod h1:oUhWkIvk5aDxtKvDDuw8gItl8pKl42LzjC9KZE0HfGg=
github.com/onsi/ginkgo v1.12.1/go.mod h1:zj2OWP4+oCPe1qIXoGWkgMRwljMUYCdkwsT2108oapk=
github.com/onsi/ginkgo v1.14.0/go.mod h1:iSB4RoI2tjJc9BBv4NKIKWKya62Rps+oPG/Lv9klQyY=
github.com/onsi/ginkgo v1.16.5 h1:8xi0RTUf59SOSfEtZMvwTvXYMzG4gV23XVHOZiXNtnE=
github.com/onsi/ginkgo/v2 v2.11.0 h1:WgqUCUt/lT6yXoQ8Wef0fsNn5cAuMK7+KT9UFRz2tcU=
github.com/onsi/ginkgo/v2 v2.11.0/go.mod h1:ZhrRA5XmEE3x3rhlzamx/JJvujdZoJ2uvgI7kR0iZvM=
github.com/onsi/gomega v1.4.1/go.mod h1:C1qb7wdrVGGVU+Z6iS04AVkA3Q65CEZX59MT0QO5uiA=
Expand Down Expand Up @@ -1272,6 +1274,8 @@ github.com/valyala/fastjson v1.6.4 h1:uAUNq9Z6ymTgGhcm0UynUAB6tlbakBrz6CQFax3BXV
github.com/valyala/fastjson v1.6.4/go.mod h1:CLCAqky6SMuOcxStkYQvblddUtoRxhYMGLrsQns1aXY=
github.com/viant/assertly v0.4.8/go.mod h1:aGifi++jvCrUaklKEKT0BU95igDNaqkvz+49uaYMPRU=
github.com/viant/toolbox v0.24.0/go.mod h1:OxMCG57V0PXuIP2HNQrtJf2CjqdmbrOx5EkMILuUhzM=
github.com/vito/go-sse v1.0.0 h1:e6/iTrrvy8BRrOwJwmQmlndlil+TLdxXvHi55ZDzH6M=
github.com/vito/go-sse v1.0.0/go.mod h1:2wkcaQ+jtlZ94Uve8gYZjFpL68luAjssTINA2hpgcZs=
github.com/warpfork/go-testmark v0.12.1 h1:rMgCpJfwy1sJ50x0M0NgyphxYYPMOODIJHhsXyEHU0s=
github.com/warpfork/go-wish v0.0.0-20180510122957-5ad1f5abf436/go.mod h1:x6AKhvSSexNrVSrViXSHUEbICjmGXhtgABaHIySUSGw=
github.com/warpfork/go-wish v0.0.0-20190328234359-8b3e70f8e830/go.mod h1:x6AKhvSSexNrVSrViXSHUEbICjmGXhtgABaHIySUSGw=
Expand Down Expand Up @@ -1818,6 +1822,7 @@ gopkg.in/ini.v1 v1.67.0/go.mod h1:pNLf8WUiyNEtQjuu5G5vTm06TEv9tsIgeAvK8hOrP4k=
gopkg.in/resty.v1 v1.12.0/go.mod h1:mDo4pnntr5jdWRML875a/NmxYqAlA73dVijT2AXvQQo=
gopkg.in/src-d/go-cli.v0 v0.0.0-20181105080154-d492247bbc0d/go.mod h1:z+K8VcOYVYcSwSjGebuDL6176A1XskgbtNl64NSg+n8=
gopkg.in/src-d/go-log.v1 v1.0.1/go.mod h1:GN34hKP0g305ysm2/hctJ0Y8nWP3zxXXJ8GFabTyABE=
gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7 h1:uRGJdciOHaEIrze2W8Q3AKkepLTh2hOroT7a+7czfdQ=
gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7/go.mod h1:dt/ZhP58zS4L8KSrWDmTeBkI65Dw0HsyUHuEVlX15mw=
gopkg.in/warnings.v0 v0.1.2/go.mod h1:jksf8JmL6Qr/oQM2OXTHunEvvTAsrWBLb6OOjuVWRNI=
gopkg.in/yaml.v2 v2.0.0-20170812160011-eb3733d160e7/go.mod h1:JAlM8MvJe8wmxCU4Bli9HhUf9+ttbYbLASfIpnQbh74=
Expand Down
Loading