Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

schedule: operator.check will meet datarace #8263

Closed
lhy1024 opened this issue Jun 6, 2024 · 0 comments · Fixed by #8264
Closed

schedule: operator.check will meet datarace #8263

lhy1024 opened this issue Jun 6, 2024 · 0 comments · Fixed by #8264
Labels
affects-5.4 This bug affects the 5.4.x(LTS) versions. affects-6.1 This bug affects the 6.1.x(LTS) versions. affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-7.1 This bug affects the 7.1.x(LTS) versions. affects-7.5 This bug affects the 7.5.x(LTS) versions. affects-8.1 This bug affects the 8.1.x(LTS) versions. severity/major type/bug The issue is confirmed as a bug.

Comments

@lhy1024
Copy link
Contributor

lhy1024 commented Jun 6, 2024

Bug Report

What did you do?

run a server which built with -race

What did you expect to see?

no datarace

What did you see instead?

WARNING: DATA RACE
Read at 0x00c059c28fc8 by goroutine 2253766:
  github.com/tikv/pd/pkg/schedule/operator.(*Operator).Check()
      /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/pkg/schedule/operator/operator.go:382 +0x544
  github.com/tikv/pd/pkg/schedule/operator.(*Controller).pollNeedDispatchRegion()
      /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/pkg/schedule/operator/operator_controller.go:262 +0x144
  github.com/tikv/pd/pkg/schedule/operator.(*Controller).PushOperators()
      /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/pkg/schedule/operator/operator_controller.go:281 +0x2e
  github.com/tikv/pd/pkg/schedule.(*Coordinator).drivePushOperator()
      /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/pkg/schedule/coordinator.go:412 +0x224
  github.com/tikv/pd/pkg/schedule.(*Coordinator).Run.func5()
      /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/pkg/schedule/coordinator.go:491 +0x33

Previous write at 0x00c059c28fc8 by goroutine 37567:
  sync/atomic.CompareAndSwapInt64()
      /usr/local/go/src/runtime/race_amd64.s:316 +0xb
  sync/atomic.CompareAndSwapInt64()
      <autogenerated>:1 +0x1a
  github.com/tikv/pd/pkg/schedule/operator.(*Controller).Dispatch()
      /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/pkg/schedule/operator/operator_controller.go:145 +0x104
  github.com/tikv/pd/server/cluster.(*RaftCluster).HandleRegionHeartbeat()
      /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/server/cluster/cluster_worker.go:66 +0x5c6  github.com/tikv/pd/server.(*GrpcServer).RegionHeartbeat()
      /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/server/grpc_service.go:1296 +0x1c59
  github.com/pingcap/kvproto/pkg/pdpb._PD_RegionHeartbeat_Handler()
      /go/pkg/mod/github.com/pingcap/[email protected]/pkg/pdpb/pdpb.pb.go:9777 +0xb8
  github.com/grpc-ecosystem/go-grpc-prometheus.init.(*ServerMetrics).StreamServerInterceptor.func4()
      /go/pkg/mod/github.com/grpc-ecosystem/[email protected]/server_metrics.go:121 +0x15e
  go.etcd.io/etcd/etcdserver/api/v3rpc.Server.ChainStreamServer.func9.1()
      /go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:83 +0xc3
  go.etcd.io/etcd/etcdserver/api/v3rpc.newStreamInterceptor.func1()
      /go/pkg/mod/go.etcd.io/[email protected]/etcdserver/api/v3rpc/interceptor.go:238 +0x6f0
  go.etcd.io/etcd/etcdserver/api/v3rpc.Server.ChainStreamServer.func9()
      /go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:86 +0x218
  google.golang.org/grpc.(*Server).processStreamingRPC()
      /go/pkg/mod/google.golang.org/[email protected]/server.go:1687 +0x1f50
  google.golang.org/grpc.(*Server).handleStream()
      /go/pkg/mod/google.golang.org/[email protected]/server.go:1801 +0x19f3
  google.golang.org/grpc.(*Server).serveStreams.func2.1()
      /go/pkg/mod/google.golang.org/[email protected]/server.go:1027 +0x15c

Goroutine 2253766 (running) created at:
  github.com/tikv/pd/pkg/schedule.(*Coordinator).Run()
      /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/pkg/schedule/coordinator.go:491 +0x44a
  github.com/tikv/pd/pkg/schedule.(*Coordinator).RunUntilStop()
      /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/pkg/schedule/coordinator.go:455 +0x48
  github.com/tikv/pd/server/cluster.(*schedulingController).runCoordinator()
      /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/server/cluster/scheduling_controller.go:121 +0xe6
  github.com/tikv/pd/server/cluster.(*schedulingController).startSchedulingJobs.func2()
      /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/server/cluster/scheduling_controller.go:98 +0x33

Goroutine 37567 (running) created at:
  google.golang.org/grpc.(*Server).serveStreams.func2()
      /go/pkg/mod/google.golang.org/[email protected]/server.go:1038 +0x224
  google.golang.org/grpc/internal/transport.(*http2Server).operateHeaders()
      /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:619 +0x3a01
  google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams()
      /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:660 +0x285
  google.golang.org/grpc.(*Server).serveStreams()
      /go/pkg/mod/google.golang.org/[email protected]/server.go:1021 +0x6bb
  google.golang.org/grpc.(*Server).handleRawConn.func1()
      /go/pkg/mod/google.golang.org/[email protected]/server.go:957 +0x86
==================

What version of PD are you using (pd-server -V)?

@lhy1024 lhy1024 added the type/bug The issue is confirmed as a bug. label Jun 6, 2024
@lhy1024 lhy1024 added affects-6.1 This bug affects the 6.1.x(LTS) versions. affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-7.1 This bug affects the 7.1.x(LTS) versions. affects-7.5 This bug affects the 7.5.x(LTS) versions. affects-8.1 This bug affects the 8.1.x(LTS) versions. labels Jun 6, 2024
@lhy1024 lhy1024 added affects-5.4 This bug affects the 5.4.x(LTS) versions. and removed may-affects-5.4 labels Jun 7, 2024
@ti-chi-bot ti-chi-bot bot closed this as completed in #8264 Jun 7, 2024
ti-chi-bot bot pushed a commit that referenced this issue Jun 7, 2024
ti-chi-bot pushed a commit to ti-chi-bot/pd that referenced this issue Jul 16, 2024
ti-chi-bot bot pushed a commit that referenced this issue Jul 19, 2024
ti-chi-bot bot added a commit that referenced this issue Aug 1, 2024
close #8263

Signed-off-by: lhy1024 <[email protected]>

Co-authored-by: lhy1024 <[email protected]>
Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
ti-chi-bot pushed a commit to ti-chi-bot/pd that referenced this issue Aug 27, 2024
ti-chi-bot pushed a commit to ti-chi-bot/pd that referenced this issue Aug 27, 2024
ti-chi-bot pushed a commit to ti-chi-bot/pd that referenced this issue Aug 27, 2024
ti-chi-bot bot pushed a commit that referenced this issue Sep 4, 2024
close #8263

Signed-off-by: husharp <[email protected]>

Co-authored-by: husharp <[email protected]>
Co-authored-by: Hu# <[email protected]>
ti-chi-bot bot added a commit that referenced this issue Sep 10, 2024
close #8263

Signed-off-by: husharp <[email protected]>

Co-authored-by: husharp <[email protected]>
Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
affects-5.4 This bug affects the 5.4.x(LTS) versions. affects-6.1 This bug affects the 6.1.x(LTS) versions. affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-7.1 This bug affects the 7.1.x(LTS) versions. affects-7.5 This bug affects the 7.5.x(LTS) versions. affects-8.1 This bug affects the 8.1.x(LTS) versions. severity/major type/bug The issue is confirmed as a bug.
Projects
Development

Successfully merging a pull request may close this issue.

2 participants