[datahub migration] Failed to configure the sink (datahub-rest)

[laurajsdias]

Describe the bug
GMS connection not working via token.

To Reproduce
Steps to reproduce the behavior:

I'm trying to migrate Datahub from GKE to another cluster in AWS EKS. Even after redeploying Datahub and restoring the database, data is not there. Then I found this: https://datahubproject.io/docs/generated/ingestion/sources/datahub/

So, when I try to run this recipe on source Datahub, it fails to configure the sink on target Datahub. I have the right token (created again a few times already) and right permissions but no luck.

Also, not sure if it's important, but I've exposed gms service on a URL like: https://datahub-aws.dev.example.com/gms. (example here just to not disclose the name of the company)

And I do have METADATA_SERVICE_AUTH_ENABLED set to true both on frontend and gms.

Expected behavior
I need the sink to be created.

Error

raise PipelineInitError(f"Failed to {step}: {e}") from e
datahub.ingestion.run.pipeline.PipelineInitError: Failed to configure the sink (datahub-rest): 💥 Failed to connect to DataHub with DataHubRestEmitter: configured to talk to https://datahub-aws.dev.example.com/gms with token: eyJh**********2NTw

This is the recipe I have:

run_id: 'urn:li:dataHubExecutionRequest:90b157aa-870e-4362-8135-391b890286c9'
sink:
  type: datahub-rest
  config:
    server: 'https://datahub-aws.dev.example.com/gms'
    token: >token<
datahub_api:
  server: 'http://datahub-datahub-gms.datahub-dev:8080'
  token: >token<
flags:
  set_system_metadata: false
source:
  type: datahub
  config:
    include_all_versions: false
    kafka_connection:
      bootstrap: 'http://datahub-prerequisites-kafka.datahub-dev:9092'
      schema_registry_url: 'https://datahub.dev.example.com/schema-registry/api/'
    urn_pattern:
      allow:
        - '^allowed.urn.*'
      deny:
        - '^denied.urn.*'
    stateful_ingestion:
      ignore_old_state: false
      enabled: true
    database_connection:
      password: ""
      database: datahub
      scheme: postgresql+psycopg2
      host_port: 'datahub-prerequisites-gcloud-sqlproxy.datahub-dev:5432'
      username: [email protected]
pipeline_name: datahub_source_1