Generating Cohorts
Anthony G. Sena and Martijn J. Schuemie
-2023-10-10
+2024-05-28
Source:vignettes/GeneratingCohorts.Rmd
GeneratingCohorts.RmdGenerating Cohorts= cohortTableNames$cohortTable
)
#> Connecting using SQLite driver
-#> Counting cohorts took 0.124 secs
+#> Counting cohorts took 0.126 secs
#> cohortId cohortEntries cohortSubjects
#> 1 1778211 1800 1800
#> 2 1778212 569 569
@@ -382,6 +385,7 @@ Incremental Mode)
#> Connecting using SQLite driver
#> Table "cohort" already exists and in incremental mode, so not recreating it.
+#> Table "cohort" already exists and in incremental mode, so not recreating it.
#> Table "cohort_inclusion" already exists and in incremental mode, so not recreating it.
#> Table "cohort_inclusion_result" already exists and in incremental mode, so not recreating it.
#> Table "cohort_inclusion_stats" already exists and in incremental mode, so not recreating it.
@@ -429,7 +433,7 @@ Incremental Mode#> Skipping cohortId = '1778211' because it is unchanged from earlier run
#> Skipping cohortId = '1778212' because it is unchanged from earlier run
#> Skipping cohortId = '1778213' because it is unchanged from earlier run
-#> Generating cohort set took 0.17 secs
+#> Generating cohort set took 0.11 secs
Potential Pitfalls of Incremental Mode
@@ -469,7 +473,7 @@ Potential Pitfalls of Incrementa
diff --git a/docs/articles/SamplingCohorts.html b/docs/articles/SamplingCohorts.html
new file mode 100644
index 0000000..76bc9ec
--- /dev/null
+++ b/docs/articles/SamplingCohorts.html
@@ -0,0 +1,243 @@
+
+
+
+
+
+
+
+Sampling Cohorts • CohortGenerator
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+-
+
+
+-
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ Sampling Cohorts
+ James P.
+Gilbert
+
+ 2024-05-28
+
+ Source: vignettes/SamplingCohorts.Rmd
+ SamplingCohorts.Rmd
+
+
+
+
+
+
+Sampling with CohortGenerator
+
+Large populations of individuals (e.g. all subjects receiving a
+COVID-19 vaccination) can often be too large to work with when pulling
+down a large collection of covariates for further analysis. This is
+prohibitive when designing studies or attempting to generate phenotypes.
+This guide aims to demonstrate how one can use the
+sampleCohortDefinitionSet functionality to produce
+sufficiently large sample cohorts from a base
+cohortDefinitionSet.
+
+Sampling method
+
+The approach taken in this method is to sample individuals within a
+cohort without replacement. Different database platforms implement a
+variety of different approaches for random sampling and random number
+generation that make cross platform sql difficult. Consequently, all the
+randomness computed here happens inside R itself simply
+from the count of unique individuals within a cohort.
+
+
+Using the sampler functions
+
+To create a single sample the approach below will create cohorts in
+the same base table.
+First we need to load the initial cohort definition set
+
+cds <- getCohortDefinitionSet(...)
+We then need to create the cohort tables and cohorts in the usual
+manner.
+
+connectionDetails <- Eunomia::getEunomiaConnectionDetails()
+conn <- DatabaseConnector::connect(connectionDetails = connectionDetails)
+on.exit(DatabaseConnector::disconnect(conn))
+
+
+cds <- getCohortDefinitionSet(...)
+cohortTableNames <- getCohortTableNames(cohortTable = "cohort")
+recordKeepingFolder <- file.path(outputFolder, "RecordKeepingSamples")
+
+createCohortTables(
+ connectionDetails = connectionDetails,
+ cohortDatabaseSchema = "main",
+ cohortTableNames = cohortTableNames
+)
+
+generateCohortSet(
+ cohortDefinitionSet = cds,
+ connection = conn,
+ cdmDatabaseSchema = "main",
+ cohortDatabaseSchema = "main",
+ cohortTableNames = cohortTableNames,
+ incremental = TRUE,
+ incrementalFolder = recordKeepingFolder
+)
+We can then create a new cohort definition set from the original
+sample.
+
+sampledCohortDefinitionSet <- sampleCohortDefinitionSet(
+ cohortDefinitionSet = cds,
+ connection = conn,
+ sampleFraction = 0.33,
+ seed = 64374, # OHDSI
+ cohortDatabaseSchema = "main",
+ cohortTableNames = cohortTableNames,
+ incremental = TRUE,
+ incrementalFolder = recordKeepingFolder
+)
+The resulting sampledCohortDefinitionSet is nearly
+identical to the base cohort set, however a few changes occur:
+
+- The name now include the postfix
+
[sample 33% seed=64374]
+
+- The optional parameter
idExpression changes the cohort
+name. By default this is set to cohortId * 1000 + seed
+however, this will throw an error if the ids are the same
+- The
+- The base cohortDefinitionSet is attached as an attribute to the
+
sampledCohortDefinitionSet
+
+
+To generate multiple samples, simply specify multiple seed variables
+as follows:
+
+# Generate 800 samples of size n
+sampledCohortDefinitionSet <- sampleCohortDefinitionSet(
+ cohortDefinitionSet = cds,
+ connection = conn,
+ n = 1000,
+ seed = 1:800 * 64374, # OHDSI
+ cohortDatabaseSchema = "main",
+ cohortTableNames = cohortTableNames,
+ incremental = TRUE,
+ incrementalFolder = recordKeepingFolder
+)
+Note that using incremental mode for your sampled cohorts will also
+work. In this case, a cohort will only be re-generated if the checksum
+of the base cohort has changed (the checksum is based on the cohort
+SQL). The checksum applies to the pseudorandom seed of the cohort and
+the sample size (n).
+As the sampledCohortDefinitionSet is, for all intents and purposes, a
+set of cohortDefinitions it can be passed as a parameter to all other
+OHDSI packages with minimal issues. For example,
+FeatureExtraction will be able to use this sample
+unchanged.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
diff --git a/docs/articles/index.html b/docs/articles/index.html
index 0d0e955..c0fbc19 100644
--- a/docs/articles/index.html
+++ b/docs/articles/index.html
@@ -1,5 +1,5 @@
-Articles • CohortGenerator Articles • CohortGenerator
@@ -17,7 +17,7 @@
@@ -37,6 +37,9 @@
Generating Cohorts
+
+ Sampling Cohorts
+
#> Connecting using SQLite driver#> Counting cohorts took 0.124 secs#> Counting cohorts took 0.126 secs#> cohortId cohortEntries cohortSubjects
#> 1 1778211 1800 1800
#> 2 1778212 569 569
@@ -382,6 +385,7 @@ Incremental Mode)
#> Connecting using SQLite driver#> Table "cohort" already exists and in incremental mode, so not recreating it.
+#> Table "cohort" already exists and in incremental mode, so not recreating it.
#> Table "cohort_inclusion" already exists and in incremental mode, so not recreating it.
#> Table "cohort_inclusion_result" already exists and in incremental mode, so not recreating it.
#> Table "cohort_inclusion_stats" already exists and in incremental mode, so not recreating it.
@@ -429,7 +433,7 @@ Incremental Mode#> Skipping cohortId = '1778211' because it is unchanged from earlier run
#> Skipping cohortId = '1778212' because it is unchanged from earlier run
#> Skipping cohortId = '1778213' because it is unchanged from earlier run
-#> Generating cohort set took 0.17 secs
Potential Pitfalls of Incremental Mode
@@ -469,7 +473,7 @@Potential Pitfalls of Incrementa
diff --git a/docs/articles/SamplingCohorts.html b/docs/articles/SamplingCohorts.html
new file mode 100644
index 0000000..76bc9ec
--- /dev/null
+++ b/docs/articles/SamplingCohorts.html
@@ -0,0 +1,243 @@
+
+
+
+
+
+
+
+Sampling Cohorts • CohortGenerator
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+-
+
+
+-
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ Sampling Cohorts
+ James P.
+Gilbert
+
+ 2024-05-28
+
+ Source: vignettes/SamplingCohorts.Rmd
+ SamplingCohorts.Rmd
+
+
+
+
+
+
+Sampling with CohortGenerator
+
+Large populations of individuals (e.g. all subjects receiving a
+COVID-19 vaccination) can often be too large to work with when pulling
+down a large collection of covariates for further analysis. This is
+prohibitive when designing studies or attempting to generate phenotypes.
+This guide aims to demonstrate how one can use the
+sampleCohortDefinitionSet functionality to produce
+sufficiently large sample cohorts from a base
+cohortDefinitionSet.
+
+Sampling method
+
+The approach taken in this method is to sample individuals within a
+cohort without replacement. Different database platforms implement a
+variety of different approaches for random sampling and random number
+generation that make cross platform sql difficult. Consequently, all the
+randomness computed here happens inside R itself simply
+from the count of unique individuals within a cohort.
+
+
+Using the sampler functions
+
+To create a single sample the approach below will create cohorts in
+the same base table.
+First we need to load the initial cohort definition set
+
+cds <- getCohortDefinitionSet(...)
+We then need to create the cohort tables and cohorts in the usual
+manner.
+
+connectionDetails <- Eunomia::getEunomiaConnectionDetails()
+conn <- DatabaseConnector::connect(connectionDetails = connectionDetails)
+on.exit(DatabaseConnector::disconnect(conn))
+
+
+cds <- getCohortDefinitionSet(...)
+cohortTableNames <- getCohortTableNames(cohortTable = "cohort")
+recordKeepingFolder <- file.path(outputFolder, "RecordKeepingSamples")
+
+createCohortTables(
+ connectionDetails = connectionDetails,
+ cohortDatabaseSchema = "main",
+ cohortTableNames = cohortTableNames
+)
+
+generateCohortSet(
+ cohortDefinitionSet = cds,
+ connection = conn,
+ cdmDatabaseSchema = "main",
+ cohortDatabaseSchema = "main",
+ cohortTableNames = cohortTableNames,
+ incremental = TRUE,
+ incrementalFolder = recordKeepingFolder
+)
+We can then create a new cohort definition set from the original
+sample.
+
+sampledCohortDefinitionSet <- sampleCohortDefinitionSet(
+ cohortDefinitionSet = cds,
+ connection = conn,
+ sampleFraction = 0.33,
+ seed = 64374, # OHDSI
+ cohortDatabaseSchema = "main",
+ cohortTableNames = cohortTableNames,
+ incremental = TRUE,
+ incrementalFolder = recordKeepingFolder
+)
+The resulting sampledCohortDefinitionSet is nearly
+identical to the base cohort set, however a few changes occur:
+
+- The name now include the postfix
+
[sample 33% seed=64374]
+
+- The optional parameter
idExpression changes the cohort
+name. By default this is set to cohortId * 1000 + seed
+however, this will throw an error if the ids are the same
+- The
+- The base cohortDefinitionSet is attached as an attribute to the
+
sampledCohortDefinitionSet
+
+
+To generate multiple samples, simply specify multiple seed variables
+as follows:
+
+# Generate 800 samples of size n
+sampledCohortDefinitionSet <- sampleCohortDefinitionSet(
+ cohortDefinitionSet = cds,
+ connection = conn,
+ n = 1000,
+ seed = 1:800 * 64374, # OHDSI
+ cohortDatabaseSchema = "main",
+ cohortTableNames = cohortTableNames,
+ incremental = TRUE,
+ incrementalFolder = recordKeepingFolder
+)
+Note that using incremental mode for your sampled cohorts will also
+work. In this case, a cohort will only be re-generated if the checksum
+of the base cohort has changed (the checksum is based on the cohort
+SQL). The checksum applies to the pseudorandom seed of the cohort and
+the sample size (n).
+As the sampledCohortDefinitionSet is, for all intents and purposes, a
+set of cohortDefinitions it can be passed as a parameter to all other
+OHDSI packages with minimal issues. For example,
+FeatureExtraction will be able to use this sample
+unchanged.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
diff --git a/docs/articles/index.html b/docs/articles/index.html
index 0d0e955..c0fbc19 100644
--- a/docs/articles/index.html
+++ b/docs/articles/index.html
@@ -1,5 +1,5 @@
-Articles • CohortGenerator Articles • CohortGenerator
@@ -17,7 +17,7 @@
-
+
-
+
+
+
- + + + + + +
Sampling Cohorts
+James P. +Gilbert
+ +2024-05-28
+ + Source:vignettes/SamplingCohorts.Rmd
+ SamplingCohorts.RmdSampling with CohortGenerator +
+Large populations of individuals (e.g. all subjects receiving a
+COVID-19 vaccination) can often be too large to work with when pulling
+down a large collection of covariates for further analysis. This is
+prohibitive when designing studies or attempting to generate phenotypes.
+This guide aims to demonstrate how one can use the
+sampleCohortDefinitionSet functionality to produce
+sufficiently large sample cohorts from a base
+cohortDefinitionSet.
Sampling method +
+The approach taken in this method is to sample individuals within a
+cohort without replacement. Different database platforms implement a
+variety of different approaches for random sampling and random number
+generation that make cross platform sql difficult. Consequently, all the
+randomness computed here happens inside R itself simply
+from the count of unique individuals within a cohort.
Using the sampler functions +
+To create a single sample the approach below will create cohorts in +the same base table.
+First we need to load the initial cohort definition set
+
+cds <- getCohortDefinitionSet(...)We then need to create the cohort tables and cohorts in the usual +manner.
+
+connectionDetails <- Eunomia::getEunomiaConnectionDetails()
+conn <- DatabaseConnector::connect(connectionDetails = connectionDetails)
+on.exit(DatabaseConnector::disconnect(conn))
+
+
+cds <- getCohortDefinitionSet(...)
+cohortTableNames <- getCohortTableNames(cohortTable = "cohort")
+recordKeepingFolder <- file.path(outputFolder, "RecordKeepingSamples")
+
+createCohortTables(
+ connectionDetails = connectionDetails,
+ cohortDatabaseSchema = "main",
+ cohortTableNames = cohortTableNames
+)
+
+generateCohortSet(
+ cohortDefinitionSet = cds,
+ connection = conn,
+ cdmDatabaseSchema = "main",
+ cohortDatabaseSchema = "main",
+ cohortTableNames = cohortTableNames,
+ incremental = TRUE,
+ incrementalFolder = recordKeepingFolder
+)We can then create a new cohort definition set from the original +sample.
+
+sampledCohortDefinitionSet <- sampleCohortDefinitionSet(
+ cohortDefinitionSet = cds,
+ connection = conn,
+ sampleFraction = 0.33,
+ seed = 64374, # OHDSI
+ cohortDatabaseSchema = "main",
+ cohortTableNames = cohortTableNames,
+ incremental = TRUE,
+ incrementalFolder = recordKeepingFolder
+)The resulting sampledCohortDefinitionSet is nearly
+identical to the base cohort set, however a few changes occur:
-
+
- The name now include the postfix
+
[sample 33% seed=64374]+
+ - The optional parameter
idExpressionchanges the cohort +name. By default this is set tocohortId * 1000 + seed+however, this will throw an error if the ids are the same
+ - The +
- The base cohortDefinitionSet is attached as an attribute to the
+
sampledCohortDefinitionSet+
+
To generate multiple samples, simply specify multiple seed variables +as follows:
+
+# Generate 800 samples of size n
+sampledCohortDefinitionSet <- sampleCohortDefinitionSet(
+ cohortDefinitionSet = cds,
+ connection = conn,
+ n = 1000,
+ seed = 1:800 * 64374, # OHDSI
+ cohortDatabaseSchema = "main",
+ cohortTableNames = cohortTableNames,
+ incremental = TRUE,
+ incrementalFolder = recordKeepingFolder
+)Note that using incremental mode for your sampled cohorts will also +work. In this case, a cohort will only be re-generated if the checksum +of the base cohort has changed (the checksum is based on the cohort +SQL). The checksum applies to the pseudorandom seed of the cohort and +the sample size (n).
+As the sampledCohortDefinitionSet is, for all intents and purposes, a
+set of cohortDefinitions it can be passed as a parameter to all other
+OHDSI packages with minimal issues. For example,
+FeatureExtraction will be able to use this sample
+unchanged.