Fix/#869 - Investigate using Queue without lock

[trgiangdo]

Resolves #869

[github-actions]

Taipy Overall Coverage Report

File	Stmts	Miss	Cover	Missing
__init__.py	16	1	93%	31
_entrypoint.py	36	4	88%	44, 46, 50–51
_run.py	38	2	94%	77–78
version.py	9	0	100%
_cli
__init__.py	0	0	100%
_help_cli.py	19	0	100%
_run_cli.py	25	11	56%	40, 42–46, 48–49, 51, 53, 59
_scaffold_cli.py	18	2	88%	41–42
_cli/_base_cli
__init__.py	1	0	100%
_cli.py	41	0	100%
config
__init__.py	38	9	76%	41–42, 44–49, 70
_config.py	60	1	98%	68
_init.py	3	0	100%
config.py	145	4	97%	87, 154, 178, 233
section.py	42	6	85%	35, 40, 44, 48, 53, 57
setup.py	15	15	0%	14–16, 18, 20–21, 23, 25–29, 31, 33, 35
unique_section.py	5	0	100%
version.py	9	0	100%
config/_config_comparator
__init__.py	0	0	100%
_comparator_result.py	95	3	96%	67, 110–111
_config_comparator.py	68	0	100%
config/_serializer
__init__.py	0	0	100%
_base_serializer.py	124	5	95%	39, 141–142, 144–145
_json_serializer.py	24	3	87%	33–35
_toml_serializer.py	23	0	100%
config/checker
__init__.py	0	0	100%
_checker.py	14	0	100%
issue.py	15	0	100%
issue_collector.py	28	0	100%
config/checker/_checkers
__init__.py	0	0	100%
_auth_config_checker.py	25	25	0%	12–14, 17–19, 21–24, 26–30, 32–34, 39–40, 47–49, 54–55
_config_checker.py	30	1	96%	28
config/common
__init__.py	0	0	100%
_classproperty.py	5	0	100%
_config_blocker.py	28	0	100%
_repr_enum.py	7	0	100%
_template_handler.py	111	11	90%	39, 53, 55, 59, 61, 63, 99–100, 118, 149–150
_validate_id.py	10	0	100%
frequency.py	7	0	100%
scope.py	22	0	100%
typing.py	3	0	100%
config/exceptions
__init__.py	1	0	100%
exceptions.py	5	0	100%
config/global_app
__init__.py	0	0	100%
global_app_config.py	32	1	96%	38
core
__init__.py	7	0	100%
_core.py	70	0	100%
_core_cli.py	46	0	100%
_init.py	19	0	100%
_init_version.py	10	0	100%
setup.py	26	26	0%	14–17, 19, 21, 23–24, 26–31, 35, 37–43, 45, 48, 50, 59
taipy.py	399	49	87%	164, 220, 263, 268, 273, 278, 283, 288, 293, 298, 303, 341–343, 348, 353, 358, 363, 368, 373, 378, 383, 423, 462, 491–492, 499–505, 531–534, 892–893, 920, 923, 969, 1008, 1024, 1073–1077
core/_backup
__init__.py	0	0	100%
_backup.py	33	2	93%	54–55
core/_entity
__init__.py	0	0	100%
_dag.py	65	1	98%	94
_entity.py	22	0	100%
_entity_ids.py	22	0	100%
_labeled.py	48	0	100%
_migrate_cli.py	74	4	94%	98–99, 117–118
_properties.py	35	0	100%
_reload.py	62	1	98%	72
submittable.py	75	4	94%	47, 102, 106, 110
core/_entity/_migrate
__init__.py	3	0	100%
_migrate_fs.py	66	5	92%	43, 68, 104–106
_migrate_mongo.py	76	5	93%	52, 71, 87, 105, 127
_migrate_sql.py	106	53	50%	30–31, 33–36, 38–46, 53, 57, 64–65, 69, 76–77, 81, 89–90, 94, 100–101, 105, 110–111, 115, 120–121, 125, 127–137, 139–140, 154, 198–199, 201–202
_utils.py	199	30	84%	32, 34–36, 38, 59–61, 63–67, 69–71, 73–74, 157, 160, 169, 197–199, 202–204, 241–242, 316
core/_manager
__init__.py	0	0	100%
_manager.py	81	1	98%	113
_manager_factory.py	21	2	90%	28, 32
core/_orchestrator
__init__.py	0	0	100%
_abstract_orchestrator.py	23	4	82%	27, 40, 53, 58
_orchestrator.py	189	14	92%	146, 190–191, 193–199, 201–203, 291
_orchestrator_factory.py	64	11	82%	38, 58, 73–74, 76, 79, 89, 94, 97–98, 100
core/_orchestrator/_dispatcher
__init__.py	3	0	100%
_development_job_dispatcher.py	19	4	78%	27, 30, 33, 36
_job_dispatcher.py	93	8	91%	68, 75–77, 99–100, 117, 135
_standalone_job_dispatcher.py	31	0	100%
_task_function_wrapper.py	56	3	94%	74–75, 83
core/_repository
__init__.py	0	0	100%
_abstract_converter.py	10	2	80%	19, 24
_abstract_repository.py	36	10	72%	29, 41, 53, 63, 73, 80, 90, 101, 113, 124
_base_taipy_model.py	31	4	87%	27–28, 50, 53
_decoder.py	21	2	90%	36, 51
_encoder.py	19	3	84%	30, 36, 41
_filesystem_repository.py	159	21	86%	147, 173–174, 176, 178–181, 190–193, 197–200, 224, 237–238, 257–258
_sql_repository.py	131	16	87%	79–80, 123, 137, 143–144, 147–148, 153–157, 163, 180, 216
core/_repository/db
__init__.py	0	0	100%
_sql_base_model.py	3	0	100%
_sql_connection.py	46	2	95%	82–83
core/_version
__init__.py	0	0	100%
_utils.py	26	0	100%
_version.py	14	0	100%
_version_converter.py	14	0	100%
_version_fs_repository.py	68	5	92%	44–45, 100–101, 134
_version_manager.py	149	8	94%	98, 104, 148–149, 171, 212, 220, 230
_version_manager_factory.py	19	2	89%	26, 29
_version_mixin.py	19	0	100%
_version_model.py	21	0	100%
_version_repository_interface.py	26	7	73%	22, 26, 30, 34, 38, 42, 46
_version_sql_repository.py	53	0	100%
core/_version/_cli
__init__.py	0	0	100%
_bcolor.py	24	10	58%	21, 23–25, 27–31, 33
_version_cli.py	138	4	97%	81, 85, 193, 195
core/common
__init__.py	1	0	100%
_check_dependencies.py	5	1	80%	28
_check_instance.py	23	0	100%
_listattributes.py	29	3	89%	21–22, 32
_mongo_connector.py	17	2	88%	37, 45
_repr_enum.py	7	0	100%
_utils.py	39	0	100%
_warnings.py	21	0	100%
mongo_default_document.py	5	0	100%
typing.py	7	0	100%
warn_if_inputs_not_ready.py	14	0	100%
core/config
__init__.py	30	0	100%
core_section.py	197	5	97%	139, 148, 157, 166, 179
data_node_config.py	350	7	98%	287, 291, 637, 639, 643, 645, 1044
job_config.py	73	3	95%	125–127
migration_config.py	46	1	97%	57
scenario_config.py	149	4	97%	211, 215, 223, 227
task_config.py	97	7	92%	70–71, 149, 152, 220–222
core/config/checkers
__init__.py	0	0	100%
_config_id_checker.py	18	0	100%
_core_section_checker.py	18	0	100%
_data_node_config_checker.py	91	2	97%	112, 147
_job_config_checker.py	19	0	100%
_migration_config_checker.py	30	0	100%
_scenario_config_checker.py	61	0	100%
_task_config_checker.py	34	0	100%
core/cycle
__init__.py	0	0	100%
_cycle_converter.py	11	0	100%
_cycle_fs_repository.py	6	0	100%
_cycle_manager.py	101	0	100%
_cycle_manager_factory.py	20	2	90%	28, 31
_cycle_model.py	23	0	100%
_cycle_sql_repository.py	6	0	100%
cycle.py	109	2	98%	81, 147
cycle_id.py	3	0	100%
core/data
__init__.py	13	0	100%
_abstract_file.py	17	0	100%
_abstract_sql.py	183	30	83%	95, 143, 145, 149, 162–163, 166–167, 179, 185, 187, 189, 195, 202, 235, 254–261, 268, 274, 290, 299–301, 307
_abstract_tabular.py	45	1	97%	40
_data_converter.py	175	5	97%	114, 183, 225, 258, 267
_data_fs_repository.py	6	0	100%
_data_manager.py	117	2	98%	93, 179
_data_manager_factory.py	20	2	90%	28, 31
_data_model.py	32	0	100%
_data_sql_repository.py	6	0	100%
_filter.py	193	17	91%	41, 63, 83–91, 143, 183, 190, 205, 213, 222
aws_s3.py	40	2	95%	103, 106
csv.py	109	3	97%	91, 205, 209
data_node.py	293	7	97%	158, 226, 278, 287, 464, 467, 471
data_node_id.py	5	0	100%
excel.py	202	28	86%	95, 279, 282, 284–288, 292–293, 296, 298–300, 302, 304–305, 307, 312, 314–315, 317–320, 322, 343, 365
generic.py	48	1	97%	83
in_memory.py	28	0	100%
json.py	115	5	95%	91, 170, 174, 179, 183
mongo.py	113	12	89%	207–214, 222, 232, 237, 289
operator.py	11	0	100%
parquet.py	114	3	97%	107, 122, 230
pickle.py	63	0	100%
sql.py	44	1	97%	100
sql_table.py	66	0	100%
core/exceptions
__init__.py	1	0	100%
exceptions.py	118	5	95%	183, 191, 229, 267, 309
core/job
__init__.py	0	0	100%
_job_converter.py	31	2	93%	59–60
_job_fs_repository.py	6	0	100%
_job_manager.py	57	0	100%
_job_manager_factory.py	20	2	90%	28, 31
_job_model.py	26	0	100%
_job_sql_repository.py	6	0	100%
job.py	187	5	97%	30, 77, 149, 295, 336
job_id.py	3	0	100%
status.py	11	0	100%
core/notification
__init__.py	7	0	100%
_registration.py	18	0	100%
_topic.py	23	1	95%	65
core_event_consumer.py	26	1	96%	84
event.py	40	1	97%	139
notifier.py	47	0	100%
registration_id.py	3	0	100%
core/scenario
__init__.py	0	0	100%
_scenario_converter.py	32	0	100%
_scenario_fs_repository.py	6	0	100%
_scenario_manager.py	273	9	96%	88–91, 183, 188, 384, 429, 440
_scenario_manager_factory.py	20	2	90%	28, 31
_scenario_model.py	30	0	100%
_scenario_sql_repository.py	6	0	100%
scenario.py	319	14	95%	127, 130, 132–133, 175, 222, 242, 323, 369, 412, 604, 606, 662, 668
scenario_id.py	3	0	100%
core/sequence
__init__.py	0	0	100%
_sequence_converter.py	19	0	100%
_sequence_manager.py	238	12	94%	227, 241, 243, 246, 251, 257, 283–286, 353, 371
_sequence_manager_factory.py	11	1	90%	23
sequence.py	144	3	97%	81, 160, 172
sequence_id.py	3	0	100%
core/submission
__init__.py	0	0	100%
_submission_converter.py	20	0	100%
_submission_fs_repository.py	6	0	100%
_submission_manager.py	60	0	100%
_submission_manager_factory.py	20	2	90%	28, 31
_submission_model.py	31	0	100%
_submission_sql_repository.py	6	0	100%
submission.py	171	7	95%	111, 119, 140, 186, 189, 195, 248
submission_id.py	3	0	100%
submission_status.py	10	0	100%
core/task
__init__.py	0	0	100%
_task_converter.py	28	1	96%	68
_task_fs_repository.py	6	0	100%
_task_manager.py	121	5	95%	153–154, 159–160, 200
_task_manager_factory.py	20	2	90%	28, 31
_task_model.py	25	0	100%
_task_sql_repository.py	6	0	100%
task.py	107	5	95%	91, 96, 167–169
task_id.py	3	0	100%
gui
__init__.py	16	0	100%
_default_config.py	3	0	100%
_gui_cli.py	37	0	100%
_gui_section.py	30	0	100%
_init.py	1	0	100%
_page.py	32	2	93%	20–21
_warnings.py	11	0	100%
config.py	110	16	85%	175, 177, 179–180, 184, 193, 201, 203, 207, 209, 211, 275–276, 281–282, 323
gui.py	1259	246	80%	51, 372, 374, 420–426, 429–436, 438–439, 441, 443–444, 446–448, 450–451, 453, 455–457, 459–462, 464–465, 467–472, 526, 537, 539, 541, 561, 584, 586, 599–600, 606–607, 628–633, 663, 667, 679, 693, 698, 705–706, 730–732, 742, 771, 780, 800, 817–818, 822–828, 830–835, 868–869, 883, 892, 980, 982, 987–988, 991, 994–995, 997, 999, 1003, 1005–1006, 1011, 1013, 1019, 1024, 1030–1037, 1039, 1041, 1054–1058, 1060–1063, 1077–1079, 1086, 1088–1089, 1092, 1101–1110, 1115, 1119–1120, 1128, 1133, 1144–1151, 1183–1185, 1252–1253, 1318, 1320, 1325, 1334–1335, 1346, 1356, 1362, 1390, 1392–1393, 1446, 1536–1538, 1548, 1557–1565, 1589, 1595, 1751–1754, 1800, 1802, 1853, 1869, 1883–1885, 1887, 1890–1891, 1903, 1908–1910, 1912, 1970, 1984–1985, 2014, 2019, 2031–2034, 2040, 2052–2057, 2077, 2084–2086, 2101, 2114, 2126, 2138, 2140, 2144–2145, 2153–2155, 2169, 2285, 2417, 2430, 2440, 2444
gui_actions.py	96	27	71%	61, 101, 133, 148, 174, 196–199, 220, 234, 238–239, 261–263, 289, 312–314, 318–319, 371, 380, 382, 387, 393
icon.py	23	16	30%	58–63, 66–75
page.py	51	18	64%	21, 46, 48, 50, 55–57, 62–67, 96, 107, 110–112
partial.py	27	3	88%	21, 66, 73
server.py	172	37	78%	42, 67, 91, 109, 152–158, 160–161, 174–175, 184, 187, 194, 205, 256, 258–259, 282, 291, 293, 298–299, 306–307, 309–311, 321, 323, 325, 328–329
setup.py	35	35	0%	14, 16–18, 20–21, 23, 25, 27–32, 35, 37–43, 45, 48, 50, 60–64, 67–70, 73
state.py	108	18	83%	24, 128, 140, 154, 180, 195, 210–212, 222–223, 235–238, 241–242, 245
types.py	90	4	95%	152, 157, 174, 176
gui/_renderers
__init__.py	73	10	86%	21–22, 40, 49, 60, 85–87, 91, 98
builder.py	664	113	82%	45, 124, 143–144, 160–162, 164, 177, 204, 236–237, 241, 259–260, 264, 279, 297, 302–303, 307, 318, 335–338, 347, 350–352, 365, 378, 380–381, 384, 388–391, 393, 400, 442, 454, 500–503, 505–509, 511–514, 516–520, 530, 550, 567, 574, 579, 594, 601, 604–605, 607, 610, 614, 629, 636, 662, 674, 676–680, 683–684, 708, 711, 713, 799, 802, 825–826, 830, 862, 867–870, 872–875, 881–882, 892, 945, 962, 967, 974–978, 1001
factory.py	78	3	96%	20–21, 579
json.py	50	9	82%	29, 33, 39, 46–47, 63–66
utils.py	66	8	87%	38–39, 54, 66–67, 78–79, 90
gui/_renderers/_html
__init__.py	1	0	100%
factory.py	9	1	88%	22
parser.py	98	13	86%	41–42, 52–53, 63, 70, 74, 82–83, 108–109, 121, 134
gui/_renderers/_markdown
__init__.py	18	1	94%	30
blocproc.py	35	4	88%	48–49, 67–68
control.py	11	0	100%
factory.py	14	0	100%
postproc.py	19	0	100%
preproc.py	122	6	95%	23, 103, 132, 136, 140, 199
gui/builder
__init__.py	4	0	100%
_api_generator.py	59	6	89%	25, 43, 45, 51, 70, 73
_context_manager.py	13	1	92%	17
_element.py	95	10	89%	24, 49–50, 65, 73, 78, 160, 171, 189, 192
_factory.py	9	1	88%	22
page.py	30	9	70%	57, 61–66, 70, 72
gui/custom
__init__.py	1	0	100%
_page.py	30	12	60%	28–32, 45, 48, 52, 63, 70, 79, 86
gui/data
__init__.py	3	0	100%
array_dict_data_accessor.py	40	6	85%	47–48, 50–51, 59, 66
content_accessor.py	96	18	81%	51, 58–60, 64, 66, 73, 85–86, 91–92, 109–110, 124–126, 128, 131
data_accessor.py	72	15	79%	27, 33, 37, 43, 48, 51, 72, 74, 77, 83, 87–88, 97–99
data_format.py	4	0	100%
data_scope.py	37	2	94%	61–62
numpy_data_accessor.py	21	7	66%	30, 33–35, 40–42
pandas_data_accessor.py	240	65	72%	81, 83–92, 94, 96–103, 109, 117, 149–151, 156–159, 182, 204, 207–212, 231, 262–263, 279–280, 283, 292–294, 300–301, 311, 317–319, 353–357, 359, 364–365, 374–376, 406, 409
utils.py	68	37	45%	22, 53–54, 57–59, 82, 98, 118–119, 121–125, 127–133, 135, 137–140, 142–151
gui/data/decimator
__init__.py	4	0	100%
lttb.py	38	2	94%	53, 56
minmax.py	25	1	96%	45
rdp.py	64	5	92%	85, 112–114, 140
scatter_decimator.py	50	13	74%	62–63, 77–83, 87–90
gui/extension
__init__.py	2	0	100%
library.py	139	18	87%	27–28, 62, 71, 75, 81, 132, 141, 143, 176, 240, 270, 343, 349, 351, 364, 412, 422
gui/utils
__init__.py	22	0	100%
_adapter.py	117	54	53%	39–40, 44–45, 48–55, 58–70, 82, 87, 92, 98–100, 103–108, 116, 126–133, 141–145, 149, 151, 153
_attributes.py	30	7	76%	16, 31–32, 39–40, 56–57
_bindings.py	50	1	98%	20
_evaluator.py	228	42	81%	22, 124–126, 149, 151, 161, 169–170, 199–201, 244–248, 250–260, 270, 290–293, 295–298, 301, 313, 326–327, 338
_locals_context.py	50	2	96%	55, 62
_map_dict.py	66	5	92%	34, 55, 70–71, 73
_runtime_manager.py	13	2	84%	17, 30
_variable_directory.py	100	8	92%	47–50, 61, 72, 84, 88
boolean.py	9	1	88%	25
chart_config_builder.py	138	12	91%	23, 91–92, 107, 154–155, 195, 198–200, 266, 268
clientvarname.py	14	2	85%	23, 26
datatype.py	11	3	72%	20, 22, 24
date.py	17	3	82%	27, 30–31
expr_var_name.py	13	1	92%	23
filename.py	12	0	100%
filter_locals.py	5	0	100%
get_imported_var.py	11	0	100%
get_module_name.py	13	0	100%
get_page_from_module.py	5	1	80%	19
getdatecolstrname.py	8	1	87%	21
html.py	9	1	88%	22
is_debugging.py	3	1	66%	17
is_port_open.py	6	0	100%
isnotebook.py	1	0	100%
proxy.py	65	65	0%	12–17, 19–22, 24, 27, 29, 36–37, 40–46, 49, 52–53, 55–60, 62–63, 70–71, 73–79, 87–88, 91–95, 97–103, 106–109, 111–115
singleton.py	7	0	100%
table_col_builder.py	48	19	60%	21, 29, 38–40, 42, 52–54, 56, 59–61, 63, 66–68, 70, 116
types.py	149	31	79%	49–50, 104–110, 138–141, 144–146, 185–186, 195–199, 201–208
varnamefromcontent.py	6	1	83%	17
gui_core
_GuiCoreLib.py	34	0	100%
__init__.py	1	0	100%
_adapters.py	82	38	53%	59–64, 100–101, 103, 113, 116–129, 133, 146–147, 149, 166–172, 196–197, 199
_context.py	617	256	58%	21–22, 163, 172–173, 199, 203–204, 217, 235–236, 245–251, 253–254, 257–260, 267–268, 271–276, 288, 315–319, 322–328, 330–339, 354, 356–359, 361–362, 371–375, 377, 382, 384–387, 389, 392, 395, 399–410, 415, 426, 438–439, 441–446, 448, 452, 461, 467, 483–490, 495–498, 501–503, 513–517, 524–525, 532–534, 541–542, 550–553, 573–574, 580, 592–593, 604–605, 615, 625–626, 631, 643, 645–646, 653, 659, 662–665, 668–671, 685–687, 689, 693, 699, 706–712, 714–715, 724–725, 730, 733, 736, 743–750, 757–760, 766–769, 783, 803–804, 827–831, 835–836, 838–850, 852, 856–857, 864–867, 886–888, 904, 907–908, 925–926, 930–941, 944–951, 955–959
_init.py	5	0	100%
logger
__init__.py	0	0	100%
_taipy_logger.py	22	2	90%	30–31
rest
__init__.py	4	0	100%
_init.py	1	0	100%
app.py	20	0	100%
extensions.py	3	0	100%
rest.py	7	1	85%	45
setup.py	23	23	0%	11–13, 15, 17, 19–20, 22–27, 30, 32–38, 40, 43
version.py	9	0	100%
rest/api
__init__.py	2	0	100%
error_handler.py	49	1	97%	88
views.py	64	4	93%	165–166, 212–213
rest/api/exceptions
__init__.py	0	0	100%
exceptions.py	9	0	100%
rest/api/middlewares
__init__.py	0	0	100%
_middleware.py	14	1	92%	34
rest/api/resources
__init__.py	7	0	100%
cycle.py	49	0	100%
datanode.py	87	3	96%	465, 584, 603
job.py	70	4	94%	201–204
scenario.py	65	0	100%
sequence.py	66	1	98%	293
task.py	65	1	98%	202
rest/api/schemas
__init__.py	7	0	100%
cycle.py	10	0	100%
datanode.py	63	0	100%
job.py	12	0	100%
scenario.py	12	0	100%
sequence.py	10	0	100%
task.py	11	0	100%
rest/commons
__init__.py	0	0	100%
apispec.py	51	9	82%	35, 49, 84, 87, 91, 98–100, 103
encoder.py	13	6	53%	22–25, 27–28
pagination.py	14	14	0%	12, 14, 16–17, 20–23, 26–29, 36, 44
to_from_model.py	8	0	100%
templates/default/{{cookiecutter.__root_folder_name}}
requirements.txt	1	0	100%
{{cookiecutter.__main_file}}.py	0	0	100%
templates/default/{{cookiecutter.__root_folder_name}}/algorithms
__init__.py	1	1	0%	12
algorithms.py	1	0	100%
templates/default/{{cookiecutter.__root_folder_name}}/configuration
__init__.py	1	1	0%	12
config.py	3	2	33%	18, 20
templates/default/{{cookiecutter.__root_folder_name}}/pages
__init__.py	1	1	0%	12
root.py	3	2	33%	19, 21
templates/default/{{cookiecutter.__root_folder_name}}/pages/page_example
page_example.md	0	0	100%
page_example.py	3	2	33%	19, 21
templates/default/{{cookiecutter.__root_folder_name}}/sections
import.txt	1	0	100%
page_content.txt	0	0	100%
templates/scenario-management/{{cookiecutter.__root_folder_name}}
.taipyignore	0	0	100%
templates/scenario-management/{{cookiecutter.__root_folder_name}}/algos
__init__.py	1	1	0%	12
algos.py	3	1	66%	15
templates/scenario-management/{{cookiecutter.__root_folder_name}}/config
__init__.py	0	0	100%
config.py	9	7	22%	17, 22–25, 31, 34
config.toml	18	9	50%	3, 15–17, 19–22, 24
config_with_toml.py	4	4	0%	12, 15–17
templates/scenario-management/{{cookiecutter.__root_folder_name}}/pages/job_page
__init__.py	1	1	0%	12
job_page.py	2	1	50%	12
templates/scenario-management/{{cookiecutter.__root_folder_name}}/pages/scenario_page
__init__.py	1	1	0%	12
data_node_management.py	14	12	14%	20–21, 26, 34–35, 42–43, 46–50
scenario_page.py	11	10	9%	12, 17–21, 23, 26–27, 30
TOTAL	17756	2166	87%

[jrobinAV]

I am concerned about the orchestrator.cancel_job() method.

If we remove the lock on the job dispatcher, we can be in a strange state:
Let's consider a job A. We can start canceling A by removing some subsequent jobs from the blocked_job list. Then in parallel, A is enqueued to be executed. Then finally we continue canceling A trying to remove it from the jobs_to_run queue (it is not in the queue anymore).

What do you think?

[trgiangdo]

We keep the lock in the orchestrator.cancel_job(). We can only remove locks from method that interact with Queue

[trgiangdo]

The except here is wrong because in incase the last job has been removed, the job.id in the warning will be the previous job id

[toan-quach]

I noticed you're focusing only on locks used in dispatcher, what about the ones in orchestrator like submit function?

[trgiangdo]

For some code blocks using a lock in the _orchestrator.py, they are either:

1. Executing multiple commands, not just get and put from queue or
2. Doesn't interact with Queue, which is not thread-safe.

Either case, the lock is needed there

[jrobinAV]

We keep the lock in the orchestrator.cancel_job(). We can only remove locks from method that interact with Queue

I believe we cannot remove the lock at all because when canceling a job or when failing subsequent jobs, we do instantiate a new queue

    @classmethod
    def _remove_jobs_to_run(cls, jobs):
        new_jobs_to_run = Queue()

That means we cannot rely on the locking mechanism of a queue instance since we may have multiple instances.

[jrobinAV]

Well spotted. We cannot log the job id!

However, I would still except all Exceptions, just to be sure that we keep running if any other exception is raised.

Suggested change

	except Empty:
	pass
	except Exception:
	pass

[trgiangdo]

After more investigation, I believe the mechanism when using _Orchestrator.lock and Queue.lock are not working properly before

First:

• _Orchestrator.lock is a multiprocessing.Lock which works between processes
• The Queue.lock is a threading.lock which works between threads, which is what we actually needed here to properly communicate between the orchestrator thread and dispatcher thread.

Second, all _remove_jobs_to_run() calls are also in the _Orchestrator.lock context, so:

• If the dispatcher also uses the _Orchestrator.lock, the dispatcher can not access the Queue when the orchestrator is using the lock.
• By removing the lock from the dispatcher, the dispatcher can still access the queue when the orchestrator is inside its lock context, but can not access the queue at the same time as the orchestrator since they are handling the same lock of the same queue. This can cause issue for sure.

Initializing a new Queue in __remove_jobs_to_run() is also an interesting point and may cause issue.

However, we were still suffer from these problem with the previous implementation since the _Orchestrator.lock is a multiprocess lock.

In conclusion, I propose to replace the _Orchestrator.lock by a threading.lock, and then use that lock in the dispatcher.

[toan-quach]

Interesting findings!! Didn't think about that before!

[github-actions]

☂️ Python Coverage

current status: ✅

Overall Coverage

Lines	Covered	Coverage	Threshold	Status
17768	15662	88%	0%	🟢

New Files

No new covered files...

Modified Files

File	Coverage	Status
taipy/core/_orchestrator/_dispatcher/_job_dispatcher.py	89%	🟢
TOTAL	89%	🟢

updated for commit: 82ef3d0 by action🐍

[trgiangdo]

Accessing the _Orchestrator.jobs_to_run Queue without the lock is not safe since the jobs_to_run Queue can be altered