Detailed Results of SA

Table 1. Sizes of constrained covering arrays generated by different constraint handlers

	Verify	Solver	Tolerate	Replace	P-value
Apache	34.47	34.37	34.73	34	0.06617
Bugzilla	16.47	16.47	16.43	17.17	6.1e-07
GCC	25.13	24.97	20.8	23.2	5.9e-16
SPIN-S	21.3	21.33	20.97	23.6	2.6e-13
SPIN-V	35.8	35.83	36.4	42.47	<2e-16
M1	43	43.13	43.03	42.67	0.30603
M2	30.9	30.37	30.73	31.97	2.8e-10
M3	18	18	18	18.87	<2e-16
M4	24	24.13	20.5	22.53	3.0e-07
M5	55.23	55.7	56.17	54.77	0.01741
M6	24.03	24	24.07	24.3	0.00302
M7	9	9	9	10.2	<2e-16
M8	46.2	45.7	46.27	46.93	0.00332
M9	20	20	20	21.07	<2e-16
M10	63.27	63.17	51.03	49.73	<2e-16
M11	65.6	63.57	47.17	48.43	<2e-16
M12	44.23	43.7	44.4	43	0.00003
M13	37.73	38.03	37.97	37.93	0.62026
M14	39.33	39.93	39.63	39.5	0.22753
M15	89.1	89.03	54.43	34.53	<2e-16
M16	24.2	24.3	24.4	24.97	0.00001
M17	43.2	42.57	43.07	42.77	0.16046
M18	51.77	50.43	48.73	47.7	4.2e-07
M19	100.23	99.23	62.73	56.5	<2e-16
M20	62.07	61.43	62.27	61.7	0.13805
M21	37.57	37.6	38.6	39.5	3.5e-13
M22	38.57	39.73	36.03	36	0.00097
M23	12.07	12.03	12.17	13.73	<2e-16
M24	51	49.97	48.4	47.83	1.1e-07
M25	56.37	55.83	56.3	56.33	0.64163
M26	32.37	32.67	32.37	35.57	<2e-16
M27	43.43	44.37	36.17	36.93	4.2e-14
M28	105.43	108.2	62.9	59.4	<2e-16
M29	31.07	31.1	31.17	30.2	5.2e-07
M30	19.37	19.3	19.6	21.17	<2e-16

Table 2. Computational cost (seconds) of the generation algorithm with different constraint handlers

	Verify	Solver	Tolerate	Replace	P-value
Apache	258.93	337.96	314.47	252.21	8.6e-08
Bugzilla	26.63	61.96	29.73	21.6	5.8e-12
GCC	490.74	631.88	433.4	528.03	3.8e-07
SPIN-S	4.04	35.44	4.27	1.93	<2e-16
SPIN-V	46.01	110.72	49.53	24.74	<2e-16
M1	113.22	237.44	134.16	98.02	<2e-16
M2	105.2	243.23	130.07	97.01	<2e-16
M3	3.81	23.88	4.35	10.25	<2e-16
M4	19.8	57.74	40.96	26.78	1.1e-12
M5	444.84	624.32	827.98	389.47	<2e-16
M6	75.26	164.11	144.92	67.33	<2e-16
M7	9.55	34.93	11.73	10.39	<2e-16
M8	173.91	288.3	330.49	145.12	<2e-16
M9	46.65	113.94	83.73	39.3	<2e-16
M10	385.91	934.19	576.06	260.77	<2e-16
M11	76.5	150.01	206.75	99.08	<2e-16
M12	223.63	340.29	487.97	223.4	<2e-16
M13	149.89	223.21	376.56	146.21	<2e-16
M14	73.73	129.64	145.55	74.12	<2e-16
M15	29.26	83.05	48.54	42.36	<2e-16
M16	104.11	230.79	114.33	60.78	<2e-16
M17	210.9	358.43	296	187.74	<2e-16
M18	311.19	479.06	290.68	199.86	<2e-16
M19	719.88	1166.78	705.71	507.66	<2e-16
M20	491.35	1122.11	462.42	313.09	<2e-16
M21	81.67	242.63	85.83	63.65	<2e-16
M22	47.91	102.49	91.75	65.5	9.6e-14
M23	9.01	38.64	9.87	9.12	<2e-16
M24	222.8	358.02	212.82	149.68	<2e-16
M25	231.52	358.29	289.03	220.97	<2e-16
M26	98.7	177.92	116.93	85.47	<2e-16
M27	28.76	76.11	62.53	43.78	<2e-16
M28	539.24	734.48	699.36	505.12	3.5e-15
M29	217.78	304.04	250.64	190.34	4.2e-11
M30	54.5	99.71	55.72	38.56	<2e-16

Table 3. Proportions of failures detected by covering arrays generated by different constraint handlers

	k = 3					k = 4					k = 5					k = 6
	Verify	Solver	Tolerate	Replace	P-value	Verify	Solver	Tolerate	Replace	P-value	Verify	Solver	Tolerate	Replace	P-value	Verify	Solver	Tolerate	Replace	P-value
Apache	0.97	0.97	0.97	0.97	0.18406	0.84	0.85	0.85	0.83	0.35088	0.58	0.6	0.6	0.59	0.69715	0.36	0.35	0.36	0.36	0.76375
Bugzilla	0.92	0.91	0.91	0.91	0.82672	0.64	0.65	0.63	0.66	0.05945	0.38	0.38	0.38	0.38	0.94569	0.2	0.19	0.22	0.2	0.03693
GCC	0.96	0.96	0.95	0.96	0.01190	0.78	0.79	0.71	0.76	1.9e-08	0.54	0.51	0.46	0.48	0.00002	0.3	0.3	0.27	0.26	0.00250
SPIN-S	0.83	0.83	0.83	0.85	0.22273	0.52	0.52	0.5	0.54	0.00569	0.27	0.26	0.26	0.28	0.21630	0.15	0.14	0.14	0.16	0.06977
SPIN-V	0.9	0.91	0.91	0.94	7.4e-07	0.68	0.69	0.68	0.72	0.00008	0.46	0.47	0.46	0.5	0.00152	0.22	0.22	0.23	0.28	3.0e-07
M1	0.97	0.97	0.97	0.97	0.22089	0.81	0.83	0.84	0.83	0.02953	0.6	0.6	0.59	0.58	0.34225	0.38	0.4	0.39	0.39	0.59095
M2	0.96	0.96	0.96	0.96	0.73359	0.81	0.8	0.8	0.8	0.78375	0.53	0.55	0.53	0.53	0.35576	0.31	0.32	0.31	0.33	0.13488
M3	0.88	0.88	0.89	0.88	0.60501	0.68	0.66	0.67	0.68	0.43912	0.41	0.4	0.38	0.39	0.18870	0.22	0.22	0.23	0.23	0.51655
M4	0.93	0.93	0.9	0.92	0.00002	0.75	0.76	0.71	0.73	0.00136	0.5	0.48	0.44	0.45	0.00050	0.26	0.27	0.22	0.26	0.00059
M5	0.98	0.98	0.98	0.98	0.96393	0.9	0.91	0.9	0.91	0.24047	0.69	0.7	0.7	0.69	0.84651	0.44	0.45	0.45	0.44	0.32689
M6	0.96	0.97	0.97	0.96	0.36850	0.81	0.82	0.82	0.81	0.71544	0.57	0.59	0.57	0.58	0.44581	0.37	0.36	0.37	0.36	0.40781
M7	0.83	0.82	0.81	0.84	0.01024	0.58	0.57	0.56	0.61	0.00001	0.33	0.31	0.3	0.33	0.00726	0.18	0.17	0.17	0.19	0.10339
M8	0.98	0.98	0.98	0.98	0.44400	0.88	0.87	0.87	0.88	0.34103	0.65	0.65	0.65	0.65	0.97934	0.42	0.42	0.41	0.43	0.33228
M9	0.94	0.93	0.93	0.93	0.39815	0.76	0.74	0.76	0.76	0.15838	0.58	0.58	0.59	0.58	0.85743	0.34	0.34	0.35	0.36	0.19537
M10	0.98	0.98	0.97	0.97	0.11145	0.91	0.91	0.87	0.88	2.1e-09	0.69	0.71	0.67	0.65	0.00003	0.5	0.49	0.43	0.43	2.7e-09
M11	0.98	0.98	0.97	0.98	0.28497	0.87	0.86	0.85	0.84	0.06503	0.7	0.71	0.63	0.64	1.8e-11	0.44	0.43	0.39	0.37	9.4e-07
M12	0.97	0.97	0.98	0.97	0.19719	0.87	0.87	0.87	0.86	0.35506	0.65	0.63	0.66	0.64	0.13873	0.45	0.46	0.47	0.43	0.05362
M13	0.98	0.98	0.98	0.98	0.89671	0.87	0.86	0.85	0.86	0.06466	0.64	0.62	0.63	0.62	0.64068	0.39	0.37	0.38	0.38	0.51428
M14	0.96	0.97	0.98	0.97	0.05043	0.85	0.84	0.84	0.84	0.74705	0.59	0.58	0.56	0.56	0.00371	0.36	0.37	0.35	0.35	0.15226
M15	0.99	0.99	0.97	0.95	<2e-16	0.93	0.93	0.88	0.78	<2e-16	0.8	0.8	0.67	0.53	<2e-16	0.58	0.58	0.45	0.34	<2e-16
M16	0.96	0.96	0.95	0.95	0.03947	0.81	0.81	0.82	0.81	0.74727	0.61	0.59	0.62	0.62	0.04977	0.38	0.38	0.39	0.4	0.41660
M17	0.98	0.98	0.98	0.99	0.04906	0.88	0.87	0.87	0.87	0.67585	0.68	0.68	0.67	0.67	0.98419	0.42	0.43	0.42	0.41	0.12493
M18	0.98	0.98	0.98	0.98	0.97044	0.86	0.86	0.85	0.86	0.55374	0.65	0.64	0.64	0.63	0.08496	0.4	0.39	0.38	0.37	0.09155
M19	0.99	0.99	0.99	0.98	0.00001	0.96	0.96	0.91	0.91	<2e-16	0.83	0.82	0.71	0.68	<2e-16	0.63	0.62	0.49	0.48	<2e-16
M20	0.99	0.98	0.98	0.99	0.09459	0.9	0.89	0.89	0.9	0.52567	0.69	0.71	0.71	0.69	0.41416	0.5	0.5	0.5	0.51	0.81701
M21	0.98	0.98	0.98	0.98	0.61764	0.88	0.87	0.89	0.88	0.40496	0.68	0.68	0.68	0.68	0.92577	0.46	0.44	0.46	0.46	0.17985
M22	0.97	0.98	0.98	0.97	0.92719	0.86	0.87	0.85	0.84	0.05661	0.66	0.66	0.65	0.63	0.02766	0.41	0.41	0.4	0.39	0.43635
M23	0.85	0.84	0.85	0.86	0.27288	0.6	0.59	0.59	0.61	0.50156	0.37	0.37	0.35	0.39	0.01388	0.19	0.19	0.18	0.22	0.00051
M24	0.98	0.99	0.99	0.99	0.45669	0.89	0.88	0.88	0.89	0.60541	0.7	0.69	0.67	0.67	0.03133	0.46	0.46	0.43	0.42	0.00172
M25	0.98	0.98	0.98	0.98	0.79822	0.86	0.83	0.86	0.86	0.00507	0.64	0.64	0.64	0.64	0.98362	0.43	0.42	0.43	0.41	0.32760
M26	0.97	0.97	0.97	0.97	0.73766	0.82	0.83	0.83	0.85	0.00317	0.58	0.57	0.58	0.6	0.08771	0.35	0.34	0.36	0.37	0.16406
M27	0.96	0.97	0.95	0.96	0.01134	0.82	0.82	0.8	0.8	0.01005	0.6	0.61	0.54	0.56	1.6e-06	0.36	0.35	0.32	0.33	0.00731
M28	0.99	0.99	0.98	0.99	1.6e-09	0.96	0.96	0.92	0.91	<2e-16	0.85	0.87	0.75	0.73	<2e-16	0.58	0.61	0.44	0.42	<2e-16
M29	0.98	0.97	0.97	0.97	0.24315	0.82	0.83	0.84	0.81	0.07779	0.54	0.57	0.55	0.53	0.04200	0.32	0.32	0.33	0.32	0.47765
M30	0.92	0.92	0.93	0.94	0.03434	0.72	0.71	0.71	0.72	0.64686	0.45	0.45	0.46	0.48	0.09140	0.27	0.27	0.3	0.29	0.02328