forked from rsyslog/rsyslog
-
Notifications
You must be signed in to change notification settings - Fork 0
/
ChangeLog
15314 lines (15280 loc) · 861 KB
/
ChangeLog
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
----------------------------------------------------------------------------------------
Scheduled Release 8.2112.0 (aka 2021.12) 2021-12-??
- 2021-10-27: ommysql: fix threading bug
When the MariaDB connection was (re)established, old or NULL handle
could be used. This is fixed now.
We need to synchronize access to the mysql handle, because multiple threads
use it and we may need to (re)init it during processing. This could lead to
races with potentially wrong addresses or NULL accesses. If this really
matters mostly depends on the MariaDB/MySQL client library. It looks like
they guard against fatal failuers. Anyhow, logging errors inside rsyslog
could happen in any case.
- 2021-10-25: testbench: false positive when impstats was not built
Test omfwd_fast_imuxsock failed when impstats was not built. This
has been corrected, test is now only executed when impstats is
present.
- 2021-10-25: imtcp: add support for permittedPeers setting at input() level
The permittedPeers settig was actually forgotten during the refactoring
of TLS input() level settings. This functionality is now added.
closes: https://github.com/rsyslog/rsyslog/issues/4706
----------------------------------------------------------------------------------------
Scheduled Release 8.2110.0 (aka 2021.10) 2021-10-19
- 2021-10-13: config bugfix: global(security.abortonidresolutionfail=) did not work
when used with rscript based configuration, it was not checked.
- 2021-10-13: config bugfix: global param $privDropToUser did not work correctly
The parameter was not implemented for rscript based configuration and
did not properly apply to legacy configuration. In essence, it almost always
did not work as expected.
see also: https://github.com/rsyslog/rsyslog/issues/4642
see also: https://github.com/rsyslog/rsyslog/commit/cbcaf2c7e5b67e5465e47bc7cc67af2eae47bd31
- 2021-10-12: rscript bugfix: ruleset called async when ruleset had queue.type="direct"
The call rscript statement is able to call a rule set either synchronously or
asynchronously. We did this, because practice showed that both modes
are needed. For various reasons we decided to make async
calls if the ruleset has a queue assigned and sync if not.
To know if a "queue is assigned" we just checked if queue parameters were
given. It was overlookeded the case of someone explicitly specifying a
"direct queue", aka "no queue". As such, queue="direct" triggered async
calls. That in turn meant that when a write operation to a variable was
made inside that rule set, other rulesets could or could not see the
write. While if was often not seen, this was a data race where the
change could also be seen by the outside.
This is now fixed. No matter if queue.type="direct" is specified or
left out, the call will always by synchronous. Any values written to
variables will also be seen by the "outside world" in later processing
stages.
Note that this has some potential to BREAK EXISTING CONFIGURATIONS.
We deem this acceptable because:
1. this was racy at all, so unexpected behaviour could alwas occur
2. it is actually unlikely that someone used the triggering conditions
in practice. But we can not outrule this, especially when the
configuration was auto-generated.
Potential compatibility issues can be solved by defining a small
array-memory queue on the ruleset in question instead of specifying
direct type.
Again, we expect that almost all users will never experience any
problems. If you do, however, please let us know: we may add an
option to re-enable the bug.
- 2021-10-12: ksi bugfix: locking bug fixed in rsksiCtxOpenFile
Thanks to Taavi Valjaots for the patch.
- 2021-10-11: core bugfix: fix typo in error message
Thanks to github user jkschulz for the patch.
- 2021-10-11: tcpsrv bugfix: compilation without exceptions
tcpsrv.c:992:1: error: label at end of compound statement
finalize_it:
^~~~~~~~~~~
Quoting from pthread.h:
pthread_cleanup_push and pthread_cleanup_pop are macros and must always
be used in matching pairs at the same nesting level of braces.
Amends commit bcdd220142ec9eb106550195ba331fd114adb0bd.
Thanks to Orgad Shaneh for the patch.
- 2021-10-11: mkubernetes bugfix: no connection retry to kubernetes APP
When connection to the kubernates API was not possible, mmkubernetes
did not retry. This does now happen via regular rsyslog retry
mechanism.
Thanks to github user jayme-github for the analysis and patch.
closes https://github.com/rsyslog/rsyslog/issues/4669
- 2021-10-11: openssl bugfix: Correct gnutlsPriorityString (custom ciphers) behaviour
- Only apply default anon ciphers if gnutlsPriorityString is NULL and
Authentication Mode is set to anon. Otherwise we do not set them
as they overwrite custom Ciphers.
- Added two tests for custom cipher configuration (anon/certvalid mode).
- Add call for applyGnutlsPriorityString if gnutlsPriorityString changes.
- Merged openssl init code from Connect into osslInitSession
closes: https://github.com/rsyslog/rsyslog/issues/4686
- 2021-10-11: build issue: handle undefined MAXPATHLEN, PATH_MAX
While we handled missing PATH_MAX, we did not handle missing MAXPATHLEN.
This happens under GNU/Hurd, because there is no official limit. However,
extremely long pathes are extremely uncommon, so we do not want to
use slow dynamic alloc each time we need to build pathes. So we
impose a limit of 4KiB, which should be fairly enough. Note that
this obviously increases stack requirements in GNU/Hurd.
As suggested by Michael Biebl, we have now implemented a generic
approach to handle this via autoconf.
- 2021-09-12: openssl: extended output information on connection failure
Now includes the remote client/server IP address in the log output.
- 2021-09-12: imhttp enhancements - query parameter ingestion & basic auth support
- Basic Authentication support & tests
* configured via imhttp option "basicAuthFile". This option should be configured
to point to your htpasswd file generated via a standard htpasswd tool.
tests:
* imhttp-post-payload-basic-auth.sh
* imhttp-post-payload-basic-auth-vg.sh
- Query parameter ingestion capability & tests
use t `addmetadata` option to inject query parameters into
metadata for imhttp input.
DISTRO PACKAGERS BEWARE: NEW DEPENDENCY FOR IMHTTP:
libaprutil (libaprutil1-dev on debian'ish, apr-util-devel on Red Hat)
Thanks to Nelson Yen for the patch.
- 2021-09-07: testbench bugfix: privdrop tests under root user did not work
When running under root, the privdrop tests did not properly work. This
patch fixes the issue and skips test where necessary.
This also includes some modernization of the related tests.
closes https://github.com/rsyslog/rsyslog/issues/4619
- 2021-09-07: core/ratelimiting: fix rate limiting for already parsed messages
Rate limiting may not have worked if the considered message had already
been parsed (not having NEEDS_PARSING in msgFlags).
This affects also imuxsock in its default configuration
(useSpecialParser="true" and ratelimit.severity="1")
- 2021-09-07: core bugfix: use of property $wday terminates string
When $wday is used inside a template, all template parts after it
are ignored. For exmaple:
template(name="json_filename" type="string" string="/var/log/%$wday%.log")
would generate something like "/var/log/0" - the ".log" part would be
missing. For the same reason, $wday can not reliably checked in script
filters.
Thanks to Alain Thivillon for reporting the bug and providing an
excellent analysis, which essentiellay was exactly this fix here.
closes https://github.com/rsyslog/rsyslog/issues/4670
- 2021-09-07: core/queue bugfix: potential misadressing when queue discarded messages
When a discard mark was set, the queue was very busy and discarded messages, a
NULL pointer access could happen. Depending on circumstances, several problems
could occur, including a SEGFAULT. This is now fixed.
closes: https://github.com/rsyslog/rsyslog/issues/4437
- 2021-09-07: imdiga bugfix: iOverallQueueSize calculation could be incorrect
This issue only affects testbench and rsyslog development debugging. The active
messages counter, used for synchronizing test steps, went wrong when the queue
discarded messages on it's consumer thread. Now fixed.
- 2021-09-06: gnutls driver: SAN priority did not work correctly on server side
PrioritizeSAN was not propagated when accepting a new connection, this is now fixed.
Thanks to Attila Lakatos for the patch.
- 2021-08-24: config: implement script-equavalent for $PrivDrop* statements
closes https://github.com/rsyslog/rsyslog/issues/891
----------------------------------------------------------------------------------------
Scheduled Release 8.2108.0 (aka 2021.08) 2021-08-17
- 2021-08-16: openssl tls: Improved error message output on tls failures.
closes: https://github.com/rsyslog/rsyslog/issues/4645
- 2021-08-16: impstats: add percentile metrics tracking functionality
Brief overview:
TO configure tracking percentile metrics in rainerscript:
User would need to define:
- which percentile to track, such as [p50, p99, etc.]
- window size - note, this correlates directly with memory usage to
track the percentiles.
To track a value, user would call built-in function `percentile_observe()` in their configurations to
record an integer value, and percentile metrics would be emitted every
impstats interval.
Thanks to Nelson Yen for the patch.
- 2021-08-12: imfile: add parameter "ignoreolderthanoption"
instructs imfile not to ingest a file that has not been modified in the
specified number of seconds.
Thanks to github user yanjunli76 for the patch (submitted from Nelson Yen)
- 2021-08-10: imklog bugfix: invalid memory adressing, could cause abort
This is a regeression from commit 94c4a87. It introduced a free() call
using an object that was no longer valid (the main pointer to the
to-be-freed object) was already freed at time of use. This could
cause various issues, including a segfault.
Note: this bug was triggerred only during late phase of rsyslog
shutdown, so it did not affect regular operation.
Special thanks to github user wxiaoguang for analyzing the issue
and providing a draft fix proposal, on which this patch builds.
see also https://github.com/rsyslog/rsyslog/pull/4629
closes https://github.com/rsyslog/rsyslog/issues/4625
- 2021-08-09: imfile bugfix: deleteStateOnFileDelete missed some state files
When the log file is deleted, imfile would attempt to delete the statefile but it
was missing the file_id part of the statefile name. This means the statefiles were
only removed in the log file was less than 512 characters, because for very small
files the file ID hash is not created. This lead to some state files not being
deleted.
Thanks to pearseimperva for the patch.
- 2021-08-09: imfile bugfix: hash char invalidly added in readmode != 0
If imfile is ingesting log files with readMode set to 2 or 1, the resulting
messages all have a '#' character at the end. This patch corrects the behaviour.
Note: if some external script "supported" the bug of extra hash character at
the end of line, it may be necessary to update them.
closes https://github.com/rsyslog/rsyslog/issues/4491
- 2021-08-09: omelasticsearch bugfix: errorFile mutex was not consistently locked
Lock the file during SIGHUPs to avoid issues with concurrent accesses by
writeDataError().
Thanks to François Poirotte for the patch.
- 2021-08-09: imudp: add socket type (IPv4 vs. 6) to input name
Most importantly, the input name is used for stats counter names as
well. Previously, the same name was used for IPv4 and IPv6, so we had
two counters with an equal name. That left users puzzled.
Unfortunately, this change can potentially require changes to existing
analysis scripts, as the name is now slightly different.
closes https://github.com/rsyslog/rsyslog/issues/4364
- 2021-08-06: omfwd: add capability for action-specific TLS certificate settings
This permits to override the global definitions for TLS certificates
at the action() level.
- 2021-08-06: imfile bugfix: file handle leak if "freshStartTail" was turned on
- 2021-08-05: imtcp: permit to use different certificate files per input/action
This completes the ability to override global/default TLS settings at the imtcp
input() level. Support for using multiple CAs/Certs per Connection is now provided.
- 2021-08-04: imptcp bugfix: keep alive interval was incorrectly set
The interval was accidentally set to keep alive interval. This has been
corrected.
closes https://github.com/rsyslog/rsyslog/issues/4609
- 2021-07-08: openssl network driver bugfix: small memory leak
Fixes a static, non-growing memory leak which existed when parameter
"GnutTLSPriorityString" was used. This was primarily a cosmetic issue,
but caused some grief during development in regard to memory leak
detectors.
Note: yes, this is for openssl -- the parameter name is historical.
- 2021-07-07: psrv bugfix: abort if no listener could be started
Modules (like imtcp and imdiag) which use tcpsrv could abort or
otherwise malfunction if no listener for a specific input could
be started.
Found during implementing a new feature, no report from practice.
But could very well happen.
- 2021-07-07: mmkubernetes bugfix: apiserver error handling
- Added graceful handling of apiserver errors with unexpected responses,
i.e., anything other than 200, 404, or 429. Idea is that apiserver
transient error state will recover. We don't want mmkubernetes to miss
metadata resolution for containers that don't have cached metadata.
During these transient error states, mmkubernetes will provide basic
container file path based resolution of namespace and pod metadata for
new pods whose metadata is not yet cached. After this error state
recovers, mmkubernetes is expected to resume its metadata resolution as
expected.
- Added a unit test case for apiserver return 500 with changes to mock server
- Fixed existing unit test that was failing due to missing expected results file
- Added mmkubernetes unit tests to testbench
Thanks to Abdul Waheed for the patch (submitted from Nelson Yen).
- 2021-07-07: ommongodb bugfixes
- Fix Segmentation fault when server is down
- Add server connexion check while resuming
Thanks to Kevin Guillemot for the patch.
- 2021-06-28: omkafka improvements
- drain librdkafka queues and retry later during rsyslog restart or hup. This
re-injects messages into rsyslog's native queues.
- add statsname on per kafka instance for better visibility
- omkafka - count errors related ssl as "errors_ssl"
Thanks to Nelson Yen for the patch.
- 2021-06-23: some CI/QA improvements, Travis-CI disabled
For the time being, Travis CI is disabled because it was outdated and Travis also
changed their system. We will re-evaluate if we re-enable it. Since quite a while
the Travits tests were redundant with the rest of CI, so this does not reduce
coverage.
- 2021-06-23: omhttp bugfix: dynrestpath param in batch mode invalid
When batchmode was used, the templates could not be used to
expand dynrestpath. We are now storing the restpath param
within the batch data if we are in batch mode.
When we are in batch mode, and the restpath value changes, the
batch is submitted and reinitialized
closes: https://github.com/rsyslog/rsyslog/issues/4567
- 2021-06-17: add predefined template RSYSLOG_SyslogRFC5424Format
This is essentially the same as RSYSLOG_SyslogProtocol23Format with
a better name and a fix to remove the unnecessary LF at the end of
the message.
The different name also enables us to fix the LF issue without
any concern about backwards compatibility.
closes https://github.com/rsyslog/rsyslog/issues/4384
- 2021-06-17: impstats/bugfix: _sender_stats reports integer counter as string
Note that this introduces a small backwards incompatibility: in previous output
the field was of string type, now it is integer (as intended). We discussed this
on the mailing list and the overwhelming thought was that this is not a problem
because almost all analysis backends are able to cover that format change. This made
the bugfix essentially costmetic.
HOWEVER, if you still experience issues, please let us know. We can add an option
to provide the previous format, and just spared to do so because there was no
evidence it was needed.
----------------------------------------------------------------------------------------
Scheduled Release 8.2106.0 (aka 2021.06) 2021-06-15
NOTE: the prime new feature is support for TLS and non-TLS connections
via imtcp in parallel. Furthermore, most TLS parameters can now be overriden
at the input() level. The notable exceptions are certificate files, something
that is due to be implemented as next step.
- 2021-06-14: new global option "parser.supportCompressionExtension"
This permits to turn off rsyslog's single-message compression extension
when it interferes with non-syslog message processing (the parser
subsystem expects syslog messages, not generic text)
closes https://github.com/rsyslog/rsyslog/issues/4598
- 2021-05-12: imtcp: add more override config params to input()
It is now possible to override all module parameters at the input() level. Module
parameters serve as defaults. Existing configs need no modification.
- 2021-05-06: imtcp: add stream driver parameter to input() configuration
This permits to have different inputs use different stream drivers
and stream driver parameters.
closes https://github.com/rsyslog/rsyslog/issues/3727
- 2021-04-29: imtcp: permit to run multiple inputs in parallel
Previously, a single server was used to run all imtcp inputs. This
had a couple of drawsbacks. First and foremost, we could not use
different stream drivers in the varios inputs. This patch now
provides a baseline to do that, but does still not implement the
capability (in this sense it is a staging patch).
Secondly, we now ensure that each input has at least one exclusive
thread for processing, untangling the performance of multiple
inputs from each other.
see also: https://github.com/rsyslog/rsyslog/issues/3727
- 2021-04-27: tcpsrv bugfix: potential sluggishnes and hang on shutdown
tcpsrv is used by multiple other modules (imtcp, imdiag, imgssapi, and,
in theory, also others - even ones we do not know about). However, the
internal synchornization did not properly take multiple tcpsrv users
in consideration.
As such, a single user could hang under some circumstances. This was
caused by improperly awaking all users from a pthread condition wait.
That in turn could lead to some sluggish behaviour and, in rare cases,
a hang at shutdown.
Note: it was highly unlikely to experience real problems with the
officially provided modules.
- 2021-04-22: refactoring of syslog/tcp driver parameter passing
This has now been generalized to a parameter block, which makes it much cleaner and
also easier to add new parameters in the future.
- 2021-04-22: config script: add re_match_i() and re_extract_i() functions
This provides case-insensitive regex functionality.
closes https://github.com/rsyslog/rsyslog/issues/4429
----------------------------------------------------------------------------------------
Scheduled Release 8.2104.0 (aka 2021.04) 2021-04-20
- 2021-04-19: new contributed module imhiredis
Thanks to Théo Bertin (frikilax) for the patch.
- 2021-04-19: new built-in function get_property() to access property vars
Provides ability to evaluate a rsyslog variable using dynamically
evaluated parameters.
1st param is the rsyslog param, 2nd param is a key, can be an array
index or key string.
Useful for accessing json sub-objects, where a key
needs to be evaluated at runtime. Can be used to access arrays as well.
Thanks to Nelson Yen for contributing this module.
- 2021-04-19: mmdblookup: add support for mmdb DB reload on HUP
Thanks to Théo Bertin (frikilax) for the patch.
- 2021-04-19: script bugfix: empty array in foreach() improperly handled
When running a foreach() loop inside a ruleset, if the json array/object iterated
over is empty but valid, the foreach will make the message processing in the
ruleset abort operation, no following operation (such as actions) will be
executed after this.
Thanks to Théo Bertin (frikilax) for the patch.
- 2021-04-19: imjournal bugfixes (handle leak, empty file)
Flush the FILE* buffer before rename & fsync in order
to not end up syncing an empty file.
Also, close WorkDir on fsync in order to prevent
file descriptor leakage.
Thanks to github user gerd-rausch for the fix.
- 2021-04-06: new contributed function module fmunflatten
This commit adds a new rainerscript function to unflatten keys in a JSON tree. It
provides a way to expand dot separated fields.
<result> = unflatten(<source-tree>, <key-separator-character>);
It allows for instance to produce this: { "source": { "ip": "1.2.3.4", "port": 443 } }
from this source data: { "source.ip": "1.2.3.4", "source.port": 443 }
Thanks to Julien Thomas for the contribution.
- 2021-02-22: test bugfix: some tests did not work with newer TLS library versions
Newer versions provide TLS versions that cannot be disabled in older versions as they
are unknown there. This is solved by setting restrictions in multiple steps. For
older library versions, the final step will error out, but the other one be applied.
This permits to achieve proper test results.
closes: https://github.com/rsyslog/rsyslog/issues/4534
- some improvements to project CI
----------------------------------------------------------------------------------------
Scheduled Release 8.2102.0 (aka 2021.02) 2021-02-16
- 2021-02-15: omfwd: add stats counter for sent bytes
Thanks to John Chivian for suggesting this feature.
- 2021-02-15: omfwd: add error reporting configuration option
RSyslog on a plain TCP cannot guarantee the message delivery
without using RELP protocol. Besides that the logs may be
flooded with connection errors making the rest of messages
difficult to find. To alleviate the problem (see issue 3910),
this patch adds a configuration option that enables to reduce
the number of network errors logged and reported.
For example, if each 10th network error message should be logged,
the rsyslog configuration has to be updated as follows.
action(type="omfwd" Target="<IP_ADDR>" Port="<PORT>" Protocol="tcp" ConErrSkip="10")
Thanks to Libor Bukata for the patch.
- 2021-02-15: action stats counter bugfix: failure count was not properly incremented
In some cases the counter was not incremented, most notably with transaction-enabled
actions.
Thanks to github user thinkst-marco for the patch.
- 2021-02-15: action stats counter bugfix: resume count was not incremented
And so it always stayed at zero.
Thanks to github user thinkst-marco for the patch.
- 2021-02-15: omfwd bugfix: segfault or error if port not given
If omfwd is configured via RainerScript config format and the "port"
parameter is not given, a segfault will most likely happen on
connection establishment for TCP connections. For UDP, this is
usually not the case.
Alternatively, in any case, errors may happen.
Note that the segfault will usually happen right on restart so this
was easy to detect.
We did not receive reports from practice. Instead, we found the bug
while conducting other work.
- 2021-01-29: lookup table bugfix: data race on lookup table reload
A data race could happen when a lookup table was reloaded. We found
this while moving to newer version of TSAN, but have no matching
report from practice. However, there is a potential for this to cause
a segfault under "bad circumstances".
- 2021-01-18: testbench modernization
Bump dependency versions, use newer distro versions for some tests.
Make kafka distcheck separate to help diagnose flaky kafka tests.
- 2021-01-16: testbench: fix invalid sequence of kafka tests runs
kafka tests can not run well in parallel (mostly due to ressource
constraints on CI machines). Accidentally, this was not enforced for
one of the tests. That could lead to random failures and false positives.
- 2021-01-14: testbench: fix kafkacat issues
The kafkacat tool has an upper limit of how many messages it can send
at once. Going over that limit causes messages loss. The exact limit
seems to depend on the environment. This causes testbench false positives.
This commit fixes two related issues:
- errors during kafkacat run were not detected - this has been added
- we now have a "max messages at once" setting, after which kafkacat
is restarted for the next batch of messages. It currently is set
to 25,000 msgs per incarnation. All tests loop now to send the
required number of messages. This has been fixed at the testbench
framework level, so no need to adjust individual tests.
- 2021-01-14: testbench: fix year-dependendt clickhouse test
A test had the year value hardcoded and as such failed whenever the
year changed. This patch corrects that.
----------------------------------------------------------------------------------------
Scheduled Release 8.2012.0 (aka 2020.12) 2020-12-08
- 2020-12-07: testbench bugfix: some tests did not work in make distcheck
- certificate file missing in dist tarball
- some test cases did not properly specify path to cert file
Thanks to Michael Biebl for alerting us and providing part of
the fix.
closes https://github.com/rsyslog/rsyslog/issues/4446
- 2020-12-07: immark: rewrite with many improvements
- mark message text can now be specified
- support for rulesets
- support for using syslog API vs. regular internal interface
- support for output template system
- ability to specify is mark message flag can be set
- minor changes and improvements
- 2020-11-30: usability: re-phrase error message to help users better understand cause
see also https://github.com/rsyslog/rsyslog/issues/3910
- 2020-11-10: add new system property $now-unixtimestamp
Among others, this may be used as a monotonic counter
for doing load-balancing and other things.
Thanks to Nicholas Brown for suggesting this feature.
- 2020-11-04: omfwd: add new rate limit option
Adding new rate limit option to omfwd for rate limiting
syslog messages sent to the remote server
ratelimit.interval:
Specifies the rate-limiting interval in seconds.
Default value is 0, which turns off rate limiting.
ratelimit.burst
Specifies the rate-limiting burst in number of messages.
closes https://github.com/rsyslog/rsyslog/issues/4423
Thanks to Dinesh-Ramakrishnan for the patch.
- 2020-11-03: omfwd bug: param "StreamDriver.PermitExpiredCerts" is not "off" by default
The default behaviour of expired certificates of stream driver in TLS mode, should
have been that the see tcp transmission is closed due to expired certificates, and
error messages emited in rsyslog status. This was not the case. That in turn could
lead to permitting sessions which should not be permitted.
Thanks to Vincent Zhu for alerting us and providing a great problem analysis
closes: https://github.com/rsyslog/rsyslog/issues/4425
----------------------------------------------------------------------------------------
Scheduled Release 8.2010.0 (aka 2020.10) 2020-10-20
- 2020-10-13: gnutls TLS subsystem bugfix: handshake error handling
If the tls handshake does not immediatelly finish, gnutls_handShake is called in
doRetry handler again. However the error handling was not
complete in the doRetry handler. A failed gnutls_handShake call
did not abort the connection and properly caused unexpected
problems like in issues:
https://github.com/rsyslog/rsyslog/issues/4270
https://github.com/rsyslog/rsyslog/issues/4288
- 2020-10-13: core/msg bugfix: memory leak
There is a missing call to json_object_put(json) if the call to
jsonPathFindParent() failed. It's leaking memory. Depending on workload and config,
this leak can potentially grow large (albeit we did not see reports from practice).
Thanks to Julien Thomas for the patch.
- 2020-10-13: core/msg bugfix: segfault in jsonPathFindNext() when <root> not an object
The segfault gets happens when <bCreate> is 1 and when the <root>
container where to insert the <namebuf> key is not an object.
Here is simple reproducible test case:
// ensure we start fresh
// unnecessary if there was no previous set
unset $!;
set $! = "";
set $!event!created = 123;
Thanks to Julien Thomas for the patch.
- 2020-10-13: openssl TLS subsystem: improvments of error and status messages
Adding error logs at the ssl handshake failure scenarios.
Adding the header "nsd_ossl:" tag to these logs to identify
the origin module from which logs are generated.
Thanks to Anusha Pai G for the patch.
- 2020-10-06: add 'exists()' script function to check if variable exists
This implements a way to check if rsyslog variables (e.g. '$!path!var') is
currently set of not.
Sample: if exists($!somevar) then ...
closes https://github.com/rsyslog/rsyslog/issues/4385
- 2020-10-03: core bugfix: do not create empty JSON objects on non-existent key access
Performing a condition (eg: check for an empty string) on a subtree key that do not
exists (depth > 1 from the root container), creates an empty "parent" object.
Depending on your context, you may end up with (kind of...) annoying garbage when
producing object documents (for instance to index in ES).
Also fixes a hypothetical hang condition with an almost (?) unused plugin parameter
passing mode, for details see
https://github.com/rsyslog/rsyslog/issues/4436
closes https://github.com/rsyslog/rsyslog/issues/4430
Thanks to Julien Thomas for the patch.
- 2020-09-28: gnutls subsysem bugfix: potential hang on session closure
Some TLS servers don't reply to graceful shutdown requests "for
optimization". This results in rsyslog's omfwd+gtls client to wait
forever for a reply of the TLS server which never comes, due to shutting
down the connection with gnutls_bye(GNUTLS_SHUT_RDWR).
On systemd systems, commands such as "systemctl restart rsyslog" just
hang for 1m30 and rsyslogd gets killed upon timeout by systemd.
This is fixed by replacing the call to gnutls_bye(GNUTLS_SHUT_RDWR) by calls to
gnutls_bye(GNUTLS_SHUT_WR) which is sufficient and doesn't wait for a
server reply.
As an example, Kiwi Syslog server is known to cause this issue.
Thanks to Renaud Métrich for the patch.
- 2020-09-23: core/network bugfix: obey net.enableDNS=off when querying local hostname
Local hostname resolution used DNS queries even if the enableDNS was set to off, and
this could cause unexpected delays in the HUP signal handling if the DNS server was
not responsive.
Thanks to Samu Nuutamo for the fix.
- 2020-09-14: core bugfix: potential segfault on query of PROGRAMNAME property
A data race can happen on variable iLenProgram as it is not guarded
by the message mutex at time of query. This can lead to it being
non -1 while the buffer has not yet properly set up.
Thanks to Leo Fang for alerting us and a related
patch proposal.
replaces https://github.com/rsyslog/rsyslog/pull/4300
- 2020-09-14: imtcp bugfix: broken connection not necessariy detected
Due to an invalid return code check, broken TCP sessions could not
necessarily be detected "right in time". This can result is the loss
of one message.
closes https://github.com/rsyslog/rsyslog/issues/4227
Thanks to Leo Fang for the patch.
- 2020-09-14: new module: imhttp - http input
permits to receive log data via HTTP.
uses http library to provide http input.
user would need to configure an 'endpoint' as input, along
with a ruleset, defining how the input should be routed in
rsyslog.
Thanks to Nelson Yen for contributing this module.
- 2020-09-11: mmdarwin bugfix: potential zero uuid when reusing existing one
- fix a use-after-free variable during darwin uuid message extraction
- improve debug/output by logging uuid parse errors
Thanks to github user frikilax for the patch.
- 2020-09-10: imdocker bugfix: build issue on some platforms
An invalid variable type was used, leading to compile errors at least on
all platform that use gcc 10 and above. Otherwise, however, it looks like the
issue caused no real harm.
- 2020-09-07: omudpspoof bugfix: make compatbile with Solaris build
Thanks to Dagobert Michelsen for the patch.
- 2020-09-03: testbench fix: python 3 incompatibility
- 2020-09-02: core bugfix: segfault if disk-queue file cannot be created
When using Disk Queue and a queue.filename that can not be created
by rsyslog, the service does not switch to another queue type as
supposed to and crashes at a later step.
closes: https://github.com/rsyslog/rsyslog/issues/4282
- 2020-08-26: cosmetic: fix dummy module name in debug output
When we have optional components (like imjournal) a dummy module
is used. It's sole purpose is to emit "this module is not available".
During init, the module emitted an invalid module name into the debug
log. This has now been replaced by the generic term "dummy".
Note: it is highly unlikely that someone will ever see that message
at all, as it is unlikely for the dummy modules to be build.
see also: https://github.com/rsyslog/rsyslog/commit/84a7e3d80b80106dcc86c273ed8cf78a6c11c722#r41782830
Thanks to Thomas D. (whissi) for the patch.
- 2020-08-26: config bugfix: intended warning emitted as error
When there are actions configured after a STOP, a warning should be
emitted. In fact, an error message is generated. This prevents the
construct, which may have some legit uses in exotic settings. It
may also break older configs, but as the message is an error
for so long now, this should be no longer of concern.
----------------------------------------------------------------------------------------
Scheduled Release 8.2008.0 (aka 2020.08) 2020-08-25
- 2020-08-25: imdocker bugfix: error reporting not always correct
A wrong function to obtain the error code was used. This
could lead to invalid error messages.
Thanks to Steve Grubb for the bug report and fix proposal.
closes https://github.com/rsyslog/rsyslog/issues/4381
- 2020-08-25: imptcp: add max sessions config parameter
The max is per-instance, not global across all instances.
There is also a bugfix where if epoll failed I think we could leave a
session linked in the list of sessions, this code unlinks it.
Thank to Alfred Perlstein for the patch.
- 2020-08-24: omelasticsearch bugfix: reply buffer reset after health check
The issue happens when more than one server is defined on the
action. On that condition a health check is made through
checkConn() before sending the POST. The replyLen should be
set back to 0 after the health check, otherwise the response
data received from the POST gets appended to the end of the
last health check.
Thanks to Julien Thomas for the patch.
- 2020-08-14: omfile: do no longer limit dynafile cache size in legacy format
When using obsolete legacy config format, omfile had a hard limit of
1,000 dynafile cache entries. This does not play well with very
large installation. This limit is now removed and converted into
a warning if cache size > 25,000 is specified.
Note: the problem can easily be worked-around by using modern
config format (RainerScript).
closes: https://github.com/rsyslog/rsyslog/issues/4241
- 2020-08-13: imudp: fix very small, static memory leak
When ruleset support was used, the ruleset name was not freed upon rsyslog
termination. While this has no consequences for regular runs, it generates
leak errors under memory debuggers and as such makes debugging harder than
necessary.
Thanks to github user frikilax for the patch.
- 2020-08-13: omelasticsearch: add parameter skipPipelineIfEmpty
When POST'ing a document, Elasticsearch does not allow an empty pipeline
parameter value. This patch introduces boolean option skipPipelineIfEmpty
to the omelasticsearch action. When set to true, the pipeline parameter
won't be posted. Default is false so we do not modify current behavior.
Thanks to Julien Thomas for the patch.
- 2020-08-12: systemd service file removed from project
This was done as distros nowadays have very different service files and it no
longer is useful to provide a "generic" (sic) example.
see also: https://github.com/rsyslog/rsyslog/issues/4333
- 2020-08-11: gnutls TLS driver bugfix: EKU check not done properly
When the server accepted a new connection, it did not properly set the
dataTypeCheck field based on the listening socket. That resulted in
skipping ExtendedKeyUsage (EKU) check on the client.
Thanks to Daiki Ueno for the patch.
- 2020-08-06: MMDARWIN:: improve configuration flexibility and UUID fix
-t pu now able to get fields from local variables ($.)
- now able to configure a custom root container for mmdarwin fields
- now able to put nested keys ($!key1!key2)
- don't regenerate a UUID each time, but instead check if one exists before
creating it (allow successive calls without losing previous UUID)
Thanks to github user frikilax for the contribution.
- 2020-08-06: add --enable-imjournal=optional ./configure option
- 2020-08-06: IMPCAP::Fixes: segfault, memory and build corrections
* fix bug in ethernet packets parsing
* fix removes build error with gcc10: 'multiple definition of...'
* resolve memory leak during interface init failure (device not freed after post-create error)
* add test 'impcap_bug_ether' to prove ethernet parser fix is working
Thanks to github user frikilax for the contribution.
closes https://github.com/rsyslog/rsyslog/issues/4332
- 2020-07-14: CI: add support for github actions
- 2020-07-14: imklog: add ruleset support
see also: https://github.com/rsyslog/rsyslog/issues/4344#issuecomment-658001854
see also: https://github.com/rsyslog/rsyslog/issues/106
- 2020-07-06: config system fix: ChkDisabled method to make config.enabled work
There was wrong negation in the method so it returned 0/1 in reverse
and also it did not mark the node to not be reported as unknown at all
times which is needed after all.
Thanks to Jiri Vymazal for the patch.
----------------------------------------------------------------------------------------
Scheduled Release 8.2006.0 (aka 2020.06) 2020-06-23
- 2020-06-22: queue: permit ability to double size at shutdown
This prevents message loss due to "queue full" when re-enqueueing data
under quite exotic settings.
see also https://github.com/rsyslog/rsyslog/issues/3941#issuecomment-549765813
closes https://github.com/rsyslog/rsyslog/issues/4020
- 2020-06-22:Fixing imfile segfaulting on selinux denial
If imfile is denied access to file watched trough symlink there is
unchecked condition resulting in access to not initialized memory.
- 2020-06-22: openssl: Fixed memory leak when tls handshake failed.
closes: https://github.com/rsyslog/rsyslog/issues/4319
- 2020-06-22: change systemd service file to wait for network
now that rsyslog is usually only installed for real syslog servers,
we should assume that some network listening or forwarding happens
on start. As such we need to start a bit later, after the network.
This poses no problem as systemd nowadays comes with journal which
is in almost all cases configured to buffer log data while
rsyslog is not yet running.
see also https://github.com/rsyslog/rsyslog-pkg-rhel-centos/issues/72
- 2020-06-22: NEW INPUT MODULE:: impcap, network packets input parser
Thanks to github user frikilax for the contribution.
- 2020-06-22: ksi bugfix: Optimized code in KSI module initialization fixed.
KSI module initialization will not stuck in infinite loop when code is
built with optimization -O2.
- 2020-06-05: operatingstatefile bugfix: month was given too low
The month was printed with the range 0 (January) to 11 (December).
This has now been corrected.
closes https://github.com/rsyslog/rsyslog/issues/4292
- 2020-06-05: build system: add "optional" build functionality to some components
Nameley:
--enable-libdbi=optional
--enable-mmdblookup=optional
--enable-imkafka=optional
--enable-omkafka=optional
If used, builds a dummy module which just emits a "module not supported
on this platform" error message when loaded.
Primary use case for this system is Debian-ish builds on SUSE OBS,
where we prefer to have a single package definition for all versions
(else things get much more complicated).
- 2020-05-23: config system bugfix: backticks cat segfault if file cannot be opened
when a `cat <filename>` construct is used in rsyslog.conf and <filename> can not
be accessed (does not exist, no permissions, ...), rsyslog segfaults.
Thanks to Michael Skeffington for notifying us and providing root cause analysis.
closes https://github.com/rsyslog/rsyslog/issues/4290
- 2020-05-15: imtcp bugfix: octet framing/stuffing problem with discardTruncatedMsg on
When "discardTruncatedMsg" was enabled in imtcp, messages were incorrectly
skipped if the last character before the truncation was the LFdelimiter.
Also adds two testbench tests for this case.
closes: https://github.com/rsyslog/rsyslog/issues/4281
- 2020-05-12: ompipe bugfix: race during HUP
When HUP was received, the write mutex was not aquired. This could
lead to unexpected invalidation of the output file descriptor.
Thanks to Julien Thomas for alerting us on this issue.
see also https://github.com/rsyslog/rsyslog/pull/4136#issuecomment-578326278
- 2020-05-12: ompipe: add action parameter tryResumeReopen
Sometimes we need to reopen a pipe after an ompipe action gets
suspended. Sending an HUP signal to rsyslog does the job but requires
an interraction with rsyslog. The patch adds support for a new boolean
option, tryResumeReopen, for the ompipe action. It mimics what an HUP
signal would do.
Thanks to Julien Thomas for the patch.
- 2020-05-12: imjournal: remove strcat call
Thanks to Jeff Marckel for the patch.
- 2020-05-12: build system: libzcmq version requirement needs to be bumped
Thanks to Thomas Deutschmann for pointing this out.
closes https://github.com/rsyslog/rsyslog/issues/3957
- 2020-05-12: testbench: download ElasticSearch binaries from rsyslog.com
The official ElasticSearch download site sometimes denies the download.
- 2020-05-11: openssl netstream driver bugfix: context leak
The context object was not properly freed.
Thanks to Michael Zimmermann for the fix.
- 2020-05-11: omhttp: Add support for multiple http headers
Allows the inclusion of multiple http headers on the REST call.
Thanks to callmegar for the patch.
- 2020-04-29: core bugfix: group id could not be obtained for very large groups
Thanks to github user emilbart for the patch.
- 2020-04-29: testbench additions (relp broken connection test)
- 2020-04-29: omudpspoof bugfix: issues with oversized messages
First issue was an incorrect packet length in UDP Header. It has to be the FULL UDP Packet
regardless of the MTU Setting. As a result regardless of IP fragmentation, the MTU setting
also limited the siizmax size of the UDP message.
The second issue was incorrect calculation of the UDP Checksum with libnet if
IP fragmentation was used (Based on MTU Setting). As a result, the network packets were
dropped by the tcp stack before they even could reach there target. The workarround for this
problem is, that we set the UDP Checksum to 0x0000 which allows skipping of the checksum
test. Fixing the problem by calculating the correct UDP Checksum would require some
code changes in the libnet.
Also fixed the omudpspoof bigmsg test and increased the testing size to 16KB.
- 2020-04-29: omprog: fix assert failed on HUP with output flag
If the 'output' setting of omprog was used and rsyslog received a HUP
signal just after starting (and before the omprog action received the
first log to process), an internal assertion could fail, causing
rsyslog to terminate. The failure message was "rsyslogd: omprog.c:660:
closeOutputFile: Assertion `pCtx->bIsRunning' failed."
The failure could also occur if rsyslog received a HUP signal during
the shutdown sequence.
This bug was introduced in v8.2004 by PR https://github.com/rsyslog/rsyslog/pull/4255
Although a test already existed that checked the interaction of HUPs
with the 'output' setting, it didn't always fail in this particular case
due to timing conditions. The test has been improved to cover this case
more reliably.
Thanks to Joan Sala Isern for the patch.
----------------------------------------------------------------------------------------
Scheduled Release 8.2004.0 (aka 2020.04) 2020-04-28
- 2020-04-28: ksi bugfix: When KSI module is suddenly closed, files are finalized
In async. mode all pending signature requests are closed immediately and
unsigned block marker is attached with message about sudden closure.
Similar approach is used for blocks that already contain some records.
Empty blocks are just closed without any metadata.
Thanks to Taavi Väljaots for the patch.
- 2020-04-28: ksi bugfix: Signer thread initialization is verified before usage.
When signer thread is created in rsksiInitModule thread successful
initialization is verified before returning the function. This will
prevent adding records to not initialized module and in case of an
error signature files opened will contain only magic bytes.
Thread flags replaced with thread state.
When init module fails, module is disabled.
Thanks to Taavi Väljaots for the patch.
- 2020-04-28: ksi bugfix: Hardcoded default hash algorithm replaced with 'default'
Instead of hardcoded SHA-256 KSI_getHashAlgorithmByName("default")
is used to get default hash function.
Function rsksiSetHashFunction and SetCnfParam updated.
Thanks to Taavi Väljaots for the patch.
- 2020-04-28: imfile bugfix: poential segfault in stream object on file read
- if cstrLen(pThis->prevMsgSegment) > maxMsgSize then len calculation
become negative if cstrLen(thisLine) < cstrLen(pThis->prevMsgSegment)
This causes illegal access to memory location and thus causing segfault.
- assigning len = 0 if cstrLen(pThis->prevMsgSegment) > maxMsgSize so that
it access the correct memory location.
Thanks to github user jaankit
- 2020-04-28: openssl TLS drivers: made more reliable for older openssl versions
OpenSSL can retry some failed operations, but older versions need an explicit
opt-in to do so. This is now done.
- 2020-04-28: omprog: fix bad fd errors in daemon mode
When omprog was used with the 'forceSingleInstance=on' option, and/or
the 'output' setting, "bad file descriptor" errors occurred, which
prevented the external program to be executed and/or the program output
to be correctly captured. The bug could also manifest as "resource
temporarily unavailable" errors, or other errors related to the use of
invalid/reassigned file descriptors. These errors only happened when
rsyslog ran in daemon mode (i.e. they didn't happen if rsyslogd was
run with the '-n' option).
The cause of the bug was that omprog opened the pipe fds needed by
these flags during the configuration load phase (in the 'newActInst'
module entrypoint). This is a bad place since the fork of the daemon
occurs after this phase, and all fds are closed when the daemon process
is started (see 'initAll' in rsyslogd.c), hence invalidating the
previously opened fds.
To correct this, the single child process and the output capture thread
are now started later, when the first log message is received by the
first worker thread. (Note: the 'activateCnf' module entrypoint, despite
being invoked after the fork, cannot be used for this purpose, since it
is invoked per module, not per action instance.)
Currently no automated test exists for this use case since the testbench
always runs rsyslog in non-daemon mode.
Affected versions: v8.38 and later
closes: https://github.com/rsyslog/rsyslog/issues/4247
Thanks to Joan Sala Isern for the patch.
- 2020-04-28: omfile bugfix: $outchannel split log lines at rotation time
- 2020-04-17: openssl: add support for libreSSL
Disable use of "@SECLEVEL" in default cipher string and
avoid SSL_CONF_CTX_set_flags() API when LIBRESSL is used.
This means tlscommands will not work.
closes: https://github.com/rsyslog/rsyslog/issues/4210
- 2020-03-04: imudp bugfix: build problems on some Linux kernel versions
Thanks to Wen Yang for the patch.
- 2020-03-02: conf output bugfix: -o produces missing space between call and rulename
Thanks to Tetiana Ohnieva for the patch.
closes https://github.com/rsyslog/rsyslog/issues/3761
----------------------------------------------------------------------------------------
Scheduled Release 8.2002.0 (aka 2020.02) 2020-02-25
- 2020-02-25: imfile: add per minute rate limiting
Add MaxBytesPerMinute and MaxLinesPerMinute options.
These take integer values and, respectively, limit the number
of bytes or lines that may be sent in a minute.
This can be used to put a limit on the count or volume of logs
that may be sent for an imfile.
Thanks to Greg Farrell for the patch.
- 2020-02-24: core: add global parameter "security.abortOnIDResolutionFail"
This parameter controls whether or not rsyslog aborts when a name ID
lookup fails (for user and group names). This is necessary as a security
measure, as otherwise the wrong permissions can be assigned or privileges
are not dropped.
CHANGE OF BEHAVIOR
The default for this parameter is "on". In previous versions, the default
was "off" (by virtue of this parameter not existing). As such, existing
configurations may now error out.
We have decided to accept this change of behavior because of the potential
security implications.
closes https://github.com/rsyslog/rsyslog/issues/4164
- 2020-02-24: openssl TLS driver bugfix: chained certificates were not accepted
This was supported since always inside GnuTLS driver, but was missing for openssl one.
- 2020-02-24: core bugfix: too early parsing of incoming messages
In theory, rsyslog should call parsers on the queue worker threads whenever
possible. This enables the parsers to be executed in parallel. There are
some cases where parsers needs to be called earlier, namely when parsed
data is needed for rate-limiting.
The logic to do this previously did not work correctly and was fixed six
years ago (!) by b51dd22. Unfortunately, b51dd22 was overly agressive:
it actually makes the early parser call now mandatory, effectively moving
parsing to the input side where there is no to little concurrency.
We still do not need to call the parser when all messages, regardless of
severity, need to be rate-limited. This is the default and very frequent
case. This patch introduces support for this and as such makes parsers
able to run in parallel in the frequent case again.
closes https://github.com/rsyslog/rsyslog/issues/4187
- 2020-02-20: testbench bugfix: two minor issues in omkafkadynakey.sh test
lead to false positives during test runs (depending on circumstances)
closes: https://github.com/rsyslog/rsyslog/issues/4134
- 2020-02-20: testbench: set max extra data length for tcpflood from 200 to 512KiB
Added a imrelp test for big messages (256KB).
closes: https://github.com/rsyslog/rsyslog/issues/4158
- 2020-02-20: config system bugfix: 'config.enabled' directive oddities
Previously the directive was processed way too late which caused false
errors whenever it was set to 'off' and possibly other problems.
Thanks to Jiri Vymazal for the patch.
- 2020-02-09: imfile bugfix: timeout did not work on very busy system
The timeout feature was soley based on timeouts of the poll()
system call. On a very busy system, this would probably happen
very seldomly. Moreover, the timeout could occur later than
expected on any system with high load.
The issue was not reported from practice but discovered during
CI system improvements.
- 2020-01-30: build system: change --enable-imfile-tests default to "yes"
This was accidentally set to "no" some time ago (actual commit unknown). Tests for
imfile should by default run when imfile is enabled.
see also https://github.com/rsyslog/rsyslog/issues/4120
- 2020-01-27: build system: add option --enable-gnutls-tests
This enables us to build GNUtls support but not necessarily
test it in CI. This is useful for some specialised subcomponent
test. The default is enabled if gnutls is enabled and disabled if not.
- 2020-01-26: testbench: new test for loadbalancing via global vars
This is a popular functionality which had not been routinely tested
in the past.
- 2020-01-26: mmdblookup bugfix: invalid data returned when no entry found
Since the upgrade of the package libmaxminddb on FreeBSD (1.3.2_2 -> 1.4.2),
the module mmdblookup returns the first entry of the mmdb database even if the entry
is not found. After some debug, I found the solution in the official maxminddb
repository : to check if the entry is in database, we must check the found_entry
attribute, otherwise the function MMDB_get_entry_data_list will return the first
entry of the database if the entry is not found in it.
Thanks to Kevin Guillemot for the patch.
- 2020-01-23: oversize message log bugfix: do not close fd -1
The oversize message log fd is always closed on HUP, even if it never
was opened (and thus has -1 value). This patch corrects the issue.
The bug had no know-bad effect in practice other than getting an
(ignored) error status from close(). However, it introduced warnings
in test runs (e.g. when running under valgrind).
- 2020-01-22: imfile bugfix: saving of old file_id for statefiles
Previously we saved old file_id unconditionally, which led to not
deleting old statefiles if files changes without rsyslog running.
Now it should work correctly.
Thanks to Jiri Vymazal for the patch.
- 2020-01-22: imfile bugfix: misadressing and potential segfault
Commit 3f72e8c introduced an invalid memory allocation size. This lead to
too-short alloc and thus to overwrite of non-owned memory. That in turn
could lead to segfaults or other hard to find problems.
The issue was detected by our upgraded CI system. We did not receive
any problem reports in practice. Nevertheless, the problem is real and
people should update affected versions to patched ones.
The bug was present in scheduled stable release 8.1911.0 and 8.2001.0.
see also: https://github.com/rsyslog/rsyslog/issues/4120
see also: https://github.com/rsyslog/rsyslog/pull/4141
- 2020-01-20: core bugfix: potential race during HUP
when rsyslog is HUPed immediately after startup and before it is fully
initialized, there is a potential race with the list of loaded modules.
This patch ensures no bad things can happen in that case.
Detected by LLVM TSAN, not seen in practice.
- 2020-01-20: testbench improvements and fixes
modernize tests, reduce robustness against slow machines, provide some
test framework functional enhancements, and optimize some tests.
Also includes some code changes to C testing components. Among others,
tests have slightly been speeded up by reducing the wait time at queue
shutdown. This is possible because of better overall completion checks.
----------------------------------------------------------------------------------------
Scheduled Release 8.2001.0 (aka 2020.01) 2020-01-14
- 2020-01-12: core bugfix: race condition related to libfastjson when using DA queue
Rsyslogd aborts when writing to disk queue from multiple workers simultaneously.
It is assumed that libfastjson is not thread-safe.
Resolve libfastjson race condition when writing to disk queue.
see also https://github.com/rsyslog/rsyslog/issues/4041
Thanks to MIZUTA Takeshi for the fix.
- 2020-01-12: omfwd bugfix: parameter streamdriver.permitexpiredcerts did not work
closes https://github.com/rsyslog/rsyslog/issues/4098
- 2020-01-11: Bugfix: KSI module + dynafile in asynchronous mode fixed
Thanks to Taavi Valjaots for the patch
- 2020-01-08: tls driver: add support to configure certificate verify depth
Support added in omfwd as instance parameter:
streamdriver.TlsVerifyDepth
Support added in imtcp as module parameter:
streamdriver.TlsVerifyDepth
Can be 2 or higher.
Support added into ossl driver
Support added into gtls driver
Added testcases for both drivers.
closes: https://github.com/rsyslog/rsyslog/issues/4035
- 2020-01-08: modernization of testbench
moved some tests to newer standards, hardened them against slow testbench machines,
kafka component download improvements, and prevent dangling left-over test tool
instances from aborted tests
- 2020-01-07: tls subsystem bugfix: default for permitExpiredCerts was invalidly "on"
The problem occured with commit 3d9b8df in December 2018 and went into
scheduled stable 8.1901.0. Unfortunately, the change in default was not detected
until a year later. This commit re-enables the previous default ("off"), which is
also the only sensible default from a security PoV. Unfortunately, new 2019
deployments may begin to see connection rejection when usin expired certs. As
expired certs should not be used, this hopefully will not cause problems in
practice.
Thanks to Jiri Vymazal for the patch.
- 2020-01-01: testbench: improve ElasticSearch test speed
We now support re-using suitable running ES instances, which reduces the
number of restarts.
- 2019-12-31: omelasticsearch: improve curl reply buffer handling
The curl reply buffer (pWrkrData->reply) was allocated, realloced and freed with
each request. This has now been reduced to once per module, slightly increasing
overall performance.
closes https://github.com/rsyslog/rsyslog/issues/1964
- 2019-12-31: config system: emit proper error message on $ in double-quoted string
closes https://github.com/rsyslog/rsyslog/issues/2869
- 2019-12-30: core bugfix: rsyslog aborts when config parse error is detected
In defaut settings, rsyslog tries to continue to run, but some data
structures are not properly initialized due to the config parsing error.
This causes a segfault.
In the following tracker, this is the root cause of the abort:
see also https://github.com/rsyslog/rsyslog/issues/2869
- 2019-12-30: fix some alignment issues
So far, this worked everywhere (for years). But it may still have
caused issues on some platforms.
closes https://github.com/rsyslog/rsyslog/issues/2608
- 2019-12-27: core bugfix: APP-NAME fields could become empty
RFC 5424 specifies that an empty APP-NAME needs to be indicated by
"-". Instead, the field could become empty under certain conditions.
If so, outgoing 5424 messages were invalidly formatted.
This happened under quite unusual conditions, but could be seen
in practice.
closes https://github.com/rsyslog/rsyslog/issues/4043
- 2019-12-27: core bugfix: reopen /dev/urandom file descriptor after fork on Linux
This patch updates prepareBackground() in tools/rsyslogd.c to reopen any file
descriptors used for random number generation in the child process. This fixes
an issue on Linux systems where the file descriptor obtained for /dev/urandom
by seedRandomNumber() in runtime/srutils.c was left closed after the fork. This
could be observed in procfs, where /proc/fd/ would show no open descriptors to
/dev/urandom in the forked process. /dev/urandom is reopened as the child may be
be operating in a jail, and so should not continue to use file descriptors from
outside the jail (i.e. inherited from the parent process).
I found that this issue led to rsyslog intermittently hanging during seedIV()
in runtime/libgcry.c. After the fork, the closed file descriptor number tended
to get re-assigned. randomNumber() would then read from an incorrect (although
still valid) file descriptor, and could block (depending on the state of that
file descriptor). This gave rise to the intermittent hang that I observed.
Thanks to Simon Haggett for the patch.
- 2019-12-20: imdocker bugfix: did not compile without atomic operations
- 2019-12-20: omclickhouse: new parameter "timeout"
Thanks to Pavlo Bashynskiy for the patch.
- 2019-12-20: omhiredis: add 'set' mode plus some fixes
- new mode 'set' to send SET/SETEX commands
- new parameter 'expiration' to send SETEX instead of SET commands (only applicable to 'set' mode)
- fixes to missing frees
Thanks to github user frikilax for the patch.
- 2019-12-18: relp: Add support setting openssl configuration commands.
Add new configuration parameter tls.tlscfgcmd to omrelp and imrelp.