Sagan released 1.1.0

ChangeLog: https://github.com/beave/sagan/blob/master/ChangeLog#L1-L37

The Basics:

    * Sagan now "remembers" where it left off between restarts/reboots/etc.
    * You can now create rules that focus on certain IP address or IP address ranges (ie - $EXTERNAL_NET/$HOME_NET).
    * Sagan can treat "old" Bluedot IP reputation threat Intel differently than "new" threat intel.
    * We added "qdee.pl",  a SDEE poll routine to the "extra" directory.
    * A lot of bugs were fixed 

The Details:

    * Moved all "threshold", "after", "flowbits", and "client tracking" to mmap files.  This means that Sagan "remembers" between restarts where it "left off"! 
    * Introduced "tools/sagan-peek.c" which allows you to exmaine Sagan mmap files.  Useful in debugging or just "seeing" what Sagan is "tracking".
    * $EXTERNAL_NET and $HOME_NET now function as expected.  Previous versions of Sagan did not have any concept of $EXTERNAL_NET/$HOME_NET and were ignored.  Adam Hall @ Quadrant made Sagan "aware" of "traffic flow".  Values in a rule for source/destination are tested _after_ normalization.
    * Added "mdate" (modification date) and "cdate" (creation data) to Bluedot. This allows Sagan to not trigger "aged" Bluedot Threat Intel.  For example, do _not_ alert if an IP address is seen and the Intel is over X hours/days/months/years old.
    * Threholding based on 'dstport' merged,  thanks to Bruno Coudoin.  See:  https://github.com/beave/sagan/commit/44d6752acf27d61bcd57e35f930b0f6e11dadbc7
    * Added parsing for IPTables "SPT" and "DPT"t port for iptables, thanks to Bruno Coudoin.  https://github.com/beave/sagan/commit/9de9cffd224a44f93c80eca62e6ead617a4b97a6
    * Added "qdee" to the "extra" directory.  This allows Sagan to parse older style Cisco IDS output.  This polls using the SDEE protocol. See https://github.com/beave/sagan/commit/61c4a7dd611161697785c889630dd3c8333ec8b5
    * Removed support for libjsonc (json-c) and moved to libfastjson.

The Bugs Fixed:

    * Correct issue for when Sagan cannot open a file (-F/--file) due to permissions.
    * Removed unused "SigArgs" array.
    * Clean exit when Sagan cannot load Maxmind GeoIP2 data file.
    * Change "normalize: {type}" to "normalize;".  All normalization rules now come from one file.  This keeps Sagan in line with liblognorm development.
    * Sagan now "warns" the user if old style "normalize" is encountered. See: https://github.com/beave/sagan/commit/ba3de9e43bc8623b361e34ce06a2e7808e045f88 and https://github.com/rsyslog/liblognorm/issues/206
    * Fix json_object_object_get_e) compile time warnings. See: https://github.com/beave/sagan/commit/e9bdea5b7fa5b25c1d7e740a4c856c70a1046d1d
    * Minor ARM CPU fixes.
    * Various "meta_content" fixes.  When using "meta_content" with large amounts of search data would sometimes cause failures. 
    * Major bug fixes involving "client tracking".  Thanks to Adam Hall @ Quadrant Information Security!
    * Sagan now attempts to create the FIFO if it is not detected.  Thanks to Cabrol Perales.
    * A lot of smaller bug fixes.  See: https://github.com/beave/sagan/commits/master

After 5 years of development, Sagan 1.0.0 is released!

Development code: https://github.com/beave/sagan
Stable: http://sagan.io (Version 1.0.0)

In June 2010, we completed initial work on Sagan 0.0.1 which was a very basic outline of what we thought a real-time log analysis engine should be. Historically, people treated logs as an archive of only the past activities, and in 2010, many solutions for “log analysis” were based on command line tools and concepts like “grep”. This approach is fine and certainly useful, but why was real-time log analysis not really a “thing?” We never suggested getting rid of historical log search functionality, but the lack of “real time” detection was troubling; we expect some security software, like Intrusion Detections Systems (IDS) to be “real time,” so why was log analysis not treated the same way? After all, if someone told you that their solution to packet inspection was to “look at all the packets via a 'grep' every Friday,” you would laugh at them. We decided to tackle this problem because of our own selfish needs.

When we started developing Sagan, we naturally focused on our own needs at Quadrant Information Security. Since we are an MSSP (Managed Security Service Provider), we needed to be able to monitor security appliances and software similarly to how we monitored our “Snort” instances. Back in 2010, pre-Cisco/Sourcefire buy-out, not all companies were interested in Snort. They “trusted” more “mainstream” products from companies like Cisco, Sonicwall, Fortinet, etc. As much as we argued that Snort was a better IDS/IPS solution, many potential customers simply were not interested; “we're a Cisco shop, that's the way it is,” we heard this a lot.

Initial development began so that we, as an MSSP, could say “yes, we can monitor that.” At the time, that was our primary need, which meant that Sagan had to be 100% real time. It would not be reasonable for our analysts to have to “grep” logs daily in order to search for possible malicious activity. Software should be able to provide this data and do it better. To be real-time in environments with mass amounts of log data, Sagan needed to be multi-CPU aware and memory-efficient. Therefore, we designed Sagan in C using threading (pthreads). If your analysis platform has multiple CPUs and/or cores, Sagan would need to “spread” the log analysis load across them. Since our analysts already understood packet analysis via Snort rules, it made sense to have Sagan use a similar syntax, which also meant that Snort rule management software like “pulledpork” would inherently work with Sagan.

Since we were already traveling down the “very much like” Snort path in terms of design, we decided that we might as well adopt the Snort “unified2” output format, which means that Sagan can store its data in the same place that Snort does. This also meant that we can correlate log events to our packet events, and that we are out-of-the-box compatible with Snorby, BASE, Squil, etc.

Overall, those were the basic milestones we wanted to get to. As time went on, Sagan required more complexity that was not foreseen at the time of its inception (i.e., flowbits). In August of 2015, after 5 years of development, we put Sagan into a “code freeze” which means that rather than trying to add complex new features to Sagan, we focus on stability. And although Sagan has always been pretty stable, we started testing across a lot of platforms that varied in log data flow, rules enabled, and environmental complexity. In August of 2015 released “RC1” (Release candidate #1) to the public to help us test Sagan. We made it up to “RC5”, and today, October 23rd, 2015, we're proud to call this Sagan 1.0.0.

Today, Sagan is used around the world by medical companies, hospitals, banks, credit unions, financial institutions, petroleum companies, law firms, supermarket chains, telecommunications companies, accounting firms, manufacturers, hosting providers, insurance companies, colleges, universities and various law enforcement agencies. It is even used by other network and computer security companies, and these are just the organizations that we know use Sagan!

We are very proud of how far Sagan has come since its inception. Sagan is a complex piece of software that required the input and help from many people. I like to highlight that fact since Sagan would not be where it is today had it not been for all of these people willing to spend time deep in the Sagan code, and developing rules. If you have a moment, please check out the contributors via the “sagan –credits” flag or https://github.com/beave/sagan/blob/master/src/sagan-credits.c

Now that 1.0.0 is behind us, we look forward to adding some new “killer” functionality. It is going to be a really fun ride.

Sagan 1.0.0RC4

-[Feature] - 'offset', 'depth', 'distance', and 'within' support. These options function identical to the Snort options with the same names. These options allow you too parse log message content in different ways. For more information on how they work, see:

http://blog.joelesler.net/2010/03/offset-depth-distance-and-within.html

When you read Joel Esler great artcle, please keep in mind to:
s/Sagan/Snort/g
Sagan's functionality with 'offset', 'depth', 'distance' and 'within' is identifical to Snorts.

• [Feature] - "Flowbit" allow Sagan to "track" events across multiple log lines. For example, let's say that you would like Sagan to generate an alert when a Microsoft Window's server anti-virus process is stopped. However, you would not like an alert to be generate if the anti-virus is "stopped" due to a reboot. To accomplish this, you would create two rules. The first would be used to detect when a Microsoft Window system is being rebooted.
  alert syslog $HOME_NET any -> $EXTERNAL_NET any (msg: "[WINDOWS-MISC] System shutdown [FLOWBIT SET]"; content: " 1074|3a| "; program: USER32; flowbits: set, reboot.windows, 60; flowbits: noalert; classtype: system-event; reference: url,wiki.quadrantsec.com/bin/view/Main/5002014; sid: 5002014; rev:6;)
  If a Microsoft Windows system "reboot" is detected, Sagan will "set" a flowbit named "reboot.windows". No alert will be generated for this rule.

alert syslog $HOME_NET any -> $EXTERNAL_NET any (msg: "[WINDOWS-MALWARE] System protection disabled"; pcre: "/ 7034: | 7035: | 7046: | 7040: | 4689: | 593: /" ; pcre: "/Defender/Anti-Virus/antivirus/i"; content: "stop control"; flowbits: isnotset,by_src,reboot.windows; program: Service_Control_Manager; classtype: trojan-activity; reference: url,wiki.quadrantsec.com/bin/view/Main/5002011; sid: 5002011; rev:6;)
This rule monitors for anti-virus products being stopped. If Sagan detects that anti-virus is being stopped and the "reboot.windows" flowbit "issnot", and alert is generated. The anti-virus might be being stopped by a malicious user and/or process. The flowbit code for 1.0.0RC4 was rewritten to add more flexability. The new code now support multiple flowbits within a rule, "&" and "|" operators. For more information see:

https://wiki.quadrantsec.com/twiki/bin/view/Main/SaganRuleReference#flowbits_set_flowbit_name_expire

• [Feature] - New "output/sagan-perfmon.c" (Perfmon) output tool. This will record Sagan statistics in a CSV format. Useful for preformance tuning, graphing, etc.
• [Bugfix] - "content", "pcre" and "meta_content" handling changed in sagan.c to increase performance.
• [Bugfix] - With Rainers (Rsyslog) help, fixed long outstanding issue of compiling Sagan with liblognorm that resulted in a "json.h not found" error. Added pkg-config options for json-c, liblognorm and libetr. This should help Sagan build a lot more cleanly.
• [Bugfix] - Remove hardcoded UDP 514 in sagan-plog.c check.
• [Bugfix] - Now treating meta_content like content/pcre (was "special")
• [Bugfix] - "content", "pcre" and "meta_content" handling changed in sagan.c to increase performance.