diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..b9785be --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,69 @@ +# Assemblyline contributing guide + +This guide covers the basics of how to contribute to the Assemblyline project. + +Python code should follow the PEP8 guidelines defined here: +[PEP8 Guidelines](https://www.python.org/dev/peps/pep-0008/). + +## Tell us want you want to build/fix + +Before you start coding anything you should connect with the Assemblyline community via the +[Assemblyline Discord server](https://discord.gg/GUAy9wErNu) and/or the +[central Assemblyline GitHub project](https://github.com/CybercentreCanada/assemblyline/issues) to make sure no one +else is working on the same thing and that whatever you are going to build still fits with the vision of the system. + +## Git workflow + +- Clone the repo to your own account +- Checkout and pull the latest commits from the master branch +- Make a branch +- Work on your modifications and make sure your changes work as expected +- When you're satisfied with your changes, create a pull requests to the Assemblyline repo + +#### Transfer your service repo + +If you've worked on a new service that you want to be included in the default service selection you'll have to transfer +the associated repo into our control. + +#### You are not allow to merge: + +Even if you try to merge in your pull request, you will be denied. Only a few people in our team are allowed to merge +code into our repositories. + +We check for new pull requests every day and will merge them in once they have been approved by someone in our team. + +# Guide de contribution d'Assemblyline + +Ce guide couvre les bases de la façon de contribuer au projet Assemblyline. + +Le code Python doit suivre les directives PEP8 définies ici: +[Directives PEP8](https://www.python.org/dev/peps/pep-0008/). + +## Dites-nous que vous voulez construire / réparer + +Avant de commencer à coder quoi que ce soit, vous devriez vous connecter à la communauté Assemblyline via le +[Serveur Discord Assemblyline](https://discord.gg/GUAy9wErNu) et/ou le +[projet GitHub central Assemblyline](https://github.com/CybercentreCanada/assemblyline/issues) pour vous assurer que +personne d'autre ne travaille sur la même chose et que tout ce que vous allez construire correspond toujours à la vision +du système. + +## Flux de travail avec Git + +- Clonez le référentiel sur votre propre compte +- Changez de branche pour la branche principale et la synchroniser avec le serveur de référence +- Faire une nouvelle branche +- Travaillez sur ce que vous souhaitez et assurez-vous que vos modifications fonctionnent comme prévu +- Lorsque vous êtes satisfait de vos modifications, créez une demande de fusion sur le référentiel d'Assemblyline + +#### Transférer votre référentiel de service + +Si vous avez travaillé sur un nouveau service que vous souhaitez inclure dans la sélection de service par défaut, vous +devrez transférer le référentiel associé sous notre contrôle. + +#### Vous n'êtes pas autorisé à compléter une fusion: + +Même si vous tentez de compléter une demande de fusion, vous serez refusé. Seules quelques personnes de notre équipe +sont autorisées à fusionner dans nos référentiels. + +Nous vérifions les nouvelles demande de fusion tous les jours et les fusionnerons une fois qu'elles auront été approuvées +par quelqu'un de notre équipe. diff --git a/LICENSE b/LICENSE index 402bae7..e8a2399 100644 --- a/LICENSE +++ b/LICENSE @@ -1,6 +1,6 @@ MIT License -Copyright (c) 2021 Canadian Centre for Cyber Security +Copyright (c) 2024 CybercentreCanada Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal @@ -19,3 +19,4 @@ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + diff --git a/Makefile b/Makefile new file mode 100644 index 0000000..279faa8 --- /dev/null +++ b/Makefile @@ -0,0 +1,26 @@ +ifndef VERSION +$(error VERSION is undefined) +endif + +TAG?=latest +ORG?=cccs + +ifneq ($(ORG)x, x) +ORG:=$(ORG)/ +endif +ifneq ($(REGISTRY)x, x) +ORG:=$(REGISTRY)/ +endif + +.PHONY: default +default: build + +.PHONY: build +build: + docker build \ + --pull \ + --build-arg version=$(VERSION) \ + --build-arg branch=stable \ + -t $(REGISTRY)$(ORG)assemblyline-service-badlist:$(TAG)\ + -f ./Dockerfile \ + . diff --git a/README.md b/README.md index 3e2dc5b..d4afe3d 100644 --- a/README.md +++ b/README.md @@ -1,21 +1,28 @@ -# Badlisting service +[![Discord](https://img.shields.io/badge/chat-on%20discord-7289da.svg?sanitize=true)](https://discord.gg/GUAy9wErNu) +[![](https://img.shields.io/discord/908084610158714900)](https://discord.gg/GUAy9wErNu) +[![Static Badge](https://img.shields.io/badge/github-assemblyline-blue?logo=github)](https://github.com/CybercentreCanada/assemblyline) +[![Static Badge](https://img.shields.io/badge/github-assemblyline\_service\_badlist-blue?logo=github)](https://github.com/CybercentreCanada/assemblyline-service-badlist) +[![License](https://img.shields.io/github/license/CybercentreCanada/assemblyline-service-badlist)](./LICENSE) +[![GitHub Issues or Pull Requests by label](https://img.shields.io/github/issues/CybercentreCanada/assemblyline/service-badlist)](https://github.com/CybercentreCanada/assemblyline/issues?q=is:issue+is:open+label:service-badlist) -This service allow you to mark bad a set of files and tags so they always score as malicious in Assemblyline. The content of the badlist is driven by badlist sources or user marked files or IOCs strait from Assemblyline's UI. +# Badlist Service -## Sources +This service interfaces with Assemblyline's Badlist to mark tags or files as malicious. -When adding sources to the service, there are two types of expected data formats +## Service Details + +### Sources +When adding sources to the service, there are two types of expected data formats - csv - json There are also multiple types of sources for this service: - - blocklist - malware_family_list - attribution_list -### Blocklist Data Formats +#### Blocklist Data Formats In order for the service to pull the right IOCs and categorize them per source, you'll have to instruct it on how to using the `config.updater.` key. @@ -23,9 +30,9 @@ Within each `source` map, you'll specify the type of source this is (`blocklist` You'll also have to specify the different IOC types (`domain`, `ip`, `uri`, `md5`, `sha1`, `sha256`, `ssdeep`, `tlsh`) you expect to find in the data and where. -For example if dealing with a CSV file and you expect to find `uri`s in the 3rd column per row: +For example if dealing with a CSV file and you expect to find `uri`s in the 3rd column: -ie. "`,,https://google.com,...`" +`,,https://google.com,...` Then your source configuration will look like: @@ -40,7 +47,7 @@ config: Similarly, if you're dealing with a JSON list (`[{}, {}, ...]`) and you know to find `uri`s under the key `bad_uri` in each record: -ie. `{"bad_uri": "https://google.com", "family": "bad_stuff", ...}` +`{"bad_uri": "https://google.com", "family": "bad_stuff", ...}` ```yaml config: @@ -53,9 +60,127 @@ config: You can also override Assemblyline's default scoring of badlist matches (1000 points) by providing a `score` per source. -### Automated Expiration +#### Automated Expiration By default, we assume that all the items added to the Badlist will be valid forever but that's not always the cases. You will also be able to set a DTL (Days to Live) period for items that belong to a source using `dtl`. -If there are multiple sources with DTLs configured that raise an item, then the expiry date will be extended by the sum of the DTL values at the time of importing. +If there are multiple sources with DTLs configured that raise an item, then the expiry date will be set such that it uses the longest requested DTL at the time of import. + +## Image variants and tags + +Assemblyline services are built from the [Assemblyline service base image](https://hub.docker.com/r/cccs/assemblyline-v4-service-base), +which is based on Debian 11 with Python 3.11. + +Assemblyline services use the following tag definitions: + +| **Tag Type** | **Description** | **Example Tag** | +| :----------: | :----------------------------------------------------------------------------------------------- | :------------------------: | +| latest | The most recent build (can be unstable). | `latest` | +| build_type | The type of build used. `dev` is the latest unstable build. `stable` is the latest stable build. | `stable` or `dev` | +| series | Complete build details, including version and build type: `version.buildType`. | `4.5.stable`, `4.5.1.dev3` | + +## Running this service + +This is an Assemblyline service. It is designed to run as part of the Assemblyline framework. + +If you would like to test this service locally, you can run the Docker image directly from the a shell: + + docker run \ + --name Badlist \ + --env SERVICE_API_HOST=http://`ip addr show docker0 | grep "inet " | awk '{print $2}' | cut -f1 -d"/"`:5003 \ + --network=host \ + cccs/assemblyline-service-badlist + +To add this service to your Assemblyline deployment, follow this +[guide](https://cybercentrecanada.github.io/assemblyline4_docs/developer_manual/services/run_your_service/#add-the-container-to-your-deployment). + +## Documentation + +General Assemblyline documentation can be found at: https://cybercentrecanada.github.io/assemblyline4_docs/ + +# Service Badlist + +Ce service interface avec la liste mauvaise d'Assemblyline pour marquer les tags ou les fichiers comme malveillants. + +## Détails du Service +### Sources +Lors de l'ajout de sources au service, il existe deux types de formats de données attendus +- csv +- json + +Il existe également plusieurs types de sources pour ce service : + - liste de blocage + - liste de familles de logiciels malveillants + - liste d'attribution + +#### Formats de données de la Liste de Blocage +Pour que le service puisse extraire les bons tags et les catégoriser par source, vous devrez lui indiquer comment le faire en utilisant la clé `config.updater.`. + +Dans chaque structure de données `source`, vous spécifierez le type de source (blocklist) ainsi que le format (`json` | `csv`). + +Vous devrez également spécifier les différents types de tags (`domain`, `ip`, `uri`, `md5`, `sha1`, `sha256`, `ssdeep`, `tlsh`) que vous vous attendez à trouver dans les données et où. + +Par exemple, si vous utilisez un fichier CSV et que vous vous attendez à trouver des `uri` dans la 3ème colonne: + +`,,https://google.com,...` + +Votre configuration de `source` ressemblera à ceci : +```yaml +config: + updater: + my_source: + type: blocklist + format: csv + uri: 2 +``` +De même, si vous utilisez une liste JSON (`[{}, {}, ...]`) et que vous avez des `uri` sous la clé `bad_uri` dans chaque entrée : + +`{"bad_uri": "https://google.com", "family": "bad_stuff", ...}` + +```yaml +config: + updater: + my_source: + type: blocklist + format: json + uri: "bad_uri" +``` +Vous pouvez également remplacer le score par défaut d'Assemblyline pour les éléments trouvés dans la liste mauvaise (1000 points) en fournissant un `score` par source. + +#### Expiration Automatisée +Par défaut, nous supposons que tous les éléments ajoutés à la liste noire seront valides pour toujours, mais ce n'est pas toujours le cas. Vous pourrez également définir une période d'expiration pour les éléments appartenant à une source en utilisant `dtl` (`days to live`, ou jours à vivre en anglais). + +S'il y a plusieurs sources avec des expirations configurées contenant un même élément, la date d'expiration sera prolongée à la date d'expiration la plus lointaine au moment de l'importation. + +## Variantes et étiquettes d'image + +Les services d'Assemblyline sont construits à partir de l'image de base [Assemblyline service](https://hub.docker.com/r/cccs/assemblyline-v4-service-base), +qui est basée sur Debian 11 avec Python 3.11. + +Les services d'Assemblyline utilisent les définitions d'étiquettes suivantes: + +| **Type d'étiquette** | **Description** | **Exemple d'étiquette** | +| :------------------: | :------------------------------------------------------------------------------------------------------------- | :------------------------: | +| dernière version | La version la plus récente (peut être instable). | `latest` | +| build_type | Type de construction utilisé. `dev` est la dernière version instable. `stable` est la dernière version stable. | `stable` ou `dev` | +| série | Détails de construction complets, comprenant la version et le type de build: `version.buildType`. | `4.5.stable`, `4.5.1.dev3` | + +## Exécution de ce service + +Il s'agit d'un service d'Assemblyline. Il est optimisé pour fonctionner dans le cadre d'un déploiement d'Assemblyline. + +Si vous souhaitez tester ce service localement, vous pouvez exécuter l'image Docker directement à partir d'un terminal: + + docker run \ + --name Badlist \ + --env SERVICE_API_HOST=http://`ip addr show docker0 | grep "inet " | awk '{print $2}' | cut -f1 -d"/"`:5003 \ + --network=host \ + cccs/assemblyline-service-badlist + +Pour ajouter ce service à votre déploiement d'Assemblyline, suivez ceci +[guide](https://cybercentrecanada.github.io/assemblyline4_docs/fr/developer_manual/services/run_your_service/#add-the-container-to-your-deployment). + +## Documentation + +La documentation générale sur Assemblyline peut être consultée à l'adresse suivante: https://cybercentrecanada.github.io/assemblyline4_docs/ diff --git a/badlist/__init__.py b/badlist/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/pipelines/.cruft.json b/pipelines/.cruft.json new file mode 100644 index 0000000..e2d685a --- /dev/null +++ b/pipelines/.cruft.json @@ -0,0 +1,20 @@ +{ + "template": "https://github.com/CybercentreCanada/assemblyline-service-pipeline-generator.git", + "commit": "3f3f922ec39c7b953a1c22d29828cb3231d7d47a", + "checkout": null, + "context": { + "cookiecutter": { + "__directory_name": "pipelines", + "classification": "UNCLASSIFIED", + "is_public": true, + "test_in_container": true, + "test_versions": "default", + "__setupscript": "setup.bash", + "_extensions": [ + "local_extensions.extract_versions" + ], + "_template": "https://github.com/CybercentreCanada/assemblyline-service-pipeline-generator.git" + } + }, + "directory": null + } diff --git a/pipelines/azure-build.yaml b/pipelines/azure-build.yaml index bbbc6c0..43cc457 100644 --- a/pipelines/azure-build.yaml +++ b/pipelines/azure-build.yaml @@ -2,23 +2,6 @@ name: build variables: - group: unittest-samples - - name: self_location - value: "self_location" - - name: full_self_location - value: "$(Agent.BuildDirectory)/$(self_location)" - - name: samples_location - value: "samples_location" - - name: full_samples_location - value: "$(Agent.BuildDirectory)/$(samples_location)" - -resources: - repositories: - - repository: unittest-samples - type: github - name: $(unittest_samples_repository) - ref: main - endpoint: github-repo-sa - trigger: none trigger: tags: @@ -28,54 +11,25 @@ pr: none pool: vmImage: "ubuntu-20.04" -stages: - - stage: deploy - jobs: - - job: deploy - displayName: Deploy containers to dockerhub - variables: - - group: deployment-information - steps: - - task: Docker@2 - displayName: Login to docker hub - inputs: - command: login - containerRegistry: dockerhub - - task: Docker@2 - displayName: Login to chimera - inputs: - command: login - containerRegistry: CHIMERA-U-ACR - - checkout: self - fetchDepth: 1 - path: $(self_location) - - checkout: unittest-samples - fetchDepth: 1 - path: $(samples_location) - - script: | - export TAG=${BUILD_SOURCEBRANCH#"refs/tags/v"} - if [[ "$TAG" == *stable* ]]; then export BUILD_TYPE=stable; else export BUILD_TYPE=latest; fi - docker build --build-arg version=$TAG --build-arg branch=$BUILD_TYPE -t cccs/${BUILD_REPOSITORY_NAME##*/}:$TAG -t cccs/${BUILD_REPOSITORY_NAME##*/}:$BUILD_TYPE -f ./Dockerfile . - workingDirectory: $(full_self_location) - displayName: Build containers - - script: | - [ ! -d "$(pwd)/tests" ] && echo "No tests found" && exit - export TAG=${BUILD_SOURCEBRANCH#"refs/tags/v"} - if [[ "$TAG" == *stable* ]]; then export BUILD_TYPE=stable; else export BUILD_TYPE=latest; fi - [ -f "$(pwd)/tests/requirements.txt" ] && docker run -e FULL_SELF_LOCATION=/opt/al_service -e FULL_SAMPLES_LOCATION=/opt/samples -v /usr/share/ca-certificates/mozilla:/usr/share/ca-certificates/mozilla -v $(pwd)/tests/:/opt/al_service/tests/ -v ${FULL_SAMPLES_LOCATION}:/opt/samples cccs/${BUILD_REPOSITORY_NAME##*/}:$BUILD_TYPE bash -c 'pip install -U -r tests/requirements.txt; pytest -p no:cacheprovider -vv' && exit - docker run -e FULL_SELF_LOCATION=/opt/al_service -e FULL_SAMPLES_LOCATION=/opt/samples -v /usr/share/ca-certificates/mozilla:/usr/share/ca-certificates/mozilla -v $(pwd)/tests/:/opt/al_service/tests/ -v ${FULL_SAMPLES_LOCATION}:/opt/samples cccs/${BUILD_REPOSITORY_NAME##*/}:$BUILD_TYPE bash -c 'pytest -p no:cacheprovider -vv' - workingDirectory: $(full_self_location) - displayName: Test containers - - script: | - export TAG=${BUILD_SOURCEBRANCH#"refs/tags/v"} - if [[ "$TAG" == *stable* ]]; then export BUILD_TYPE=stable; else export BUILD_TYPE=latest; fi - export SERIES="`expr $TAG : '\([0-9]\+\.[0-9]\+\.\)'`${BUILD_TYPE}" +resources: + repositories: + - repository: PipelineTemplates + type: github + name: CybercentreCanada/assemblyline-pipeline-templates + ref: refs/heads/main + endpoint: github-repo-sa + trigger: none + - repository: unittest-samples + type: github + name: $(unittest_samples_repository) + ref: main + endpoint: github-repo-sa + trigger: none - for IMAGE in "cccs/" "uchimera.azurecr.io/cccs/" - do - docker tag cccs/${BUILD_REPOSITORY_NAME##*/}:$BUILD_TYPE ${IMAGE}${BUILD_REPOSITORY_NAME##*/}:$TAG - docker tag cccs/${BUILD_REPOSITORY_NAME##*/}:$BUILD_TYPE ${IMAGE}${BUILD_REPOSITORY_NAME##*/}:$BUILD_TYPE - docker tag cccs/${BUILD_REPOSITORY_NAME##*/}:$BUILD_TYPE ${IMAGE}${BUILD_REPOSITORY_NAME##*/}:$SERIES - docker push ${IMAGE}${BUILD_REPOSITORY_NAME##*/} --all-tags - done - displayName: Deploy to container repositories +extends: + template: stages/deploy-service.yaml@PipelineTemplates + parameters: + is_public: "true" + samples_repo: unittest-samples + labels: + classification: "UNCLASSIFIED" diff --git a/pkglist.txt b/pkglist.txt new file mode 100644 index 0000000..e69de29 diff --git a/pyproject.toml b/pyproject.toml new file mode 100644 index 0000000..6350a1b --- /dev/null +++ b/pyproject.toml @@ -0,0 +1,2 @@ +[tool.cruft] +skip = ["pkglist.txt", "README.md", "badlist", "tests"] diff --git a/requirements.txt b/requirements.txt new file mode 100644 index 0000000..ff91064 --- /dev/null +++ b/requirements.txt @@ -0,0 +1,2 @@ +assemblyline +assemblyline-v4-service