Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add dynamic analysis #1697

Merged
merged 556 commits into from
Nov 29, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
556 commits
Select commit Hold shift + click to select a range
aacd9f5
delete empty files
yelhamer Aug 7, 2023
a185341
features/address.py: rename `CallAddress` `DynamicCallAddress`
yelhamer Aug 7, 2023
681d4fb
Merge pull request #1678 from yelhamer/call-scope
williballenthin Aug 7, 2023
582bb7c
docstrings: improve wording
williballenthin Aug 10, 2023
ae9d773
add TODO for typing.TypeAlias
williballenthin Aug 10, 2023
85b58d0
process: simplify string enumeration loop
williballenthin Aug 10, 2023
3cf748a
vverbose: render both scopes nicely
williballenthin Aug 10, 2023
c1fbb27
Merge branch 'master' into dynamic-feature-extraction
williballenthin Aug 10, 2023
19495f6
freeze: pydantic v2 fixes
williballenthin Aug 10, 2023
34db631
sync submodule testfiles
williballenthin Aug 11, 2023
1cf33e4
tests: create workspaces only during tests, not import
williballenthin Aug 11, 2023
6de23a9
tests: main: demonstrate CAPE analysis (and bug #1702)
williballenthin Aug 11, 2023
dafbefb
render: verbose: render call address
williballenthin Aug 11, 2023
f48e4a8
render: verbose: render dynamic call return address
williballenthin Aug 11, 2023
c91dc71
result document: wire analysis flavor through metadata
williballenthin Aug 11, 2023
3057b5f
render: show analysis flavor
williballenthin Aug 11, 2023
e100a63
rules: use set instead of tuple, add doc
williballenthin Aug 11, 2023
3c069a6
rules: don't change passed-in argument
williballenthin Aug 11, 2023
8202e9e
main: don't use analysis flavor to filter rules
williballenthin Aug 11, 2023
fd1cd05
vverbose: render relevant scope at top of match tree
williballenthin Aug 11, 2023
c6d400b
address: remove dynamic return address concept, as its unused today
williballenthin Aug 11, 2023
751231b
fixtures.py: fix the path of '0000a567' in `get_data_path_by_name()` …
yelhamer Aug 14, 2023
2a83f1f
Merge pull request #1708 from mandiant/fix/issue-1707
yelhamer Aug 14, 2023
84cddc7
Merge pull request #1709 from mandiant/fix/issue-1702
yelhamer Aug 14, 2023
846bd62
Merge remote-tracking branch 'parentrepo/dynamic-feature-extraction' …
yelhamer Aug 14, 2023
d648fdf
Merge pull request #1713 from mandiant/fix/issue-1711
williballenthin Aug 14, 2023
88aa17f
Merge pull request #1716 from mandiant/fix/issue-1697-2
yelhamer Aug 15, 2023
0e1ce21
Merge pull request #1715 from mandiant/fix/issue-1710
yelhamer Aug 15, 2023
4411911
Merge remote-tracking branch 'parentrepo/dynamic-feature-extraction' …
yelhamer Aug 15, 2023
4978aa7
tests: temporarily xfail script test
williballenthin Aug 15, 2023
476c7ff
main: provide encoding to open
williballenthin Aug 15, 2023
c001c88
Merge pull request #1714 from mandiant/fix/issue-1697-1
williballenthin Aug 15, 2023
2a31b16
merge
williballenthin Aug 15, 2023
827b4b2
test_rules: fix rule scoping logic
williballenthin Aug 15, 2023
db40d9b
wip: add initial CAPE model
williballenthin Aug 15, 2023
59a129d
cape: add pydantic model for v2.2
williballenthin Aug 15, 2023
bb2b182
Merge branch 'master' into dynamic-feature-extraction
williballenthin Aug 15, 2023
d8bea81
cape: models: add more fields
williballenthin Aug 15, 2023
25aabcd
cape: models: more shapes
williballenthin Aug 16, 2023
046427c
cape: model: document the data we'll use in cape
williballenthin Aug 16, 2023
26539e6
cape: models: add tests
williballenthin Aug 16, 2023
2eda053
cape: models: more data shapes
williballenthin Aug 16, 2023
4be1c89
cape: models: more data shapes
williballenthin Aug 16, 2023
e943a71
cape: models: relax deserializing FlexibleModels
williballenthin Aug 16, 2023
6f7bf96
cape: use pydantic model
williballenthin Aug 16, 2023
8788a40
Merge branch 'dynamic-feature-extraction' into feat/cape-pydantic
williballenthin Aug 16, 2023
724db83
cape: require PE analysis
williballenthin Aug 16, 2023
3350a93
ida: use ida_nalt not idaapi
williballenthin Aug 16, 2023
c80542d
cape: call: fix argument type switch
williballenthin Aug 16, 2023
3614ce1
cape: fix test failures
williballenthin Aug 16, 2023
264958e
Update capa/features/common.py
yelhamer Aug 16, 2023
120917e
cape: models: tweaks from Avast dataset
williballenthin Aug 18, 2023
8cd5e03
ci: pre-commit: show-diff-on-failure
williballenthin Aug 18, 2023
05cf720
Merge branch 'dynamic-feature-extraction' into feat/cape-pydantic
williballenthin Aug 18, 2023
b10275e
black
williballenthin Aug 18, 2023
5e31f0d
cape: models: more fixes thanks to avast
williballenthin Aug 18, 2023
3500947
main.py: look up rules scope with scopes attribute, not their meta field
yelhamer Aug 18, 2023
18dff9d
cape: models: more fixes thanks to avast
williballenthin Aug 18, 2023
d741544
result_document.py: use the scopes attribute instead of meta["scope"]
yelhamer Aug 18, 2023
9144d12
add error message for invalid report files
yelhamer Aug 18, 2023
a8bd5b1
disable packed-sample warning for dynamic feature extractors
yelhamer Aug 18, 2023
e7c0bea
Match.from_capa(): remove reliance on the meta field to get the scope
yelhamer Aug 18, 2023
521bd25
remove file-limitations checks for dynamic extractors
yelhamer Aug 18, 2023
ee936f9
Merge pull request #1729 from mandiant/feat/cape-pydantic
williballenthin Aug 22, 2023
e5af716
Update capa/features/freeze/__init__.py
williballenthin Aug 22, 2023
89c8c6d
Update capa/rules/__init__.py
williballenthin Aug 22, 2023
21cefa0
Merge branch 'master' into dynamic-feature-extraction
williballenthin Aug 22, 2023
675ad36
point submodule rules to branch dynamic-syntax
williballenthin Aug 22, 2023
5b585c0
cape: better detect CAPE reports
williballenthin Aug 22, 2023
c160f45
main: fix rendering of logging message
williballenthin Aug 22, 2023
9489927
Merge pull request #1746 from mandiant/fix/issue-1745
williballenthin Aug 22, 2023
4ab240e
rules: add scope terms "unsupported" and "unspecified"
williballenthin Aug 22, 2023
d106899
test_rules_insn_scope.py: update rules missing the dynamic scope
yelhamer Aug 22, 2023
2541404
Merge pull request #1748 from mandiant/feat/issue-1744
williballenthin Aug 22, 2023
44fc335
initial commit
yelhamer Aug 22, 2023
77b3fad
lint.py: add 'unsupported' keyword
yelhamer Aug 22, 2023
901ba55
lint.py: fix boolean statement
yelhamer Aug 22, 2023
5730e55
lint.py: update recommendation messages
yelhamer Aug 22, 2023
39c8fd8
Update capa/features/freeze/__init__.py
yelhamer Aug 23, 2023
cdb469e
capa/features/freeze/__init__.py: remove comment
yelhamer Aug 23, 2023
86effec
capa/rules/__init__.py: merge features from small scopes into larger …
yelhamer Aug 23, 2023
5ba7325
Merge pull request #1753 from yelhamer/update-linter
yelhamer Aug 23, 2023
42689ef
test_main.py: revert ruleset-related xfails
yelhamer Aug 24, 2023
7c101f0
test_binja.py: revert ruleset-related xfails
yelhamer Aug 24, 2023
d66f834
Update tests/test_scripts.py
yelhamer Aug 24, 2023
9eb1255
cape2yara.py: update for use of scopes, and fix bug
yelhamer Aug 24, 2023
46217a3
test_main.py: remove unused pytest
yelhamer Aug 24, 2023
3574bd4
Merge remote-tracking branch 'parentrepo/dynamic-feature-extraction' …
yelhamer Aug 24, 2023
70eae1a
freeze/__init__.py: fix missing space
yelhamer Aug 24, 2023
b636f23
Merge branch 'fix-cape2fmt' of https://github.com/yelhamer/capa into …
yelhamer Aug 24, 2023
acd3a30
Merge pull request #1758 from yelhamer/fix-cape2fmt
yelhamer Aug 24, 2023
f74107d
initial commit
yelhamer Aug 25, 2023
0ded827
modify null extractor
yelhamer Aug 25, 2023
707dee4
base_Extractor.py: make `sample_hashes` attribute private
yelhamer Aug 25, 2023
49bf2eb
base_extractor.py: replace dunder with single underscore for `sample_…
yelhamer Aug 25, 2023
3ee56e3
Merge pull request #1762 from yelhamer/modify-sample-hashes
yelhamer Aug 25, 2023
f34b035
test_result_document.py: re-enable result-document related tests
yelhamer Aug 25, 2023
b930523
freeze: add TODO issue link
williballenthin Aug 25, 2023
164b082
extractor: tweak hashes to fix mypy
williballenthin Aug 25, 2023
f2909c8
proto: reenable tests and linters
williballenthin Aug 25, 2023
8ff74d4
proto: regenerate using 3.21 protoc
williballenthin Aug 25, 2023
ebcbad3
proto: add new scopes
williballenthin Aug 25, 2023
a734358
rules: use Scope enum instead of constants
williballenthin Aug 25, 2023
a60401f
Merge branch 'master' into dynamic-feature-extraction
williballenthin Aug 25, 2023
b88853f
changelog
williballenthin Aug 25, 2023
7e258a9
Merge branch 'dynamic-feature-extraction' into fix/scope-enum-usage
williballenthin Aug 25, 2023
17e4765
changelog
williballenthin Aug 25, 2023
e4c1361
Merge branch 'fix/scope-enum-usage' into fix/dynamic-proto
williballenthin Aug 25, 2023
9bbd318
rules: handle unsupported scopes again
williballenthin Aug 25, 2023
f96b9e6
proto: add RuleMetadata.scopes
williballenthin Aug 25, 2023
08c9bbc
proto: deprecate RuleMetadata.scope
williballenthin Aug 25, 2023
88ee6e6
wip: proto: add Metadata.[static, dynamic]_analysis
williballenthin Aug 25, 2023
d5daa79
Merge pull request #1764 from mandiant/fix/scope-enum-usage
yelhamer Aug 25, 2023
e9a9b3a
point the data file to the latest PR
yelhamer Aug 26, 2023
49adecb
add yaml representer for the Scope class, as well as other bugfixes
yelhamer Aug 26, 2023
b0133f0
various fixes
yelhamer Aug 26, 2023
8c9e676
binja: use binja api's methods to get the file hash
yelhamer Aug 27, 2023
4d538b9
Update scripts/import-to-ida.py
yelhamer Aug 27, 2023
214a355
binja extractor: remove unused pathlib.Path import
yelhamer Aug 28, 2023
9eb88e6
Merge branch 'dynamic-feature-extraction' of https://github.com/mandi…
yelhamer Aug 28, 2023
9dc457e
Update capa/features/freeze/__init__.py
yelhamer Aug 28, 2023
09afcfb
render/verbose.py: remove `frz.AddressType.FREEZE`
yelhamer Aug 29, 2023
2c75f78
main.py rdoc.Metadata creation: revert to usage of `as_posix()` withi…
yelhamer Aug 29, 2023
0987673
verbose.py: temporarily add a mypy-related assert to `render_static_m…
yelhamer Aug 29, 2023
47330e6
verbose.py render_dynamic_meta(): s/doc: rd.ResultDocument/meta: rd.M…
yelhamer Aug 29, 2023
73c158a
point submodules towards the right branch
yelhamer Aug 30, 2023
99caa87
Update capa/main.py
yelhamer Sep 4, 2023
1d8e650
freeze/__init__.py: bump freeze version to 3
yelhamer Sep 4, 2023
d83c0e7
main.py: remove comment type annotations
yelhamer Sep 4, 2023
9ec1bf3
point rules towards dynamic-syntax
yelhamer Sep 4, 2023
cfa703e
remove type comment
yelhamer Sep 4, 2023
f905ed6
Merge branch 'dynamic-feature-extraction' of https://github.com/mandi…
yelhamer Sep 4, 2023
dd0eadb
freeze/__init__.py: bump freeze version to 3
yelhamer Sep 4, 2023
766b05e
Merge branch 'dynamic-feature-extraction' into fix/dynamic-proto
williballenthin Sep 5, 2023
3725618
render: proto: use Static/Dynamic analysis types
williballenthin Sep 5, 2023
866c7c5
proto: deprecate metadata.analysis
williballenthin Sep 5, 2023
69836a0
proto: add dynamic test
williballenthin Sep 5, 2023
2a757b0
submodule: test data: update
williballenthin Sep 5, 2023
5b4c167
proto: add additional types
williballenthin Sep 5, 2023
d64ab41
tests: proto: add more dynamic proto tests
williballenthin Sep 5, 2023
72e8361
proto: better convert to/from proto
williballenthin Sep 5, 2023
7d9ae57
check for pid and ppid reuse
yelhamer Oct 4, 2023
35f64f3
cape/global_.py: throw exceptions for unrecognized OSes, formats, and…
yelhamer Oct 4, 2023
28a722d
scripts/profile_time.py: revert restriction that frozen extractors ca…
yelhamer Oct 4, 2023
8b287c1
scripts/profile_time.py: revert restriction that sample extractors ca…
yelhamer Oct 4, 2023
953b2e8
rendering: several fixes and added types/classes
yelhamer Oct 11, 2023
559f2fd
cape/file.py: flake8 fixes
yelhamer Oct 11, 2023
d1b7afb
Update capa/render/verbose.py
yelhamer Oct 14, 2023
ffe6ab6
main.py: load signatures only for the static context
yelhamer Oct 16, 2023
a1aca3a
Merge branch 'dynamic-feature-extraction' of https://github.com/mandi…
yelhamer Oct 16, 2023
9a66c26
cape/file.py: fix flake8 issue of using '+' for logging
yelhamer Oct 16, 2023
182a986
merge master
williballenthin Oct 17, 2023
bf233c1
integrate Ghidra backend with dynamic analysis
williballenthin Oct 17, 2023
44d05f9
dynamic: fix some tests
williballenthin Oct 17, 2023
8ee97ac
dynamic: fix some tests
williballenthin Oct 17, 2023
cb5fa36
flake8
williballenthin Oct 17, 2023
e1b3a3f
rules: fix rendering of yaml
williballenthin Oct 17, 2023
7205bc2
submodule: rules: update
williballenthin Oct 17, 2023
5475020
dynamic: fix tests
williballenthin Oct 17, 2023
92daf3a
elffile: fix property access
williballenthin Oct 17, 2023
1aac4a1
mypy
williballenthin Oct 17, 2023
55e4fdd
mypy
williballenthin Oct 17, 2023
21f9e07
isort
williballenthin Oct 17, 2023
2cfd450
improve and fix various dynamic parts (#1809)
mr-tz Oct 18, 2023
b6f13f3
improve vverbose rendering
mr-tz Oct 18, 2023
9609d63
Update tests/test_main.py
yelhamer Oct 19, 2023
8687c74
Merge pull request #1817 from mandiant/improve-vv-render
mr-tz Oct 19, 2023
5c48f38
capa/main.py: add a `capabilities` module and move all of the capabil…
yelhamer Oct 19, 2023
37caeb2
capabilities: add a test file for the new capabilities module, and mo…
yelhamer Oct 19, 2023
f2011c1
fix styling issues
yelhamer Oct 19, 2023
85610a8
changelog fix
yelhamer Oct 19, 2023
d5e187b
Merge branch 'master' into dynamic-feature-extraction
williballenthin Oct 19, 2023
7bc3fba
Merge branch 'dynamic-feature-extraction' into fix/dynamic-proto
williballenthin Oct 19, 2023
8b0ba1e
tests: rename freeze tests
williballenthin Oct 19, 2023
3d4facd
Merge branch 'dynamic-feature-extraction' into fix/dynamic-proto
williballenthin Oct 19, 2023
9836032
proto: fix serialization of call address
williballenthin Oct 19, 2023
3519125
tests: fix COM tests with dynamic scope
williballenthin Oct 19, 2023
fbeb33a
Merge branch 'dynamic-feature-extraction' into fix/dynamic-proto
williballenthin Oct 19, 2023
2cc6a37
ci: run fast tests before the full suite
williballenthin Oct 19, 2023
288313a
changelog
williballenthin Oct 19, 2023
1cb3ca6
pre-commit: only run fast checks during commit
williballenthin Oct 19, 2023
b6a0d6e
pre-commit: fix stages
williballenthin Oct 19, 2023
84e22b1
doc
williballenthin Oct 19, 2023
c724a4b
ci: only run BN and Ghidra tests after others complete
williballenthin Oct 19, 2023
d42829d
Merge pull request #1765 from mandiant/fix/dynamic-proto
williballenthin Oct 19, 2023
b8b55f4
identify potential JSON object data start (#1819)
mr-tz Oct 19, 2023
0ae8f34
Merge branch 'dynamic-feature-extraction' into capabilities-module
yelhamer Oct 20, 2023
0231cee
null extractor: fix typings
williballenthin Oct 20, 2023
bfecf41
freeze: add dynamic tests
williballenthin Oct 20, 2023
10dc4b9
freeze: update freeze format v3
williballenthin Oct 20, 2023
1143f2b
changelog
williballenthin Oct 20, 2023
fc4618e
Merge branch 'dynamic-feature-extraction' into fix/dynamic-freeze
williballenthin Oct 20, 2023
f9b8741
Update capa/capabilities/common.py
yelhamer Oct 20, 2023
423d942
Update capa/capabilities/dynamic.py
yelhamer Oct 20, 2023
20604c4
Update capa/capabilities/static.py
yelhamer Oct 20, 2023
96fb204
move capa.features.capabilities to capa.capabilities, and update scripts
yelhamer Oct 20, 2023
d5ae2ff
capa.capabilities: move `has_file_limitations()` from capa.main to th…
yelhamer Oct 20, 2023
d6c5d98
move `is_file_limitation_rule()` to the rules module (Rule class)
yelhamer Oct 20, 2023
aae7266
Merge branch 'capabilities-module' of https://github.com/yelhamer/cap…
yelhamer Oct 20, 2023
62d4b00
Merge pull request #1822 from mandiant/fix/dynamic-freeze
williballenthin Oct 20, 2023
788251b
vverbose: render scope for humans
williballenthin Oct 20, 2023
c9df782
Ignore DLL names for API features (#1824)
mr-tz Oct 20, 2023
ee4f029
layout: capture process name
williballenthin Oct 20, 2023
393b0e6
layout: capture process name
williballenthin Oct 20, 2023
99042f2
fix parsing base 10/16
mr-tz Oct 20, 2023
9e6919f
layout: capture call names
williballenthin Oct 20, 2023
ab06c94
capa/main.py: move `has_rule_with_namespace()` to `capa.rules.RuleSet`
yelhamer Oct 20, 2023
3572b51
test_capabilities.py: add missing `test_com_feature_matching()` test
yelhamer Oct 20, 2023
8029fed
Merge branch 'capabilities-module' of https://github.com/yelhamer/cap…
yelhamer Oct 20, 2023
db53424
Merge pull request #1826 from mandiant/fix-model-hexint
mr-tz Oct 23, 2023
a0cec3f
capa.rules: remove redundant `is_internal_rule()` and `has_file_limit…
yelhamer Oct 26, 2023
e559cc2
capa.rules: remove redundant `ceng.MatchResults` import
yelhamer Oct 26, 2023
0097822
Merge pull request #1820 from yelhamer/capabilities-module
yelhamer Oct 27, 2023
4a7e488
Update capa/render/vverbose.py
williballenthin Nov 1, 2023
274a710
report: better compute dynamic layout
williballenthin Nov 3, 2023
c141f7e
verbose: better render scopes
williballenthin Nov 3, 2023
9c81ccf
vverbose: make missing names an error
williballenthin Nov 3, 2023
0da614a
vverbose: dynamic: show rendered matching API call
williballenthin Nov 6, 2023
f7c72cd
vverbose: don't repeat rendered calls when in call scope
williballenthin Nov 6, 2023
eb12ec4
mypy
williballenthin Nov 6, 2023
75ff58e
vverbose: better render pid/tid/call index
williballenthin Nov 6, 2023
7678897
tests: fix render tests
williballenthin Nov 6, 2023
5d31bc4
verbose: render dynamic match locations
williballenthin Nov 6, 2023
a52af38
verbose: remove TODOs
williballenthin Nov 6, 2023
1acc2d1
Merge branch 'dynamic-feature-extraction' into fix/issue-1816
mr-tz Nov 8, 2023
52997e7
fix imports according to ruff
mr-tz Nov 8, 2023
41a3976
fix whitespace removal in format check
mr-tz Nov 10, 2023
dee0aa7
Merge pull request #1844 from mandiant/mr-tz-patch-1
mr-tz Nov 11, 2023
82013f0
submodule: tests: data: sync
williballenthin Nov 14, 2023
c68c68d
Merge branch 'dynamic-feature-extraction' into fix/issue-1816
williballenthin Nov 14, 2023
a705bf9
Merge pull request #1825 from mandiant/fix/issue-1816
williballenthin Nov 14, 2023
6e3fff4
use latest rules migration
williballenthin Nov 14, 2023
987eb2d
sync rules submodule
williballenthin Nov 14, 2023
de5f088
sync submodule rules
williballenthin Nov 15, 2023
a870c92
sync submodule rules
williballenthin Nov 15, 2023
f201ef1
actually get global feature values
mr-tz Nov 27, 2023
890c879
only check and display file limitation once
mr-tz Nov 27, 2023
d84457e
Merge pull request #1868 from mandiant/fix/global-features
mr-tz Nov 27, 2023
8f0eb56
only check and display file limitation once
mr-tz Nov 28, 2023
8946cb6
Merge pull request #1874 from mandiant/fix/global-features
mr-tz Nov 28, 2023
92770dd
set os, arch, format in meta table
mr-tz Nov 28, 2023
ba9da0d
Merge pull request #1876 from mandiant/fix/1867
mr-tz Nov 29, 2023
b4c6bf8
changelog
williballenthin Nov 29, 2023
583f8b5
Merge branch 'dynamic-feature-extraction' of public.github.com:mandia…
williballenthin Nov 29, 2023
e66c2ef
add documentation for dynamic capa capabilties (#1837)
yelhamer Nov 29, 2023
93cfb6e
sync testfiles submodule
williballenthin Nov 29, 2023
9a996d0
Merge branch 'dynamic-feature-extraction' of public.github.com:mandia…
williballenthin Nov 29, 2023
3c159a1
ci: revert temporary CI event subscription
williballenthin Nov 29, 2023
73ea822
Merge branch 'master' into dynamic-feature-extraction
mr-tz Nov 29, 2023
47019e4
Merge branch 'master' into dynamic-feature-extraction
mr-tz Nov 29, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 11 additions & 7 deletions .github/workflows/tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -39,13 +39,13 @@ jobs:
- name: Lint with ruff
run: pre-commit run ruff
- name: Lint with isort
run: pre-commit run isort
run: pre-commit run isort --show-diff-on-failure
- name: Lint with black
run: pre-commit run black
run: pre-commit run black --show-diff-on-failure
- name: Lint with flake8
run: pre-commit run flake8
run: pre-commit run flake8 --hook-stage manual
- name: Check types with mypy
run: pre-commit run mypy
run: pre-commit run mypy --hook-stage manual

rule_linter:
runs-on: ubuntu-20.04
Expand Down Expand Up @@ -95,6 +95,10 @@ jobs:
run: sudo apt-get install -y libyaml-dev
- name: Install capa
run: pip install -e .[dev]
- name: Run tests (fast)
# this set of tests runs about 80% of the cases in 20% of the time,
# and should catch most errors quickly.
run: pre-commit run pytest-fast --all-files --hook-stage manual
- name: Run tests
run: pytest -v tests/

Expand All @@ -103,7 +107,7 @@ jobs:
env:
BN_SERIAL: ${{ secrets.BN_SERIAL }}
runs-on: ubuntu-20.04
needs: [code_style, rule_linter]
needs: [tests]
strategy:
fail-fast: false
matrix:
Expand Down Expand Up @@ -143,7 +147,7 @@ jobs:
ghidra-tests:
name: Ghidra tests for ${{ matrix.python-version }}
runs-on: ubuntu-20.04
needs: [code_style, rule_linter]
needs: [tests]
strategy:
fail-fast: false
matrix:
Expand Down Expand Up @@ -197,4 +201,4 @@ jobs:
cat ../output.log
exit_code=$(cat ../output.log | grep exit | awk '{print $NF}')
exit $exit_code


2 changes: 2 additions & 0 deletions .gitmodules
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
[submodule "rules"]
path = rules
url = ../capa-rules.git
branch = dynamic-syntax
[submodule "tests/data"]
path = tests/data
url = ../capa-testfiles.git
branch = dynamic-feature-extractor
williballenthin marked this conversation as resolved.
Show resolved Hide resolved
28 changes: 23 additions & 5 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ repos:
hooks:
- id: isort
name: isort
stages: [commit, push]
stages: [commit, push, manual]
language: system
entry: isort
args:
Expand All @@ -45,7 +45,7 @@ repos:
hooks:
- id: black
name: black
stages: [commit, push]
stages: [commit, push, manual]
language: system
entry: black
args:
Expand All @@ -62,7 +62,7 @@ repos:
hooks:
- id: ruff
name: ruff
stages: [commit, push]
stages: [commit, push, manual]
language: system
entry: ruff
args:
Expand All @@ -79,7 +79,7 @@ repos:
hooks:
- id: flake8
name: flake8
stages: [commit, push]
stages: [push, manual]
language: system
entry: flake8
args:
Expand All @@ -97,7 +97,7 @@ repos:
hooks:
- id: mypy
name: mypy
stages: [commit, push]
stages: [push, manual]
language: system
entry: mypy
args:
Expand All @@ -109,3 +109,21 @@ repos:
- "tests/"
always_run: true
pass_filenames: false

- repo: local
hooks:
- id: pytest-fast
name: pytest (fast)
stages: [manual]
language: system
entry: pytest
args:
- "tests/"
- "--ignore=tests/test_binja_features.py"
- "--ignore=tests/test_ghidra_features.py"
- "--ignore=tests/test_ida_features.py"
- "--ignore=tests/test_viv_features.py"
- "--ignore=tests/test_main.py"
- "--ignore=tests/test_scripts.py"
always_run: true
pass_filenames: false
19 changes: 15 additions & 4 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,14 +3,25 @@
## master (unreleased)

### New Features
- ghidra: add Ghidra feature extractor and supporting code #1770 @colton-gabertan
- ghidra: add entry script helping users run capa against a loaded Ghidra database #1767 @mike-hunhoff
- add Ghidra backend #1770 #1767 @colton-gabertan @mike-hunhoff
- add dynamic analysis via CAPE sandbox reports #48 #1535 @yelhamer
- add call scope #771 @yelhamer
- add thread scope #1517 @yelhamer
- add process scope #1517 @yelhamer
- rules: change `meta.scope` to `meta.scopes` @yelhamer
- protobuf: add `Metadata.flavor` @williballenthin
- binja: add support for forwarded exports #1646 @xusheng6
- binja: add support for symtab names #1504 @xusheng6
- add com class/interface features #322 @Aayush-goel-04

### Breaking Changes

- remove the `SCOPE_*` constants in favor of the `Scope` enum #1764 @williballenthin
- protobuf: deprecate `RuleMetadata.scope` in favor of `RuleMetadata.scopes` @williballenthin
- protobuf: deprecate `Metadata.analysis` in favor of `Metadata.analysis2` that is dynamic analysis aware @williballenthin
- update freeze format to v3, adding support for dynamic analysis @williballenthin
- extractor: ignore DLL name for api features #1815 @mr-tz

### New Rules (34)

- nursery/get-ntoskrnl-base-address @mr-tz
Expand Down Expand Up @@ -49,9 +60,9 @@
-

### Bug Fixes
- ghidra: fix ints_to_bytes performance #1761 @mike-hunhoff
- ghidra: fix `ints_to_bytes` performance #1761 @mike-hunhoff
- binja: improve function call site detection @xusheng6
- binja: use binaryninja.load to open files @xusheng6
- binja: use `binaryninja.load` to open files @xusheng6
- binja: bump binja version to 3.5 #1789 @xusheng6

### capa explorer IDA Pro plugin
Expand Down
127 changes: 108 additions & 19 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
[![License](https://img.shields.io/badge/license-Apache--2.0-green.svg)](LICENSE.txt)

capa detects capabilities in executable files.
You run it against a PE, ELF, .NET module, or shellcode file and it tells you what it thinks the program can do.
You run it against a PE, ELF, .NET module, shellcode file, or a sandbox report and it tells you what it thinks the program can do.
For example, it might suggest that the file is a backdoor, is capable of installing services, or relies on HTTP to communicate.

Check out:
Expand Down Expand Up @@ -125,6 +125,96 @@ function @ 0x4011C0
...
```

Additionally, capa also supports analyzing [CAPE](https://github.com/kevoreilly/CAPEv2) sandbox reports for dynamic capabilty extraction.
In order to use this, you first submit your sample to CAPE for analysis, and then run capa against the generated report (JSON).

Here's an example of running capa against a packed binary, and then running capa against the CAPE report of that binary:

```yaml
$ capa 05be49819139a3fdcdbddbdefd298398779521f3d68daa25275cc77508e42310.exe
WARNING:capa.capabilities.common:--------------------------------------------------------------------------------
WARNING:capa.capabilities.common: This sample appears to be packed.
WARNING:capa.capabilities.common:
WARNING:capa.capabilities.common: Packed samples have often been obfuscated to hide their logic.
WARNING:capa.capabilities.common: capa cannot handle obfuscation well using static analysis. This means the results may be misleading or incomplete.
WARNING:capa.capabilities.common: If possible, you should try to unpack this input file before analyzing it with capa.
WARNING:capa.capabilities.common: Alternatively, run the sample in a supported sandbox and invoke capa against the report to obtain dynamic analysis results.
WARNING:capa.capabilities.common:
WARNING:capa.capabilities.common: Identified via rule: (internal) packer file limitation
WARNING:capa.capabilities.common:
WARNING:capa.capabilities.common: Use -v or -vv if you really want to see the capabilities identified by capa.
WARNING:capa.capabilities.common:--------------------------------------------------------------------------------

$ capa 05be49819139a3fdcdbddbdefd298398779521f3d68daa25275cc77508e42310.json

┍━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┑
│ ATT&CK Tactic │ ATT&CK Technique │
┝━━━━━━━━━━━━━━━━━━━━━━━━┿━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┥
│ CREDENTIAL ACCESS │ Credentials from Password Stores T1555 │
├────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤
│ DEFENSE EVASION │ File and Directory Permissions Modification T1222 │
│ │ Modify Registry T1112 │
│ │ Obfuscated Files or Information T1027 │
│ │ Virtualization/Sandbox Evasion::User Activity Based Checks T1497.002 │
├────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤
│ DISCOVERY │ Account Discovery T1087 │
│ │ Application Window Discovery T1010 │
│ │ File and Directory Discovery T1083 │
│ │ Query Registry T1012 │
│ │ System Information Discovery T1082 │
│ │ System Location Discovery::System Language Discovery T1614.001 │
│ │ System Owner/User Discovery T1033 │
├────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤
│ EXECUTION │ System Services::Service Execution T1569.002 │
├────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤
│ PERSISTENCE │ Boot or Logon Autostart Execution::Registry Run Keys / Startup Folder T1547.001 │
│ │ Boot or Logon Autostart Execution::Winlogon Helper DLL T1547.004 │
│ │ Create or Modify System Process::Windows Service T1543.003 │
┕━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┙

┍━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┑
│ Capability │ Namespace │
┝━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┿━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┥
│ check for unmoving mouse cursor (3 matches) │ anti-analysis/anti-vm/vm-detection │
│ gather bitkinex information │ collection/file-managers │
│ gather classicftp information │ collection/file-managers │
│ gather filezilla information │ collection/file-managers │
│ gather total-commander information │ collection/file-managers │
│ gather ultrafxp information │ collection/file-managers │
│ resolve DNS (23 matches) │ communication/dns │
│ initialize Winsock library (7 matches) │ communication/socket │
│ act as TCP client (3 matches) │ communication/tcp/client │
│ create new key via CryptAcquireContext │ data-manipulation/encryption │
│ encrypt or decrypt via WinCrypt │ data-manipulation/encryption │
│ hash data via WinCrypt │ data-manipulation/hashing │
│ initialize hashing via WinCrypt │ data-manipulation/hashing │
│ hash data with MD5 │ data-manipulation/hashing/md5 │
│ generate random numbers via WinAPI │ data-manipulation/prng │
│ extract resource via kernel32 functions (2 matches) │ executable/resource │
│ interact with driver via control codes (2 matches) │ host-interaction/driver │
│ get Program Files directory (18 matches) │ host-interaction/file-system │
│ get common file path (575 matches) │ host-interaction/file-system │
│ create directory (2 matches) │ host-interaction/file-system/create │
│ delete file │ host-interaction/file-system/delete │
│ get file attributes (122 matches) │ host-interaction/file-system/meta │
│ set file attributes (8 matches) │ host-interaction/file-system/meta │
│ move file │ host-interaction/file-system/move │
│ find taskbar (3 matches) │ host-interaction/gui/taskbar/find │
│ get keyboard layout (12 matches) │ host-interaction/hardware/keyboard │
│ get disk size │ host-interaction/hardware/storage │
│ get hostname (4 matches) │ host-interaction/os/hostname │
│ allocate or change RWX memory (3 matches) │ host-interaction/process/inject │
│ query or enumerate registry key (3 matches) │ host-interaction/registry │
│ query or enumerate registry value (8 matches) │ host-interaction/registry │
│ delete registry key │ host-interaction/registry/delete │
│ start service │ host-interaction/service/start │
│ get session user name │ host-interaction/session │
│ persist via Run registry key │ persistence/registry/run │
│ persist via Winlogon Helper DLL registry key │ persistence/registry/winlogon-helper │
│ persist via Windows service (2 matches) │ persistence/service │
┕━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┙
```

capa uses a collection of rules to identify capabilities within a program.
These rules are easy to write, even for those new to reverse engineering.
By authoring rules, you can extend the capabilities that capa recognizes.
Expand All @@ -135,31 +225,30 @@ Here's an example rule used by capa:
```yaml
rule:
meta:
name: hash data with CRC32
namespace: data-manipulation/checksum/crc32
name: create TCP socket
namespace: communication/socket/tcp
authors:
- [email protected]
scope: function
- [email protected]
- [email protected]
- [email protected]
scopes:
static: basic block
dynamic: call
mbc:
- Data::Checksum::CRC32 [C0032.001]
- Communication::Socket Communication::Create TCP Socket [C0001.011]
examples:
- 2D3EDC218A90F03089CC01715A9F047F:0x403CBD
- 7D28CB106CB54876B2A5C111724A07CD:0x402350 # RtlComputeCrc32
- 7EFF498DE13CC734262F87E6B3EF38AB:0x100084A6
- Practical Malware Analysis Lab 01-01.dll_:0x10001010
features:
- or:
- and:
- mnemonic: shr
- number: 6 = IPPROTO_TCP
- number: 1 = SOCK_STREAM
- number: 2 = AF_INET
- or:
- number: 0xEDB88320
- bytes: 00 00 00 00 96 30 07 77 2C 61 0E EE BA 51 09 99 19 C4 6D 07 8F F4 6A 70 35 A5 63 E9 A3 95 64 9E = crc32_tab
- number: 8
- characteristic: nzxor
- and:
- number: 0x8320
- number: 0xEDB8
- characteristic: nzxor
- api: RtlComputeCrc32
- api: ws2_32.socket
- api: ws2_32.WSASocket
- api: socket
- property/read: System.Net.Sockets.TcpClient::Client
```

The [github.com/mandiant/capa-rules](https://github.com/mandiant/capa-rules) repository contains hundreds of standard library rules that are distributed with capa.
Expand Down
Empty file added capa/capabilities/__init__.py
Empty file.
Loading
Loading