Skip to content

Commit

Permalink
Change grep flag to pass/skip, update usage and tests (#11)
Browse files Browse the repository at this point in the history
  • Loading branch information
vearutop authored Dec 24, 2023
1 parent adddc26 commit 9a21465
Show file tree
Hide file tree
Showing 10 changed files with 404 additions and 41 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/golangci-lint.yml
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ jobs:
uses: golangci/[email protected]
with:
# Required: the version of golangci-lint is required and must be specified without patch version: we always use the latest patch version.
version: v1.54.1
version: v1.55.2

# Optional: working directory, useful for monorepos
# working-directory: somedir
Expand Down
136 changes: 136 additions & 0 deletions .github/workflows/test-unit.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,136 @@
# This script is provided by github.com/bool64/dev.
name: test-unit
on:
push:
branches:
- master
- main
pull_request:

# Cancel the workflow in progress in newer build is about to start.
concurrency:
group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}
cancel-in-progress: true

env:
GO111MODULE: "on"
RUN_BASE_COVERAGE: "on" # Runs test for PR base in case base test coverage is missing.
COV_GO_VERSION: 1.21.x # Version of Go to collect coverage
TARGET_DELTA_COV: 90 # Target coverage of changed lines, in percents
jobs:
test:
strategy:
matrix:
go-version: [ 1.19.x, 1.20.x, 1.21.x ]
runs-on: ubuntu-latest
steps:
- name: Install Go stable
if: matrix.go-version != 'tip'
uses: actions/setup-go@v4
with:
go-version: ${{ matrix.go-version }}

- name: Install Go tip
if: matrix.go-version == 'tip'
run: |
curl -sL https://storage.googleapis.com/go-build-snap/go/linux-amd64/$(git ls-remote https://github.com/golang/go.git HEAD | awk '{print $1;}').tar.gz -o gotip.tar.gz
ls -lah gotip.tar.gz
mkdir -p ~/sdk/gotip
tar -C ~/sdk/gotip -xzf gotip.tar.gz
~/sdk/gotip/bin/go version
echo "PATH=$HOME/go/bin:$HOME/sdk/gotip/bin/:$PATH" >> $GITHUB_ENV
- name: Checkout code
uses: actions/checkout@v3

- name: Go cache
uses: actions/cache@v3
with:
# In order:
# * Module download cache
# * Build cache (Linux)
path: |
~/go/pkg/mod
~/.cache/go-build
key: ${{ runner.os }}-go-cache-${{ hashFiles('**/go.sum') }}
restore-keys: |
${{ runner.os }}-go-cache
- name: Restore base test coverage
id: base-coverage
if: matrix.go-version == env.COV_GO_VERSION && github.event.pull_request.base.sha != ''
uses: actions/cache@v2
with:
path: |
unit-base.txt
# Use base sha for PR or new commit hash for master/main push in test result key.
key: ${{ runner.os }}-unit-test-coverage-${{ (github.event.pull_request.base.sha != github.event.after) && github.event.pull_request.base.sha || github.event.after }}

- name: Run test for base code
if: matrix.go-version == env.COV_GO_VERSION && env.RUN_BASE_COVERAGE == 'on' && steps.base-coverage.outputs.cache-hit != 'true' && github.event.pull_request.base.sha != ''
run: |
git fetch origin master ${{ github.event.pull_request.base.sha }}
HEAD=$(git rev-parse HEAD)
git reset --hard ${{ github.event.pull_request.base.sha }}
(make test-unit && go tool cover -func=./unit.coverprofile > unit-base.txt) || echo "No test-unit in base"
git reset --hard $HEAD
- name: Test
id: test
run: |
make test-unit
go tool cover -func=./unit.coverprofile > unit.txt
TOTAL=$(grep 'total:' unit.txt)
echo "${TOTAL}"
echo "total=$TOTAL" >> $GITHUB_OUTPUT
- name: Annotate missing test coverage
id: annotate
if: matrix.go-version == env.COV_GO_VERSION && github.event.pull_request.base.sha != ''
run: |
curl -sLO https://github.com/vearutop/gocovdiff/releases/download/v1.4.2/linux_amd64.tar.gz && tar xf linux_amd64.tar.gz && rm linux_amd64.tar.gz
gocovdiff_hash=$(git hash-object ./gocovdiff)
[ "$gocovdiff_hash" == "c37862c73a677e5a9c069470287823ab5bbf0244" ] || (echo "::error::unexpected hash for gocovdiff, possible tampering: $gocovdiff_hash" && exit 1)
git fetch origin master ${{ github.event.pull_request.base.sha }}
REP=$(./gocovdiff -mod github.com/$GITHUB_REPOSITORY -cov unit.coverprofile -gha-annotations gha-unit.txt -delta-cov-file delta-cov-unit.txt -target-delta-cov ${TARGET_DELTA_COV})
echo "${REP}"
cat gha-unit.txt
DIFF=$(test -e unit-base.txt && ./gocovdiff -mod github.com/$GITHUB_REPOSITORY -func-cov unit.txt -func-base-cov unit-base.txt || echo "Missing base coverage file")
TOTAL=$(cat delta-cov-unit.txt)
echo "rep<<EOF" >> $GITHUB_OUTPUT && echo "$REP" >> $GITHUB_OUTPUT && echo "EOF" >> $GITHUB_OUTPUT
echo "diff<<EOF" >> $GITHUB_OUTPUT && echo "$DIFF" >> $GITHUB_OUTPUT && echo "EOF" >> $GITHUB_OUTPUT
echo "total<<EOF" >> $GITHUB_OUTPUT && echo "$TOTAL" >> $GITHUB_OUTPUT && echo "EOF" >> $GITHUB_OUTPUT
- name: Comment test coverage
continue-on-error: true
if: matrix.go-version == env.COV_GO_VERSION && github.event.pull_request.base.sha != ''
uses: marocchino/sticky-pull-request-comment@v2
with:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
header: unit-test
message: |
### Unit Test Coverage
${{ steps.test.outputs.total }}
${{ steps.annotate.outputs.total }}
<details><summary>Coverage of changed lines</summary>
${{ steps.annotate.outputs.rep }}
</details>
<details><summary>Coverage diff with base branch</summary>
${{ steps.annotate.outputs.diff }}
</details>
- name: Store base coverage
if: ${{ github.ref == 'refs/heads/master' || github.ref == 'refs/heads/main' }}
run: cp unit.txt unit-base.txt

- name: Upload code coverage
if: matrix.go-version == env.COV_GO_VERSION
uses: codecov/codecov-action@v1
with:
file: ./unit.coverprofile
flags: unittests
5 changes: 4 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
#GOLANGCI_LINT_VERSION := "v1.54.1" # Optional configuration to pinpoint golangci-lint version.
#GOLANGCI_LINT_VERSION := "v1.55.2" # Optional configuration to pinpoint golangci-lint version.

# The head of Makefile determines location of dev-go to include standard targets.
GO ?= go
Expand Down Expand Up @@ -33,9 +33,12 @@ BUILD_LDFLAGS=-s -w

-include $(DEVGO_PATH)/makefiles/main.mk
-include $(DEVGO_PATH)/makefiles/lint.mk
-include $(DEVGO_PATH)/makefiles/test-unit.mk
-include $(DEVGO_PATH)/makefiles/build.mk
-include $(DEVGO_PATH)/makefiles/release-assets.mk
-include $(DEVGO_PATH)/makefiles/reset-ci.mk

# Add your custom targets here.

## Run tests
test: test-unit
49 changes: 31 additions & 18 deletions cmd/catp/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,36 +18,49 @@ wget https://github.com/bool64/progress/releases/latest/download/linux_amd64.tar
## Usage

```
catp dev, go1.22rc1 CGO_ZSTD
catp prints contents of files to STDOUT or dir/file output,
while printing current progress status to STDERR.
It can decompress data from .gz and .zst files.
Usage of catp:
catp [OPTIONS] PATH ...
-dbg-cpu-prof string
write first 10 seconds of CPU profile to file
write first 10 seconds of CPU profile to file
-dbg-mem-prof string
write heap profile to file after 10 seconds
-grep value
grep pattern, may contain multiple OR patterns separated by \|,
each -grep value is added with AND logic, akin to extra '| grep foo',
for example, you can use '-grep bar\|baz -grep foo' to only keep lines that have (bar OR baz) AND foo
write heap profile to file after 10 seconds
-no-progress
disable progress printing
disable progress printing
-out-dir string
output to directory instead of STDOUT
files will be written to out dir with original base names
disables output flag
output to directory instead of STDOUT
files will be written to out dir with original base names
disables output flag
-output string
output to file instead of STDOUT
output to file instead of STDOUT
-parallel int
number of parallel readers if multiple files are provided
lines from different files will go to output simultaneously
use 0 for multi-threaded zst decoder (slightly faster at cost of more CPU) (default 1)
number of parallel readers if multiple files are provided
lines from different files will go to output simultaneously (out of order of files, but in order of lines in each file)
use 0 for multi-threaded zst decoder (slightly faster at cost of more CPU) (default 1)
-pass value
filter matching, may contain multiple AND patterns separated by ^,
if filter matches, line is passed to the output (unless filtered out by -skip)
each -pass value is added with OR logic,
for example, you can use "-pass bar^baz -pass foo" to only keep lines that have (bar AND baz) OR foo
-progress-json string
write current progress to a file
write current progress to a file
-skip value
filter matching, may contain multiple AND patterns separated by ^,
if filter matches, line is removed from the output (even if it passed -pass)
each -skip value is added with OR logic,
for example, you can use "-skip quux^baz -skip fooO" to skip lines that have (quux AND baz) OR fooO
-version
print version and exit
print version and exit
```

## Examples

Feed a file into `jq` field extractor.
Feed a file into `jq` field extractor with progress printing.

```
catp get-key.log | jq .context.callback.Data.Nonce > get-key.jq
Expand All @@ -65,7 +78,7 @@ Run log filtering (lines containing `foo bar` or `baz`) on multiple files in bac
new file.

```
screen -dmS foo12 ./catp -output ~/foo-2023-07-12.log -grep "foo bar\|baz" /home/logs/server-2023-07-12*
screen -dmS foo12 ./catp -output ~/foo-2023-07-12.log -pass "foo bar" -pass "baz" /home/logs/server-2023-07-12*
```

```
Expand Down
Loading

0 comments on commit 9a21465

Please sign in to comment.