Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

license check: add license check skip id #28

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

license check: add license check skip id #28

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
e62d888
checks: use linguist to detect file types
acomodi May 21, 2021
26ef65f
add license to Dockerfile
acomodi May 21, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .gitattributes
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# Linguist - vendored override

third_party/* linguist-vendored

# Linguist - generated override

checks/tests/license/test-missing-spdx-generated.v linguist-generated
21 changes: 21 additions & 0 deletions checks/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,31 @@
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
#
# Copyright (C) 2020 The SymbiFlow Authors.
#
# Use of this source code is governed by a ISC-style
# license that can be found in the LICENSE file or at
# https://opensource.org/licenses/ISC
#
# SPDX-License-Identifier: ISC

FROM python:slim-buster

# Install git
RUN apt-get update -qq \
&& DEBIAN_FRONTEND=noninteractive apt-get -y install --no-install-recommends \
ca-certificates \
git \
make \
cmake \
pkg-config \
libicu-dev \
zlib1g-dev \
libcurl4-openssl-dev \
ruby-dev \
libssl-dev \
build-essential \
&& gem install github-linguist \
&& apt-get autoclean && apt-get clean && apt-get -y autoremove \
&& update-ca-certificates \
&& rm -rf /var/lib/apt/lists/*
Expand Down
15 changes: 13 additions & 2 deletions checks/checks.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@
import re
import sys
import tempfile
import subprocess
import json


__path__ = pathlib.Path(__file__).resolve().parent
Expand Down Expand Up @@ -417,6 +419,15 @@ def main(args):
logging.debug('Starting search in: %s', root_dir)

errors = {}

json_data = subprocess.check_output("github-linguist --json", shell=True).decode('utf8')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The output of github-linguist doesn't seem to actually specify if a file is vendored or generated.

Looking at https://github.com/github/linguist/blob/master/bin/github-linguist#L61-L87

tansell@tansell-glapstation:~/github/SymbiFlow/actions$ github-linguist third_party/make-env/os.mk 
os.mk: 95 lines (85 sloc)
  type:      Text
  mime type: text/plain
  language:  Makefile
  appears to be a vendored file

However the generated attribute doesn't seem to be working?

tansell@tansell-glapstation:~/github/SymbiFlow/actions$ git check-attr --all checks/tests/license/test-missing-spdx-generated.v
checks/tests/license/test-missing-spdx-generated.v: linguist-generated: set

tansell@tansell-glapstation:~/github/SymbiFlow/actions$ github-linguist checks/tests/license/test-missing-spdx-generated.v
test-missing-spdx-generated.v: 5 lines (3 sloc)
  type:      Text
  mime type: text/plain
  language:  Verilog

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have dove a bit in the linguist code, and, for what I understand, the .gitattributes file is taken into account only when dealing with full repository statistics, therefore it does not have an effect when dealing with single files stats, hence the verilog file is not detected as generated.

Furthermore, when the full-repo statistics are generated, only programming and markup files are taken to report statistics, and data and nil types are left out, as written here.
It turns out that YAML is defined as a data type, and all yaml files are currently excluded by the license check.

I think we can still use linguist to detect the file types, but not to correctly override the generated files.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@acomodi - I think we probably want to extend linguist to support outputting the information we want. I started looking at that and ran into issues understanding how to use the Ruby rugged module.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

github-linguist/linguist#5382

files_data = json.loads(json_data)

ftypes = {}
for ftype, fpaths in files_data.items():
for fpath in fpaths:
ftypes[fpath] = ftype

for root, dirs, files in os.walk(root_dir):
rpath = pathlib.Path(root).resolve()
fdebug(rpath, 'Searching')
Expand Down Expand Up @@ -463,9 +474,9 @@ def main(args):
fwarn(fpath, 'Skipping nonfile')
continue

ftype = detect_file_type(fpath)
ftype = ftypes.get(os.path.relpath(fpath, root_dir), None)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Either we should remove the detect_file_type function or do something like;

Suggested change
ftype = ftypes.get(os.path.relpath(fpath, root_dir), None)
ftype = ftypes.get(os.path.relpath(fpath, root_dir), detect_file_type(fpath))

if ftype is None:
finfo(fpath, 'Skipping unknown file type')
finfo(fpath, 'Skipping generated or unknown file')
continue

ferrors = []
Expand Down
5 changes: 5 additions & 0 deletions checks/tests/license/test-missing-spdx-generated.v
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.