Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

documentation: fix a typo in Quickstart #50

Open
wants to merge 13 commits into
base: dev
Choose a base branch
from
17 changes: 17 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,23 @@

This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [1.24.2] -
- Documentation consistency cleanup, changes are mostly aesthetic, the content is not changed:
- Expanding use of fixed width fonts for file, function, argument, variable, macro, ... names.
- Fixed missing capitalization and punctuation, mostly for lists.
- Capitalization of acronyms and proper nouns:
- python => Python,
- riscof => RISCOF,
- isa => ISA,
- yaml => YAML,
- verilator => Verilator,
- cli => CLI,
- makefile => Makefile (depending on context),
- ...
- Few spelling and grammar fixes.
- Few instances where names in documentation did not match names in the source code.
This were either copy/paste errors or code changes not propagated to documentation.

## [1.24.1] - 2022-07-19
- Account for the same test to be included with both XLEN variants in the isa generation.
- Add markdown report for coverage statistics.
Expand Down
2 changes: 1 addition & 1 deletion CONTRIBUTING.rst
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ be followed while assigning a new version number :
Note: You can have either a patch or minor or major update.
Note: In case of a conflict, the maintainers will decide the final version to be assigned.

To update the version of the python package for deployment you can use the following::
To update the version of the Python package for deployment you can use the following::

$ bumpversion --no-tag --config-file setup.cfg patch # possible: major / minor / patch

Expand Down
10 changes: 5 additions & 5 deletions PLUGINS.rst
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,9 @@
List of Reference RISCOF Plugins
================================

This section provides a list of pre-built riscof-plugins which users can refer
to, to build plugins for their own DUT
This section provides a list of pre-built RISCOF plugins which users can refer
to, to build plugins for their own DUT:

- Spike: https://gitlab.com/incoresemi/riscof-plugins/-/tree/master/spike_parallel
- SAIL_cSim: https://gitlab.com/incoresemi/riscof-plugins/-/blob/master/sail_cSim/README.md
- InCore Plugins: https://gitlab.com/incoresemi/riscof-plugins (This is a collection of riscof based plugins for various targets hosted purely for reference.)
- `Spike <https://gitlab.com/incoresemi/riscof-plugins/-/tree/master/spike_parallel>`_,
- `SAIL_cSim <https://gitlab.com/incoresemi/riscof-plugins/-/blob/master/sail_cSim/README.md>`_,
- `InCore Plugins <https://gitlab.com/incoresemi/riscof-plugins>`_ (This is a collection of RISCOF based plugins for various targets hosted purely for reference.)
2 changes: 1 addition & 1 deletion docs/source/TestFormatSpec.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,7 @@ significant amount of the framework shall depend on the existence of these macro
`RVTEST_ISA(isa_str)`::

- defines the Test Virtual Machine (TVM, the ISA being tested) +
- empty macro to specify the isa required for compilation of the test. +
- empty macro to specify the ISA required for compilation of the test. +
- this is mandated to be present at the start of the test.


Expand Down
14 changes: 7 additions & 7 deletions docs/source/arch-tests.rst
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ Running RISCV-ARCH-TESTS
The following guide provides a walkthrough on how to run the tests available at
the `riscv-arch-test <https://github.com/riscv/riscv-arch-test>`_ repository.

The following assumes you have installed riscof as a cli on your system. If not, then please refer
The following assumes you have installed RISCOF as a CLI on your system. If not, then please refer
to the :ref:`install_riscof` section for the same.


Expand All @@ -17,7 +17,7 @@ Setup all the DUT and Ref Plugins

1. You will first need to install the SAIL C-emulator on your system. You can refer to the
:ref:`plugin_models` section for steps on installing the SAIL C-emulator.
2. You will then need to download/clone the ``sail_cSim`` riscof plugin. You can do this with the
2. You will then need to download/clone the ``sail_cSim`` RISCOF plugin. You can do this with the
following command:

.. code-block:: console
Expand All @@ -28,13 +28,13 @@ Setup all the DUT and Ref Plugins
You will need the path of the `riscof-plugins` directory from the above repo for the next
steps.

3. You will also need to create a riscof-plugin for you own DUT. If you haven't already done so,
3. You will also need to create a RISCOF plugin for you own DUT. If you haven't already done so,
please refer to the :ref:`plugins` section for details on building one.

Create a config.ini file
------------------------
Create a ``config.ini`` file
----------------------------

1. You will need to create a `config.ini` file with the following contents.
1. You will need to create a ``config.ini`` file with the following contents.

.. code-block:: ini

Expand Down Expand Up @@ -77,7 +77,7 @@ Running Tests with RISCOF

The above step will first create a database of the all tests from the ``rv32i_m`` directory
(recursively). This database can be found in the `riscof_work/database.yaml` file that is
generated. From this database, RISCOF selects the applicable test depending on the ISA yaml
generated. From this database, RISCOF selects the applicable test depending on the ISA YAML
provided and then runs them first on the DUT and then on the REFERENCE plugins. The end, it
compares the signatures and provides an html report of the result.

Expand Down
189 changes: 94 additions & 95 deletions docs/source/commands.rst
Original file line number Diff line number Diff line change
Expand Up @@ -8,81 +8,81 @@ RISCOF Commands
This section provides an overview and working of the various sub commands available in RISCOF.
The current list of subcommands includes:

- arch-tests
- coverage
- gendb
- setup
- validateyaml
- testlist
- run

arch-tests
----------
- ``arch-tests``
- ``coverage``
- ``gendb``
- ``setup``
- ``validateyaml``
- ``testlist``
- ``run``

``arch-tests``
--------------

This command is used to clone and update the tests from the official `riscv-arch-test <https://github.com/riscv/riscv-arch-test>`_ repository.

This command requires one of the following flags to be specified from the cli.
This command requires one of the following flags to be specified from the CLI.

- show-version: Display the current version of the official suite present at the specified directory path.
- clone: Clone the suite from github.
- update: Update the suite to reflect latest changes from github.
- ``--show-version``: Display the current version of the official suite present at the specified directory path.
- ``--clone``: Clone the suite from github.
- ``--update``: Update the suite to reflect latest changes from github.

Optional arguments from the cli:
Optional arguments from the CLI:

- get-version: The specific version of the tests to be fetched. Can be used with both the clone and
- ``--get-version``: The specific version of the tests to be fetched. Can be used with both the clone and
update flags. The latest release is fetched if skipped.
- dir: The path to the directory where the suite is to be cloned to. Defaults to
- ``--dir``: The path to the directory where the suite is to be cloned to. Defaults to
``./riscv-arch-test`` if skipped.

coverage
--------
``coverage``
------------

This command is used to collect the ISA coverage metrics of a given test-suite and generate a coverage
report in html.

This command will require the following inputs from the cli:
This command will require the following inputs from the CLI:

- suite: The test suite path on which coverage needs to be run
- env: The path to the environment directory containing the suite-specific header files.
- cgf: list of covergroup-format files specifying the coverpoints that need to be covered by the
the suite
- ``--suite``: The test suite path on which coverage needs to be run.
- ``--env``: The path to the environment directory containing the suite-specific header files.
- ``--cgf``: The list of covergroup-format files specifying the coverpoints that need to be covered by the suite.

Optional arguments from the cli:
Optional arguments from the CLI:

- config: path to the ``config.ini`` file. Defaults to ``./config.ini`` if skipped.
- work-dir: path to the working directory where all artifacts need to be dumped. Defaults to
``./riscof_work``
- no-browser: when used, RISCOF skips automatically opening the html report in the default web
- ``--config``: The path to the ``config.ini`` file. Defaults to ``./config.ini`` if skipped.
- ``--work-dir``: The path to the working directory where all artifacts need to be dumped. Defaults to
``./riscof_work``.
- ``--no-browser``: When used, RISCOF skips automatically opening the html report in the default web
browser.

The coverage command simply passes the cgf files to the reference plugin's runTests function. The
Reference plugin is responsible to generating a yaml based coverage report for each test using ``riscv-isac``.
The yaml file should be named ``coverage.rpt``. The ``riscv-isac`` run will also generate a data-propagation
Reference plugin is responsible to generating a YAML based coverage report for each test using ``riscv-isac``.
The YAML file should be named ``coverage.rpt``. The ``riscv-isac`` run will also generate a data-propagation
report which should be named as ``ref.md``.

Once the coverage files for each test has been generated, RISCOF will parse through the working
directories and merge all the ``coverage.rpt`` files to create a single yaml coverage report:
directories and merge all the ``coverage.rpt`` files to create a single YAML coverage report:
``suite_coverage.rpt``. RISCOF then also converts this to an HTML based reports and open it on the
default web-browser.

For a example on using this feature please refer to the :ref:`coverage` section.

gendb
-----
``gendb``
---------

This command is used to generate a database yaml file for all tests available in the test-suite. The
commands requires the following inputs from the cli:
This command is used to generate a database YAML file for all tests available in the test-suite. The
commands requires the following inputs from the CLI:

- suite: The test suite path for which database needs to be generated.
- work-dir: path to the working directory where all artifacts need to be dumped. Defaults to
``./riscof_work``
- ``--suite``: The test suite path for which database needs to be generated.
- ``--work-dir``: The path to the working directory where all artifacts need to be dumped. Defaults to
``./riscof_work``.

This utility parses the ``suite`` directory and collects all the .S files. For each .S file, the
utility will parse the test and collect informations from various macros such as RVTEST_ISA,
RVTEST_CASE, etc. For each test the utility will create a new entry in a dictionary which captures
the different parts of the tests, the enabling conditions of each part, the coverage contributions
of each part, any compile macros required for each part and muc more.
of each part, any compile macros required for each part and much more.

The generated database yaml will follow the syntax described in section :ref:`database`.
The generated database YAML will follow the syntax described in section :ref:`database`.

The output of this utility is a ``database.yaml`` located in the ``work_dir`` directory. This file is
used by RISCOF to select and filter tests based on input DUT configuration.
Expand All @@ -91,95 +91,94 @@ used by RISCOF to select and filter tests based on input DUT configuration.
<https://github.com/riscv/riscv-arch-test/blob/master/spec/TestFormatSpec.adoc>`_ set forth
by the riscv-arch-test SIG.

setup
-----
``setup``
---------

The setup command is used to generate a series of Template files that are required by RISCOF.
These files are meant to provide ease to users integrating their DUT to RISCOF for the first time
and should be modified by the users.

The setup utility takes in the following optional inputs from the cli:
The setup utility takes in the following optional inputs from the CLI:

- dutname: name of the dut for running the tests on. The utility will use this name to create a
- ``--dutname``: The name of the DUT for running the tests on. The utility will use this name to create a
template plugin directory with all the relevant files. These files will have to be modified by
the user. Defaults to "spike" when skipped.
- refname: name of the reference plugin to be used in RISCOF. The utility will use this name to
the user. Defaults to ``spike`` when skipped.
- ``--refname``: The name of the reference plugin to be used in RISCOF. The utility will use this name to
create a reference plugin directory with all the relevant files.


The setup utility will also create a sample config.ini file using the above inputs.

validateyaml
------------
``validateyaml``
----------------

This command simply performs a validation of the isa spec and the platform pspec yamls of the DUT
as mentioned in the ``config.ini`` using riscv-config. The outputs are checked version of the yamls in
the directory pointed by ``work_dir``
This command simply performs a validation of the ISA ``ispec`` and the platform ``pspec`` YAMLs of the DUT
as mentioned in the ``config.ini`` using riscv-config. The outputs are checked for the version of the YAMLs in
the directory pointed to by ``work_dir``.

testlist
--------
``testlist``
------------

This command is used to filter tests from the database.yaml based on the configuration of DUT
present in the isa and platform yamls as mentioned in the ``config.ini``. This command will require
the following inputs from the cli:
This command is used to filter tests from the ``database.yaml`` based on the configuration of DUT
present in the ISA and platform YAMLs as mentioned in the ``config.ini``. This command will require
the following inputs from the CLI:

- suite: The test suite from which the test need to filtered.
- ``--suite``: The test suite from which the tests need to be filtered.

This command takes the following optional inputs from cli
This command takes the following optional inputs from CLI:

- config: path to the ``config.ini`` file. Defaults to ``./config.ini`` if skipped.
- work-dir: path to the working directory where all artifacts need to be dumped. Defaults to
``./riscof_work``
- ``--config``: The path to the ``config.ini`` file. Defaults to ``./config.ini`` if skipped.
- ``--work-dir``: The path to the working directory where all artifacts need to be dumped. Defaults to
``./riscof_work``.

The utility first creates a ``database.yaml`` for the input suite. For each test in the database yaml,
this utility will check if the conditions of any parts of a test are enabled based on the isa and
platform yaml specs of the DUT. If any part is enabled, then the corresponding test is entered into
The utility first creates a ``database.yaml`` for the input suite. For each test in the database YAML,
this utility will check if the conditions of any parts of a test are enabled based on the ISA and
platform YAML specs of the DUT. If any part is enabled, then the corresponding test is entered into
the teslist along with the respective coverage labels and compile macros.

The utility will dump the test list in the ``testlist.yaml`` file in the ``work_dir`` directory. This
yaml will follow the same syntax as defined in the :ref:`testlist` section.
YAML will follow the same syntax as defined in the :ref:`testlist` section.

run
---
``run``
-------

This is probably the primary command of RISCOF which is going to be widely used. This command is
currently responsible for first validating the inputs yamls,
currently responsible for first validating the inputs YAMLs,
creating a database of the tests in the ``suite`` directory, generate a
filtered test-list, run the tests on the DUT and then the Reference Plugins, and finally compare the
generated signatures and present an html report of the findings.

The following inputs are required on the cli by this command:
The following inputs are required on the CLI by this command:

- suite: The test suite path on which coverage needs to be run
- env: The path to the environment directory containing the suite-specific header files.
- ``--suite``: The test suite path on which coverage needs to be run
- ``--env``: The path to the environment directory containing the suite-specific header files.

Optional arguments from the cli:
Optional arguments from the CLI:

- config: path to the ``config.ini`` file. Defaults to ``./config.ini`` if skipped.
- work-dir: path to the working directory where all artifacts need to be dumped. Defaults to
``./riscof_work``
- no-browser: when used, RISCOF skips automatically opening the html report in the default web
browser.
- dbfile: The path to the database file, from which testlist will be generated
- testfile: The path to the testlist file on which tests will be run
- no-ref-run: when used, RISCOF will not run tests on Reference and will quit before signatures comparison
- no-dut-run: when used, RISCOF will not run tests on DUT and will quit before signatures comparison
- no-clean: when used, RISCOF will not remove the ``work_dir``, if it exists.
- ``--config``: The path to the ``config.ini`` file. Defaults to ``./config.ini`` if skipped.
- ``--work-dir``: The path to the working directory where all artifacts need to be dumped. Defaults to
``./riscof_work``.
- ``--no-browser``: When used, RISCOF skips automatically opening the html report in the default web browser.
- ``--dbfile``: The path to the database file, from which testlist will be generated.
- ``--testfile``: The path to the testlist file on which tests will be run.
- ``--no-ref-run``: When used, RISCOF will not run tests on Reference and will quit before signatures comparison.
- ``--no-dut-run``: When used, RISCOF will not run tests on DUT and will quit before signatures comparison.
- ``--no-clean``: When used, RISCOF will not remove the ``work_dir``, if it exists.

The ``work_dir`` is cleaned by default. However, if one of ``no-clean``, ``testfile`` or ``dbfile``
are specified, it is preserved as is.

All artifacts of this command are generated in the ``work_dir`` directory. Typicall artifacts will
All artifacts of this command are generated in the ``work_dir`` directory. Typically artifacts will
include:

==================== =============================================================
Artifact Description
==================== =============================================================
database.yaml This is the database of all the tests in the suite directory
Makefile.DUT* This is the Makefile generated by the DUT Plugin.
Makefile.Reference* This is the Makefile generated by the Reference Plugin.
report.html The final report generated at the end of the run after signature comparison
yaml files verified and checked yaml versions of the input isa and platform yamls
test_list.yaml This list of filtered tests from the database.yaml
src directory this will include a directory for each test in the test_list.yaml. Each test-directory will include the test, compiled-binaries, signatures from both the DUT and the Reference Plugin.
==================== =============================================================
======================== =============================================================
Artifact Description
======================== =============================================================
``database.yaml`` This is the database of all the tests in the suite directory.
``Makefile.DUT*`` This is the Makefile generated by the DUT Plugin.
``Makefile.Reference*`` This is the Makefile generated by the Reference Plugin.
``report.html`` The final report generated at the end of the run after signature comparison.
YAML files Verified and checked YAML versions of the input ISA and Platform YAMLs.
``test_list.yaml`` This list of filtered tests from the ``database.yaml``.
src directory This will include a directory for each test in the ``test_list.yaml``. Each test-directory will include the test, compiled-binaries, signatures from both the DUT and the Reference Plugin.
======================== =============================================================
Loading