Dependency Analyzer - User's documentation

Introduction

This project is a service, which provides information about built artifacts and analyse the projects' dependencies. It can lookup the Red Hat build artifacts and inform the users about alternatives instead of the artifacts used in their projects, produces dependency reports of artifacts and resolves dependency tree.

Dependency Analyzer also maintains a database of blocklisted artifacts, which should help the user to decide, which artifacts to use in their projects.

This project is hosted on GitHub and developed mainly by the JBoss Productization team, but external contributors are welcome.

Blocklist

The service maintains the database of the blocklisted artifacts. This data in the database can be changed ony by authenticated users. In future authorization may be added.

The meaning of the this lists is:

Blocklist - contains artifacts, which should not be used in the projects. There are typically artifacts, which cannot be built, has unacceptable license, contains vulnerabilities, etc. The inserted artifacts can be of community versions of artifacts (e.g. 4.2.18.Final) which will blocklist all redhat builds of that version or of redhat version (e.g. 4.2.18.Final-redhat-1) which will blocklist the specific version.

Lookup of built artifacts

Dependency Analyzer can lookup the Red Hat built artifacts and tell the user if an artifact was built or not and provide them alternative versions of the artifact. The user gets the information about:

The latest built version of the artifact
All built versions of artifacts with the same GroupId, ArtifactId
Blocklist status of the requested artifact

Dependency reports

The system is able to provide information about the dependencies of a project. It can analyse a repository specified by the SCM URL (+ revision). This feature extends the lookup and it resolves whole dependency tree and provides information for every artifact in the tree and also some helper data like how many dependencies are not built.

Data source

Dependency Analyzer is using Indy, an artifact proxy for maven, to get information about built artifacts.

Currently data about built artifacts are gathered from the public product repository (maven.repository.redhat.com).

The integration with Brew/MEAD repositories is done and Indy will proxy also artifacts from the candidate tags.

Dependency Analyzer also provides built artifacts from the Project Newcastle build system.

Interfaces

Dependency Analyzer is a server side application and there are two ways how to use it. A REST API, which provides the main logic of the system and provides an easy way to use it in other applications. A CLI tool, which is more handy for the manual usage. The CLI tool is mainly a thin client, which gets all data from the service using the REST API.

REST API

The REST API is based on a Swagger documentation, which provides an easy way to have an up to date documentation and also a simple way to try the REST API from the browser. The path to the Swagger documentation is /da/doc. From this endpoint there is also link to the latest API version.

CLI tool

The CLI tool is more handy for the manual or script usage. Most of the data is obtained from the server application, so it is not possible to use the CLI tool without connection to the server. The tool is a collection of simple Python scripts, which requires to have installed interprets for Python 3.5 and above.

CLI tool manual

In this section will be described how to use the CLI tool and what are its options.

Installation

The easiest way to make the CLI tool running is to download the ZIP archive from GitHub, which contains all the required support scripts and CLI tool. The latest versions of the scripts are always placed in the folder /cli. When you download the ZIP file, you can unzip the file and start using it. How to do that?
1. Unzip the CLI: unzip da-cli-tools-2.0.0-pack.zip
2. Enter the folder with CLI: cd da-cli-tools-2.0.0
3. Change permissions: chmod +x *.py
Second option is to clone Dependency Analyzer repository via git:
1. Clone the repository: git clone https://github.com/project-ncl/dependency-analysis
2. Enter the directory with the CLI: cd dependency-analysis/cli-wrap/src/main/resources/cli/
3. Change permissions: chmod +x *.py
After unpacking/clonning CLI tool, you need to configure servers.
You have to specify DA server and KeyCloak authentication server. It can be done by inserting URL addresses of these servers to file config.json. DA server should be filled into "daServer" field. This server has to support WebSocket and HTTP protocol. Keycloak server should be in field "keycloakServer". There are also "keycloakClientId" and "keycloakRealm" fields, which should be filled by your KeyCloak configuration for proper working scripts. DA server can be also set as environmental variable ($ export DA_SERVER=...).

Note that DA server must be inserted protocol-less because services are using different protocols.
Before first run
Ensure that you have installed python3 in version 3.5+, requests, websockets and asyncio modules. Websockets should contain asyncio module, so these two modules are installed just by websocked install command.
If you do not have them installed, it can be easily done by these commands:
- On Fedora 24+ you can install it using: dnf install python3 python3-requests python3-websockets
- On other systems you can use prepared script install-python3.5.sh
Try to run the CLI. E.g.: ./da-cli.py lookup org.hibernate:hibernate-core:4.2.21.Final

Options description

There is a couple of options, which can define the behavior of the CLI tool. The more complex options are described in this section. If you want to see description of all the available options you can find it in the help of the tool.

The output of the tool is delimited by the tab, but in this section the data are displayed in the list, which strictly conforms to the order of the output

No parameters
- The tool will print the help with a basic description of options
Lookup built artifacts (lookup)
- Lookups all built versions of the requested GA and reports if there is any exact match to the requested GAV
- Output:
  - Requested GAV
  - RH version of the artifact or "None" if the artifact was not found
  - blacklisted/greylisted/whitelisted
  - All built versions of the requested GA
- Usage: ./da-cli.py lookup [GROUP_ID:ARTIFACT_ID[:PACKAGING[:CLASSIFIER]]:VERSION[:SCOPE]] [--products PRODUCTS] [--productIDs IDS]
  When GROUP_ID:ARTIFACT_ID:VERSION is specified finds corresponding redhat versions for it. When it is not specified, reads G:A:Vs from standard input and finds corresponding redhat versions for all of them. Packaging, classifier and scope is ignored in both cases. Consider artifact only from products specified by IDs ([--productIDs IDS]) or names ([--products PRODUCTS]).
Dependency report of a SCM repository (scm-report)
- Creates a dependency report for a single SCM repository (Currently only Git repositories are supported). This option provides the same information as the report option. The dependency report is expected to start from some module/POM in the project and analyses the POM dependencies.
- Output: The same as in the option report.
- Usage: ./da-cli.py scm-report [--raw|--json] scm tag pom-path [--products PRODUCTS] [--productIDs IDS]
  - scm - URL to the remote SCM repository
  - tag - Tag, branch or revision of the SCM
  - pom-path - Path to the POM in the repository, which will be the starting POM for the analysis
  - Consider artifact only from products specified by IDs ([--productIDs IDS]) or names ([--products PRODUCTS])
Advanced dependency report of a SCM repository (scm-report-advanced)
- This option extends the standard SCM dependency report with the some summarizing information related directly to the analysed project.
  The new information are: which project GAVs are blocklisted, which GAVs are already built / not built / built in the different versions.
- Output:
  - Blocklisted artifacts in a project
  - Built community artifacts, which were built internally
  - Community artifacts with other built version - artifacts, which wasn't built, but there are builds of the same GA
  - Community artifacts - artifacts, which weren't been built
  - The same dependency tree as in the regular dependency report
  All the summarizing informations are only for the top-level dependencies (dependencies in projects' POMs) of the project's modules.
- Usage: ./da-cli.py scm-report-advanced [--json] scm tag pom-path [--products PRODUCTS] [--productIDs IDS]
  - scm - URL to the remote SCM repository
  - tag - Tag, branch or revision of the SCM
  - pom-path - Path to the POM in the repository, which will be the starting POM for the analysis
  - Consider artifact only from products specified by IDs ([--productIDs IDS]) or names ([--products PRODUCTS])
Align report
- This option checks toplevel dependencies of all modules from git-scm link and prints sumarized information.
- Output:
```
    Internaly built:
    MODULE
        DEPENDENCY
            BUILT_VERSION (PRODUCT)

    Built in different version:
    MODULE
        DEPENDENCY
            BUILT_VERSION (PRODUCT)

    Not built:
    MODULE
        DEPENDENCY

    Blacklisted:
    MODULE
        DEPENDENCY
                        
```
- Usage ./da-cli.py align-report [--json] [--unknown] [--productIDs IDS]... [--repository REPOSITORY]... SCM TAG [POM-PATH]
  - --json Output unparsed response from Dependency Analyzer.
  - --unknown Consider artifacts from unknown products.
  - --productIDs IDS Consider artifact only from specified product IDs. IDS should be comma separated.
  - --repository REPOSITORY Aditional maven repositories required by the analysed project. You can specify this option multiple times.
Difference of products
- Returns list of different versions of artifacts between two products given by their ID
- Output:
  - List of different versions
- Usage ./da-cli.py difference [--json] leftID rightID
  - --json Output unparsed response from Dependency Analyzer.
  - leftID ID of first product
  - rightID ID of second product
Administration tools
- These tools add, remove or change data. These are administrative tools, so you need to be authenticated to use them. You can authenticate with your credentials when you run admin tool. You are asked to enter your login. System user name is used as default. After you are asked to enter password. In case credentials are wrong, CLI tool exits with failure message with no data changed.

Examples

List all artifacts in the whitelist: ./da-cli.py list white
Add a product to whitelist: ./da-cli-admin.py add whitelist-product eap:7.0.0 SUPPORTED
Add an artifact to whitelist: ./da-cli-admin.py add white org.jboss.hibernate:hibernate-core:3.4.2.redhat-1 eap:7.0.0
Add several artifact to whitelist by reading G:A:Vs from standard input: ./da-cli-admin.py add white eap:7.0.0 < gavfile.text
Remove an artifact from blocklist: ./da-cli-admin.py delete black org.jboss.hibernate:hibernate-core:3.4.2
Check all dependencies from pom and print their black/white list status: ./da-cli.py pom-bw <PATH_TO_DIRECTORY>
Check all dependencies from pom and print their report status: ./da-cli.py pom-report <PATH_TO_DIRECTORY>
Lookup built versions of an artifact: ./da-cli.py lookup org.hibernate:hibernate-core:4.2.18.Final
Create dependency report of an artifact: ./da-cli.py report org.hibernate:hibernate-core:4.2.18.Final
Create SCM report for a project module: ./da-cli.py scm-report https://github.com/jboss-fuse/fuse fuseenterprise-6.2.1.redhat-084 ./esb/pom.xml