To begin setting up test coverage reporting with Code Climate, jump to our
Quick Guide
below. For those running tests in parallel, head to our section on for
parallel tests
.
Supported Languages & Formats
Quick Guide
Parallel Tests
Multiple Test Suites
Working with a Docker container
Running tests within a subdirectory
Locations of Pre-built Binaries
List of subcommands
CI Examples
Troubleshooting and Getting Help
In your CI's config file or CI's in-app config options:
Set your repo's test reporter ID as an environment variable to identify your repo. This can be found on your
Repo Settings
page in Code Climate.
Fetch the
test reporter pre-built binary
.
Run the
before-build
command before running your test suite to notify Code Climate of a pending report.
Execute your test suite.
Run the
after-build
command after your test suite completes to send a formatted report or notify Code Climate of a failing build.
For examples, click the link for your specific CI provider below:
Travis
CircleCI
Solano
Codeship
Semaphore
Jenkins
The Code Climate test reporter can join parallelized test reports, combining them into one individual test report that can be submitted to Code Climate.
Configure your CI to store partial results from each parallel run. We recommend syncing and fetching files from S3 as one approach. Alternately, you can use a temporary folder in your CI build. Then, use the test reporter's
format-coverage
,
sum-coverage
, and
upload-coverage
commands to combine the results and upload them as one complete test report.
Set your repo's
test reporter ID
as an environment variable to identify your repo. This can be found on your
Repo Settings
in Code Climate.
Fetch the
test reporter pre-built binary
.
Run
before-build
before running your test suite to notify of a pending report.
Execute your test suite.
Run
format-coverage
on the reports from each parallelized instance.
Run
sum-coverage
to combine the parallelized test reports into one unified report.
Run
upload-coverage
to upload the combined test report to Code Climate.
For examples, click the link for your specific CI provider below:
CircleCI
See the
README on GitHub
for detailed instructions.
The Code Climate test reporter can join multiple test reports together from separate test suite processes.
Configure your CI to temporarily store the reports from each separate test suite. We recommend syncing and fetching files from S3 as one approach. Alternately, you can use a temporary folder in your CI build. Then, use the test reporter's
format-coverage
,
sum-coverage
, and
upload-coverage
commands to combine the results and upload them as one complete test report.
Set your repo's
test reporter ID
as an environment variable to identify your repo. This can be found on your
Repo Settings
page in Code Climate.
Fetch the
test reporter pre-built binary
.
Run
before-build
before running your test suite to notify of a pending report.
Execute your test suite.
Run
format-coverage
on the reports from each individual test suite.
Run
sum-coverage
to combine multiple test reports into one unified report.
Run
upload-coverage
to upload the combined test report to Code Climate.
Note: the
after-build
command is not used with parallel or multi-suite tests.
For examples, click the link for your specific CI provider below:
CircleCI
Travis CI
You may need to specify a
prefix
value if your tests are run outside of your project's root directory.
For example, if you're running tests within a
docker container
and your app code is located at
/usr/src/app
, you'd use:
cc-test-reporter format-coverage --prefix /usr/src/app
The test report is distributed as a pre-built binary named
cc-test-reporter
. You can fetch the pre-built binary from the following URLs:
Linux
https://codeclimate.com/downloads/test-reporter/test-reporter-latest-linux-amd64
https://codeclimate.com/downloads/test-reporter/test-reporter-0.11.1-linux-amd64
Linux ARM64
https://codeclimate.com/downloads/test-reporter/test-reporter-latest-linux-arm64
https://codeclimate.com/downloads/test-reporter/test-reporter-0.11.1-linux-arm64
Linux netcgo
(
recommended
if you're using a VPN)
https://codeclimate.com/downloads/test-reporter/test-reporter-latest-netcgo-linux-amd64
https://codeclimate.com/downloads/test-reporter/test-reporter-0.11.1-netcgo-linux-amd64
https://codeclimate.com/downloads/test-reporter/test-reporter-latest-darwin-amd64
https://codeclimate.com/downloads/test-reporter/test-reporter-0.11.1-darwin-amd64
Windows
https://codeclimate.com/downloads/test-reporter/test-reporter-latest-windows-amd64
https://codeclimate.com/downloads/test-reporter/test-reporter-0.11.1-darwin-amd64
Along with the binaries you can download a file with a SHA 256 checksum for the given version and a GPG detached signature. Check
this docs
for further details.
📘
Running test-reporter on a Linux host using a VPN
If you're running the test-reporter on a Linux host that uses a VPN, use the
netcgo-linux
binary. More details
here
.
📘
If you'd like to pin to a specific version of the test reporter, you should refer to the
releases on GitHub
.
format-coverage
- formats test report from local test suite into generalized format, readable by Code Climate. Supports the following flags:
-t
or
--input-type
simplecov | lcov | coverage.py | gcov | clover
- Identifies the input type (format) of the COVERAGE_FILE
-o PATH
or
--output PATH
- Output to PATH. If - is given, content will be written to stdout. Defaults to coverage/codeclimate.json.
-p PATH
or
--prefix PATH
- The prefix to remove from absolute paths in coverage payloads, to make them relative to the project root. This is usually the directory in which the tests were run. Defaults to current working directory.
COVERAGE_FILE
- Path to the coverage file to process. Defaults to searching known paths where coverage files could exist and selecting the first one found.
sum-coverage
- combines test reports from multiple sources (i.e. multiple test suites or parallelized CI builds) into one test report which is readable by Code Climate. Supports the following flags:
-o PATH
or
--output PATH
- Output to PATH. If - is given, content will be written to stdout. Defaults to coverage/codeclimate.json.
-p NUMBER
or
--parts NUMBER
- Expect NUMBER payloads to sum. If this many arguments are not present, command will error. This ensures you don't accidentally sum incomplete results.
upload-coverage
- uploads formatted, singular test report to Code Climate API. Supports the following flags:
-i PATH
or
--input PATH
- Read payload from PATH. If - is given, the payload will be read from stdin. Defaults to coverage/codeclimate.json.
-r ID
or
--id ID
- The reporter identifier to use when reporting coverage information. The appropriate value can be found in your Repository Settings page on codeclimate.com. Defaults to the value in the
CC_TEST_REPORTER_ID
environment variable. The uploader will error if a value is not found.
-e URL
or
--endpoint URL
-
For use only with Code Climate: Enterprise
. The endpoint to upload coverage information to. Defaults to the value in the CC_TEST_REPORTER_COVERAGE_ENDPOINT environment variable, or a hard-coded default (currently "
https://api.codeclimate.com/v1/test_reports"
).
after-build
- combines
format-coverage
and
upload-coverage
. Supports the following flags:
--exit-code $EXIT_CODE
-
$EXIT_CODE
should be the exit code of your test suite process. Some CI system expose this as an environment variable; for others, you may need to manually capture
$?
to provide it to
after-build
later. Providing this will prevent sending test coverage results for failed tests.
📘
We support any CI provider
Our test reporter can run on any CI provider, even if you don't see a specific example here. If you'd like help setting up the test reporter on a different CI provider, feel free to
send us a message
or open an issue
on GitHub
.
Below, you'll find an examples of how to configure our test reporter with some common CI providers.
Travis CI
Solano CI
Circle CI
Codeship CI
Jenkins
GitHub Actions
GitHub Actions with XCode
More Examples
Test Coverage Troubleshooting Tips
Troubleshooting branch names
My diff-coverage and total-coverage statuses are hanging on my PR
Working with a docker container?
📘
Apex
If you're using Apex, you will be able to see coverage information by using the tool
apexcov
before using the test reporter.
For example, you can use it like this:
- apexcov
- ./cc-test-reporter format-coverage -t lcov ./coverage/lcov.info
- ./cc-test-reporter upload-coverage
For more details and examples, go to the
apexcov documentation
.
📘
Jacoco
Most often users see this message with Java projects when the path to source code can't be inferred:
Error: could not find any viable formatter. available formatters: simplecov, lcov, coverage.py, clover, gocov, gcov, cobertura, jacoco
For these cases, instead of
after-build
, use:
format-coverage
, which includes:
JACOCO_SOURCE_PATH
: the path to Java source files (a new environment variable)
coverage file
: to path to JaCoCo coverage XML file (the first argument)
upload-coverage
Example and more details in
this conversation
.
When setting
JACOCO_SOURCE_PATH
some users may receive the following error:
could not find coverage info for source files
. To move forward, remove the
group
tag from your
jacoco.xml
. The
group
tag is not currently supported.
📘
Jacoco
group
tag
When using a group tag in your jacoco.xml, you might need to adjust your source path like this:
(thanks to
https://github.com/seancarroll/jfluentvalidation/commit/9a9e7c97049a5db3441f9811698f8d9a1a72a17e
)
📘
One way to add needed information to your AWS CodeBuild build via environment variables for the Code Climate test reporter:
Add the following command to the install or pre_build phase of your buildspec:
Text
