Automated Checks

A Pattern that supports Code Review by helping Developers focus on the most important things

When using Code Reviews and you want to reduce the overhead for developers so that they can give better feedback. You want to automate parts of your Codeline Policy. This pattern describes how to simplify developer workflows related to reviews and codeline management while making it easier to maintain quality.

Varying Strengths and Levels of Attention

Developers have limited time and energy. Focusing on small details can obscure attention to more significant issues. Some mundane things are worth checking thoroughly.

Even the best developers make mistakes. There are multiple classes of errors that a code review could identify:

• Mechanical, such as

  • Syntax Errors

  • Style (indentation etc)

  • Vulnerabilities (code and dependency)

• Topical, such as:

  • Is the code using the correct design abstractions?

  • Does the code do what it’s supposed to do?

Humans can catch many of the mechanical issues, but identifying them often depends on a mix of highly detailed syntactic analysis (syntax, style) and encyclopedic knowledge (vulnerabilities). Even with the best intentions, this kind of analysis is error-prone.

Tools can identify higher-level issues like functions or blocks with excess complexity. In some cases, the complexity might be justifiable, so you don’t always want a tool to reject code automatically. A tool can detect if a change has associated added tests, but it can’t detect if the test logic is correct.

Automated tools can provide feedback but can’t offer the same nuanced feedback as team members who have context.

Code that doesn’t meet minimal standards isn’t ready for a final review, and while feedback on work in progress can be helpful, you want to help make effective use of developer time while still fostering an environment of collaboration.

Automate Rote Work; Allow for Evaluation

Add Automated Checks to your build pipeline to address issues such as style, formatting, complexity, and vulnerability issues. These checks should be able to run both in a Developer Workspace and in a CI Environment. Evaluate the results of these checks in the context of Code Review. Include Automated Checks that run with each commit to a Task Branch, and make that feedback visible as part of a Pull Request

Automated Checks can include everything from compilation to a style checker and even automatically reformatting code. Unit Tests in the context of an Integration Build can also be considered part of the Automated Checks.

Other checks can:

• Identify Vulnerabilities using dependency Analysis

• Static analysis to identify likely sources of error

• Code Analysis to identify the risk level for a pull request. ( for example, Shepherdly)

• Flag changes in test coverage.

• Format code to standards.

• Flag awkward Variable Naming (although some standards can be enforced, readability based on name choice, for example, is more subjective)

Run these checks and unit tests during development in a developers workspace and when code is merged to a Task Branch to identify obvious issues. It is ideal if the tool can flag potential problems so that the reviewer can see them in context.

Ideally, automated checks that identify critical issues (compilation, test execution, formatting etc) should pass before sharing the code for review. Other checks should run before the review so that the developer and reviewer can discuss how to address them (or if). 

Based on the kinds of Automated Checks you could evaluate the risk of a change to determine the amount of review a change should get. A small, low-risk change might not need any formal review step, though feedback is always helpful for improving understanding and quality,

Cautions

While automated checks offload work from the human reviewers, they may not capture every issue, so incorporate feedback about their effectiveness into your Retrospective process.

Getting to Automated Checks

The easiest way to add automated checks is to integrate these tools into your build pipeline. Tools can be added individually, or you can use platforms that combine tools and integrate the results with your review platform, such as Pull Requests.

Adding the ability to run tests from your build tool will enable developers to run these tests while coding. Add them to the CI pipeline to ensure that they are run, especially if the checks take a non-trivial amount of time.

Next Steps

• Unit Tests are a kind of Automated Check in that tests passing should be a prerequisite for a final review.