Pull Request Review with OpenAI GPT

In Azure DevOps, Pull requests provide a way for reviewing the code before merging the changes to a branch. Reviewers need to have a keen eye and good programming skills to be able to provide constructive feedback. The review comments are subjective and each reviewer will have their own style. while these reviews can never be fully automated,integrating an automated PR review in the process before manual review can reduce the load on reviewers

Let's see how GPT can be used to do PR reviews

Install the extension:

To install the GPT extension

click on "Organization Settings" and "Extensions"

click on "Browse marketplace"

Search for PR reviewer extension

click on the "Get it free" button on the extension page

then click install to finish the installation process

Configure the permissions:

The extension requires certain permissions to be able to access the pull request

From the Repos drop-down, select "Manage repositories"

Navigate to the security tab

Allow the build service to "Contribute to pull requests"

Modify the pipeline:

For demonstration, let's use a simple Java Maven build pipeline which looks like the one below

trigger:
- dev

pool:
  vmImage: ubuntu-latest

steps:
- task: Maven@3
  inputs:
    mavenPomFile: 'pom.xml'
    mavenOptions: '-Xmx3072m'
    javaHomeOption: 'JDKVersion'
    jdkVersionOption: '1.8'
    jdkArchitectureOption: 'x64'
    publishJUnitResults: true
    testResultsFiles: '**/surefire-reports/TEST-*.xml'
    goals: 'package'

Before adding the GPT task we need to

• add a checkout step with persistCredentials enabled
• Generate an API key from openai api https://platform.openai.com/account/api-keys

The pipeline will look like the below

pr:
  branches:
    include:
    - dev

pool:
  vmImage: ubuntu-latest

steps:
- checkout: self
  persistCredentials: true

- task: Maven@3
  inputs:
    mavenPomFile: 'pom.xml'
    mavenOptions: '-Xmx3072m'
    javaHomeOption: 'JDKVersion'
    jdkVersionOption: '1.8'
    jdkArchitectureOption: 'x64'
    publishJUnitResults: true
    testResultsFiles: '**/surefire-reports/TEST-*.xml'
    goals: 'package'

- task: GPTPRReviewer@0
  inputs:
    api_key: 'blahblahblah'
    api_key_source: 'openai'
    aoai_endpoint: 
    comment_language: 'en-US'

Enable PR Trigger:

The GPT task is designed to be run only when the build is triggered from a pull request

So we will make a few changes to the branch policies

from "Project settings", navigate to "Repositories" and select the required repository, then click on the policy tab and under Branch policies click on the specific branch (dev in this case)

Under the "Build Validation" section enable "Build must succeed in order to complete pull requests."

Create a PR:

I have added the following method with an inefficient constructor to check if the GPT reviewer will find the issue

public void sayHello(String world) {
	
	String message = new String("hello ");

	message = new String(message + world);

}

created a PR and waited for the PR pipeline to complete

Navigated to the specific PR.

Hurray! GPT has detected the inefficient constructor and posted the suggestion as a review comment.

if you are using a free version of OpenAI , you might run out of credits.

The task will no longer work after the usage limit is exhausted.

https://platform.openai.com/account/usage can be used to check the usage

Summary:

Code review is one of the most complex parts of the SDLC to be automated. Additionally, there should be serious security and privacy considerations if an AI reviewer has to be implemented at an organizational level. However, I see that at least open-source projects can readily benefit a lot from using such reviewers.

Thanks for reading, Goodbye until next week!