Last week, we automated PR review in Azure DevOps with a GPT plugin
In continuation of last week's article, we will test the plugin to check its reliability.
Add sample code:
To test the plugin - we will take the following code and intentionally make some bad practices
Note the comments starting with BAD, we have added about 8 bad practice
Let's see how many will be detected by the plugin
romantointeger.java
//BAD: unoptimized import
import java.util.*;
//BAD: class naming convention
public class romantointeger {
public static int romanToint(String input) {
int result = 0;
//BAD: code to implementation instead of interface
//BAD: unused variable
ArrayList<String> names = new ArrayList<String>();
//BAD: naming convention
List<RomanNumeral> RomanList = RomanNumeral.getromanList();
int i = 0;
//BAD: missing null check
while (i < RomanList.size()) {
RomanNumeral roman = RomanList.get(i);
//BAD: usage of == instead of equals
if (roman.toString() == "V") {
}
if (input.startsWith(roman.name())) {
result += roman.getValue();
input = input.substring(roman.name().length());
} else {
i++;
}
}
return result;
}
}
RomanNumeral.java
import java.util.Arrays;
import java.util.Comparator;
import java.util.List;
import java.util.stream.Collectors;
public enum RomanNumeral {
I(1), IV(4), V(5), IX(9), X(10), XL(40), L(50), XC(90), C(100);
RomanNumeral(int value) {
this.value = value;
}
private int value;
public int getValue() {
return value;
}
public static List<RomanNumeral> getromanList() {
return Arrays.stream(values()).sorted(Comparator.comparing((RomanNumeral x) -> x.value).reversed())
.collect(Collectors.toList());
}
}Test the plugin:
We will create a PR so that the plugin will get triggered
remember that we had configured the plugin on PR towards the dev branch
create a pr from the feature branch to the dev branch
Let's wait for the PR build to complete and then examine the PR comments
Nice! The plugin has detected almost all of the issues we have injected intentionally
Some more tests:
We will add even more bad practices to the existing code to see if the plugin flags that
Let's add two more bad methods
The romantointeger class will look like the below now
romantointeger.java
//BAD: unoptimized import
import java.util.*;
// BAD: class naming convention
public class romantointeger {
public static int romanToint(String input) {
int result = 0;
// BAD: code to implementation instead of interface
// BAD: unused variable
ArrayList<String> names = new ArrayList<String>();
// BAD: naming convention
List<RomanNumeral> RomanList = RomanNumeral.getromanList();
int i = 0;
//BAD: missing null check
while (i < RomanList.size()) {
RomanNumeral roman = RomanList.get(i);
//BAD: usage of == instead of equals
if (roman.toString() == "V") {
}
if (input.startsWith(roman.name())) {
result += roman.getValue();
input = input.substring(roman.name().length());
} else {
i++;
}
}
return result;
}
public void badPractice() {
try {
} catch (Exception e) {
//BAD: we will not do anything here to see if GPT finds this issue
}
//BAD: Instead of reporting error , simply exiting
finally {
//BAD: we will not do anything here as well to see if GPT finds this issue
}
System.exit(0);
}
//BAD: complex way of a simple XOR
boolean negateValue( boolean shouldNegate, boolean value ) {
return (shouldNegate?(!value):value);
}
}
The PR got updated so the plugin will be trigerred again
wow! again the plugin detected all the issues we injected intentionally
Accuracy and reliability:
The plugin has provided reliable results so far. All the review comments provided are legit and worth considering. The comments had a clear explanation of where is the issue and why its an issue.
Security:
As the AI models are still evolving, as of now only low-risk projects are best suited for this use case
if you aren't comfortable using OpenAI directly - Azure OpenAI can be used, this provided the capabilities of OpenAI topped up with the security capabilities of Azure
At this time, the Azure OpenAI model is open only for specific cases - please see the below snippet from the Microsoft site
"For now, we're working with customers with an existing partnership with Microsoft, lower-risk use cases, and those committed to incorporating mitigations."
Summary:
As a preliminary reviewer, the plugin did really well.
The plugin not just flagged the issue but also provides suggestions to rectify the issues.
Thanks for reading, Goodbye until next week!