As a proud and lazy developer, it has been more than a year since I've been guiltily using GitHub Copilot. While it has made me a 10x developer, it has also brought down my creativity to 10%. Guilt aside, it has helped me learn new things much faster than I would've learned from a Udemy course. GitHub has recently announced an updated version of Copilot named GitHub Copilot X which has not been officially released yet as of today. Copilot X will be powered by GPT-4 and is coming to our favourite code editors as our personal coding friend that will not only generate code for us but also debug our code, scan for issues, generate documentation, and, most importantly, talk to us when we feel lonely.
GitHub has recently faced some lawsuits. The reason is, Copilot was trained using public code on GitHub and GitHub is charging us to use and further train it so that it will ultimately be able to replace all of us. There might be other legal reasons as well but that's not the topic for today. The bigger news is, Amazon has recently released its answer to GitHub Copilot named Amazon CodeWhisperer. While there are other AI code generation tools as well such as Tabnine, CodeT5 etc., but a tool similar to Copilot released by a competitor tech giant must get proper attention. That is why, in this article today, I'm going to test Copilot and CodeWhisperer and tell you which one outshines the other.
Both the Copilot and the CodeWhisperer are available to use as extensions in popular code editors and IDEs. CodeWhisperer currently supports only VS Code and JetBrains IDEs while Copilot additionally supports VS, Neovim etc. I am testing both the tools in VS Code with a simple Vue.js project and comparing them in different aspects.
Speed
Both the tools are meant to increase our productivity, so obviously speed matters. I prompted both the tools to create two routes for login and register in Vue router with a simple comment and both the tools generated the same snippets taking reference from existing routes.
While testing I found that CodeWhisperer currently doesn't support many languages or frameworks. I tested with .vue, .html and it didn't work. So, I tested with plain JavaScript code.
// Routes for login and register
{
path: '/login',
name: 'login',
component: () => import('../views/LoginView.vue')
},
{
path: '/register',
name: 'register',
component: () => import('../views/RegisterView.vue')
}In my case, Copilot generated the code slightly faster. Also, Copilot suggested all the code in the snippet at once, but CodeWhisperer suggested a single line at once. This is both good and bad at the same time. While Copilot sometimes gives a large code snippet of unnecessary code, CodeWhisperer suggests a single line at once which helps to prevent unnecessary code. But the large snippet generated by Copilot is not always unncessary and when it generates the right code, it can save a lot of extra Tab hits and time.
Accuracy
An accurately generated code can save us a lot of time. Sometimes if the tools give incorrect code, it might take much more time to debug that code than to write that code ourselves. Therefore, I tested the tools if they can clearly understand a long instruction and generate the code.
I gave a comment prompt: Add an event listener for DOM content load and inside that, add hover effect on a button with id 'delete' and set background color to green and border to 2px solid red to both of them.
With CodeWhisperer, it generated irrelevant code for a few times. But when I repeatedly kept deleting and re-generating the code, it finally generated the code on my fifth attempt with minor error while closing the braces.
document.addEventListener("DOMContentLoaded", function () {
let button = document.getElementById("delete");
button.addEventListener("mouseover", function () {
button.style.backgroundColor = "green";
button.style.border = "2px solid red";
});
}When I manually triggered it to generate the remaining code, it successfully generated the closing brace and semicolon.
With Copilot, it generated a working code on my first attempt without any errors. But the code looked a bit inefficient as it was getting the button element each time for styling. This time, Copilot didn't generate the whole code at once, instead it took me three Tab hits to generate the whole code; but I didn't trigger it manually.
document.addEventListener('DOMContentLoaded', () => {
document.getElementById('delete').addEventListener('mouseover', () => {
document.getElementById('delete').style.backgroundColor = 'green';
document.getElementById('delete').style.border = '2px solid red';
});
});On another note, for me CodeWhisperer on VS Code looked unstable. Sometimes, it didn't generate the code automatically and I had to manually trigger it and sometimes it didn't show the loading icon at the bottom while generating. That being said, Copilot is almost two years older than CodeWhisperer and this might not be the right time to compare their integration with code editors. This might be due to some issues with VS Code extension.
Uniqueness
Both the tools were developed to do the same work and both of them do it pretty well. But CodeWhisperer offers two unique features that Copilot doesn't currently have.
One of them is the code references. The code references log shows where the code was taken from when we accept a suggestion from CodeWhisperer. To test this feature, I prompted it to create a function for API call using Axios that fetches all users. It gave me the code and when I accepted it, it showed the reference on the log terminal.
Amazon's decision to enhance the transparency of their training data is a good step. It will assist in preventing potential copyright infringement issues with third-party code.
Another feature that CodeWhisperer offers is security scans. The security scan feature scans the files and dependencies for any possible security issues and shows them. To test it, I wrote a very simple JS function that demonstrates a simple XSS attack. The following JavaScript function directly appends the given param string to DOM without any input sanitation which opens the door for an XSS (Cross-Site Scripting) attack.
function show(str){
const p = document.createElement("p");
p.innerHTML = str;
document.getElementsByTagName('body')[0].appendChild(p);
}I attempted a security scan with CodeWhisperer with this code and I was amazed to see that it successfully detected the issue, highlighted the line in the code and provided a short description of the issue.
I don't know how it performs in a large project scenario, but it did work for a simple function.
Cost
As of now, CodeWhisperer is free to use but Copilot costs 10$/month or 100$/year. This is where CodeWhisperer undoubtedly shines. But Copilot was also free for a few months as technical preview and there is a chance that Amazon is also planning to do the same.
To conclude, I think CodeWhisperer is really good considering its age. The unique features it offers will surely stand out with improvements. Both the tools have a lot of room to improve. No doubt that CodeWhsiperer will improve a lot with time and so will Copilot. The updated Copilot X from GitHub looks very promising, and its tight integration with editors and GitHub will surely change the way we code. Whatever happens, I hope I will be there to experience. Most importantly, I hope that CodeWhisperer remains free forever so that I will at least have my money when it finally takes away my job.
