Step aside developers, an AI is about to take your spot! Not really, but this is what the news say.
First, being developers ourselves, we at rockin.ai love mahcines and artificial intelligence. We love the technology, the breakthroughs and the challenges. One such breakthrough recently was ChatGPT's ability to write code.
We went ahead and did some testing, and will report our findings here. Overall, if we were giving it a grade it would be a C-, which is both disappointing and respectable at the same time. It is respectable because we realize this is only the start, and next versions will no doubt improve!
If you are a developer, you should know that OpenAI has opened up a waitlist for plugin developers.
What Is ChatGPT Developer Mode?
One of the most hyped capabilities of ChatGPT is its coding skills. Simply asking it to write a piece of code (in any language), will usually result in code being produced. So, enabling developer mode is simply asking "please write code that solves <insert problem here>". Let's have a closer look at the code ChatGPT writes, and see how good it really is.
Asking ChatGPT For A Simple Function
In the below screenshot, we're asking ChatGPT to write a function to count all 1s in a binary integer. We have to say it is ironic that we have machine producing human-readable code...
While the result actually works, it is also very slow. Maybe this is even the slowest possible implementation. The proposed code from ChatGPT will first convert the integer to a binary string, and then search it for substrings matching '1'.
Asking ChatGPT For An Optimized Function
In an effort to improve the situation, we will ask the same question like before, but ask ChatGPT to optimize the code. We were impressed that ChatGPT immediately realized that using bitwise will make sense.
Below is the code from ChatGPT so you can also try it out. While it looks like an improvement, ChatGPT forgot that integers can be negative. Also, right-shifting negative numbers in PHP will not shift the sign bit, and therefore this becomes an infinite loop! But it was still a nice try.
function countOnes($integer) {
//Irony! Machine writing human code.
$count = 0;
while ($integer) {
// Increment count if the last bit is a one
$count += $integer & 1;
// Right shift to move to the next bit
$integer >>= 1;
}
return $count;
}
Writing A Real Solution By Ourselves
The real implementation would look more like the one below, where we have to left shift the integer instead (which does shift the sign bit!). Also, we have to remember that some machines have 32 bit integers, and others have 64 bits. The below code is by no means optimal - but at least it is not buggy anymore, and works decently fast. To achieve even faster performance, we can unwrap the loop, and look to hardcode the bitwise operations, probably achieving a near constant runtime - but then the code would barely be readable at all to other developers.
function countOnes($integer) {
//Irony! Human writing Machine Code.
$count = 0;
while ($integer !== 0) {
// Increment count if the last bit is a one
if(($integer & PHP_INT_MIN) !== 0) {
$count++;
}
// Left shift to move to the next bit
$integer <<= 1;
}
return $count;
}
Would We Recommend To Use ChatGPT Code In Production?
This depends on the context of the application. If the application has to run perfectly, be decently optimized and secure - then most probably not. At best, the AI might be used as a "co-pilot" and not more than just a code suggestion tool. Its results are easily buggy or slow or both.
On the other hand, if the application does not need to be perfect or fast - and is a throw-away app anyway, then definitely yes! The solution proposed by ChatGPT at the very top was working just fine for such a use case.
We'll say for sure, ChatGPT is much better at cleaning data than writing code! For example, we have a separate tutorial about creating a fine-tuned model to fix email addresses.
How Does ChatGPT Know What Code To Write?
ChatGPT is an LLM under the hood, and it writes code just like it writes essays. It will look for common patterns and try recommending the next most probably word or "token". The model was likely fed millions of StackOverflow pages to process through, and that is how it is able to come up with logical results. That last part is only a theory by the way, as only the OpenAI team knows how the model was actually trained.
Conclusion
From our tests, we have to say two things: the results of ChatGPT developing code are poor when comparing to even a bad developer. But! The results are also extremely promising and exciting!
Just think about it, if this is only the first iteration of the solution, then what will the next years bring? All comments are welcome below.
It may interest you to connect to the ChatGPT API with a simple client, and do some of these tests in code.
