OpenAI has developed a new way to create AI models (called "fine-tunes" or "GPTs"), which is much easier than creating models from scratch with PyTorch or TensorFlow. Fine-tuning works by providing us with a complete GPT LLM as the starting point, which by itself is extraordinarily difficult to replicate, and then allowing us to train it on a dataset. The best fine-tuned GPT models tend to work with natural language or oriented around generating content.
In this tutorial, we will go though an example, covering all of the steps and details of creating a fine-tuned model. The task for the model will be to fix mistyped or misspelled email addresses. While it is easy to recognize a misformatted email address with plain regex, it is not so easy to suggest a fix, and we will harness the full power of GPT to help.
All of the code and training data are shared below, so you can reproduce the results. At the end, we will also test the model in different ways to validate that it actually learned from our training data.
As a sneak peek, this model achieved a 90% accuracy rate of finding mistyped email addresses, and also fixing them.
How Do I Fine-Tune My GPT Model?
The steps to fine-tuning a GPT model are the following:
- Create an OpenAI account and get an API key (steps covered separately)
- Develop a training data set, which will be used to train an existing OpenAI GPT model
- Create a "fine-tune" (or GPT), based on the training data, with specific OpenAI API calls
- Validate the newly created model against a second validation data set
After validating your model, you will know how good it is at the specific task.
In case a model is performing poorly, we should remember two points. First, we are training a model that is an LLM under the hood. An LLM will probably not be good at playing chess - but it will excel at any task involving language or text.
Second, the training data set might be completely wrong or imbalanced. Actually, having a balanced data set is critical in achieving good results! This means having an equal amount of cases for each possible outcome we want.
OpenAI Fine-Tune Simple API Client
Below is a simple API client written in PHP, which has all of the functions necessary to upload our training data, create a fine tuning model, and also to run prompts on the newly created model.
By default it will train GPT-3.5-turbo, however the list of possible models to train is constantly evolving with the newest models such as GPT-4 and more.
<?php
class OpenAIFineTuneSimpleClient {
private static $open_ai_key = '<your openai api key>';
private static $open_ai_url = 'https://api.openai.com/v1'; //current version of the API endpoint
/**
* Uploads a file to OpenAI (normally a training file for fine tuning an existing model).
*/
public static function uploadFile($filePath, $purpose='fine-tune') {
$message = [];
$message['file'] = new CURLFile($filePath);
$message['purpose'] = $purpose;
$result = self::_sendMessage('/files', $message, 'post', 'multipart/form-data');
return $result;
}
/**
* Create a new fine-tuned model, based on an existing model, using an uploaded training file.
*/
public static function createFineTune($trainingFile, $model = 'gpt-3.5-turbo-1106', $validationFile='') {
$message = new stdClass();
$message -> training_file = $trainingFile;
$message -> model = $model;
if($validationFile) {
$message -> validation_file = $validationFile;
}
$result = self::_sendMessage('/fine_tuning/jobs', json_encode($message));
return $result;
}
/**
* Get a list of all existing fine-tunes for my account.
*/
public static function getFineTunes() {
$result = self::_sendMessage('/fine_tuning/jobs', '', 'get');
return $result;
}
/**
* Doc: https://platform.openai.com/docs/api-reference/chat/create
* @param array $messages (each item must have "role" and "content" elements, for )
* @param int $maxTokens maximum tokens for the response in ChatGPT (1000 is the limit for gpt-3.5-turbo)
* @param string $model valid options are "gpt-3.5-turbo", "gpt-4", and in the future probably "gpt-5"
* @param int $responseVariants how many response to come up with (normally we just want one)
* @param float $frequencyPenalty between -2.0 and 2.0, penalize new tokens based on their existing frequency in the answer
* @param int $presencePenalty between -2.0 and 2.0. Positive values penalize new tokens based on whether they appear in the conversation so far, increasing the AI's chances to write on new topics.
* @param int $temperature default is 1, between 0 and 2, higher value makes the model more random in its discussion (going on tangents).
* @param string $user if you have distinct app users, you can send a user ID here, and OpenAI will look to prevent common abuses or attacks
*/
public static function chat($messages = [], $maxTokens=100, $model='gpt-3.5-turbo', $responseVariants=1, $frequencyPenalty=0, $presencePenalty=0, $temperature=1, $user='') {
//create message to post
$message = new stdClass();
$message -> messages = $messages;
$message -> model = $model;
$message -> n = $responseVariants;
$message -> frequency_penalty = $frequencyPenalty;
$message -> presence_penalty = $presencePenalty;
$message -> temperature = $temperature;
if($user) {
$message -> user = $user;
}
$result = self::_sendMessage('/chat/completions', json_encode($message));
return $result;
}
private static function _sendMessage($endpoint, $data = '', $method = 'post', $contentType = 'application/json') {
$apiEndpoint = self::$open_ai_url.$endpoint;
$curl = curl_init();
if($method == 'post') {
$params = array(
CURLOPT_URL => $apiEndpoint,
CURLOPT_SSL_VERIFYHOST => false,
CURLOPT_SSL_VERIFYPEER => false,
CURLOPT_RETURNTRANSFER => true,
CURLOPT_MAXREDIRS => 10,
CURLOPT_TIMEOUT => 90,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => "POST",
CURLOPT_NOBODY => false,
CURLOPT_HTTPHEADER => array(
"content-type: ".$contentType,
"accept: application/json",
"authorization: Bearer ".self::$open_ai_key
)
);
curl_setopt_array($curl, $params);
curl_setopt($curl, CURLOPT_POSTFIELDS, $data);
} else if($method == 'get') {
$params = array(
CURLOPT_URL => $apiEndpoint . ($data!=''?('?'.$data):''),
CURLOPT_SSL_VERIFYHOST => false,
CURLOPT_SSL_VERIFYPEER => false,
CURLOPT_RETURNTRANSFER => true,
CURLOPT_MAXREDIRS => 10,
CURLOPT_TIMEOUT => 90,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => "GET",
CURLOPT_NOBODY => false,
CURLOPT_HTTPHEADER => array(
"content-type: ".$contentType,
"accept: application/json",
"authorization: Bearer ".self::$open_ai_key
)
);
curl_setopt_array($curl, $params);
}
$response = curl_exec($curl);
curl_close($curl);
$data = json_decode($response, true);
if(!is_array($data)) return array();
return $data;
}
}
Our Training Data Set
We have come up with a list of 50 cases (all using fake email addresses).
25 of them are perfectly good email addresses and 25 are obviously bad in some way. The training data belongs in a file called "train.jsonl", and each line is a single data item.
In each case, we provide three points of information:
- Give the system a context for what it should be doing
- A sample user input
- A sample output based on the input
Please be aware that OpenAI has a policy moderating user inputs. So, it is important to filter out inappropriate texts before sending them to the OpenAI API.
{"messages": [{"role": "system", "content": "You help to ensure an email address is correct. Email address correction. Fix email format. Fix obvious email mistype."}, {"role": "user", "content": "hello.world@hotmail.com"}, {"role": "assistant", "content": "hello.world@hotmail.com"}]}
{"messages": [{"role": "system", "content": "You help to ensure an email address is correct. Email address correction. Fix email format. Fix obvious email mistype."}, {"role": "user", "content": "helloworld@hotmail.com"}, {"role": "assistant", "content": "helloworld@hotmail.com"}]}
{"messages": [{"role": "system", "content": "You help to ensure an email address is correct. Email address correction. Fix email format. Fix obvious email mistype."}, {"role": "user", "content": "johndoe@yahoo.ca"}, {"role": "assistant", "content": "johndoe@yahoo.ca"}]}
{"messages": [{"role": "system", "content": "You help to ensure an email address is correct. Email address correction. Fix email format. Fix obvious email mistype."}, {"role": "user", "content": "johny_999@gmail.com"}, {"role": "assistant", "content": "johny_999@gmail.com"}]}
{"messages": [{"role": "system", "content": "You help to ensure an email address is correct. Email address correction. Fix email format. Fix obvious email mistype."}, {"role": "user", "content": "t.w@car.com"}, {"role": "assistant", "content": "t.w@car.com"}]}
{"messages": [{"role": "system", "content": "You help to ensure an email address is correct. Email address correction. Fix email format. Fix obvious email mistype."}, {"role": "user", "content": "support@rockin.ai"}, {"role": "assistant", "content": "support@rockin.ai"}]}
{"messages": [{"role": "system", "content": "You help to ensure an email address is correct. Email address correction. Fix email format. Fix obvious email mistype."}, {"role": "user", "content": "yt@youtube.com"}, {"role": "assistant", "content": "yt@youtube.com"}]}
{"messages": [{"role": "system", "content": "You help to ensure an email address is correct. Email address correction. Fix email format. Fix obvious email mistype."}, {"role": "user", "content": "postmaster@yahoo.com"}, {"role": "assistant", "content": "postmaster@yahoo.com"}]}
{"messages": [{"role": "system", "content": "You help to ensure an email address is correct. Email address correction. Fix email format. Fix obvious email mistype."}, {"role": "user", "content": "press@facebook.com"}, {"role": "assistant", "content": "press@facebook.com"}]}
{"messages": [{"role": "system", "content": "You help to ensure an email address is correct. Email address correction. Fix email format. Fix obvious email mistype."}, {"role": "user", "content": "wiki@wikipedia.org"}, {"role": "assistant", "content": "wiki@wikipedia.org"}]}
{"messages": [{"role": "system", "content": "You help to ensure an email address is correct. Email address correction. Fix email format. Fix obvious email mistype."}, {"role": "user", "content": "mike@instagram.com"}, {"role": "assistant", "content": "mike@instagram.com"}]}
{"messages": [{"role": "system", "content": "You help to ensure an email address is correct. Email address correction. Fix email format. Fix obvious email mistype."}, {"role": "user", "content": "quote@one.com"}, {"role": "assistant", "content": "quote@one.com"}]}
{"messages": [{"role": "system", "content": "You help to ensure an email address is correct. Email address correction. Fix email format. Fix obvious email mistype."}, {"role": "user", "content": "mail@google.ca"}, {"role": "assistant", "content": "mail@google.ca"}]}
{"messages": [{"role": "system", "content": "You help to ensure an email address is correct. Email address correction. Fix email format. Fix obvious email mistype."}, {"role": "user", "content": "code@stackoverflow.com"}, {"role": "assistant", "content": "code@stackoverflow.com"}]}
{"messages": [{"role": "system", "content": "You help to ensure an email address is correct. Email address correction. Fix email format. Fix obvious email mistype."}, {"role": "user", "content": "test74430@gmail.com"}, {"role": "assistant", "content": "test74430@gmail.com"}]}
{"messages": [{"role": "system", "content": "You help to ensure an email address is correct. Email address correction. Fix email format. Fix obvious email mistype."}, {"role": "user", "content": "user.r222224@yahoo.com"}, {"role": "assistant", "content": "user.r222224@yahoo.com"}]}
{"messages": [{"role": "system", "content": "You help to ensure an email address is correct. Email address correction. Fix email format. Fix obvious email mistype."}, {"role": "user", "content": "patjohnson@aol.com"}, {"role": "assistant", "content": "patjohnson@aol.com"}]}
{"messages": [{"role": "system", "content": "You help to ensure an email address is correct. Email address correction. Fix email format. Fix obvious email mistype."}, {"role": "user", "content": "sallywork@mail.com"}, {"role": "assistant", "content": "sallywork@mail.com"}]}
{"messages": [{"role": "system", "content": "You help to ensure an email address is correct. Email address correction. Fix email format. Fix obvious email mistype."}, {"role": "user", "content": "mywork333@aol.com"}, {"role": "assistant", "content": "mywork333@aol.com"}]}
{"messages": [{"role": "system", "content": "You help to ensure an email address is correct. Email address correction. Fix email format. Fix obvious email mistype."}, {"role": "user", "content": "pj@gmail.com"}, {"role": "assistant", "content": "pj@gmail.com"}]}
{"messages": [{"role": "system", "content": "You help to ensure an email address is correct. Email address correction. Fix email format. Fix obvious email mistype."}, {"role": "user", "content": "sillycat666@gmail.com"}, {"role": "assistant", "content": "sillycat666@gmail.com"}]}
{"messages": [{"role": "system", "content": "You help to ensure an email address is correct. Email address correction. Fix email format. Fix obvious email mistype."}, {"role": "user", "content": "techsupport@toyota.jp"}, {"role": "assistant", "content": "techsupport@toyota.jp"}]}
{"messages": [{"role": "system", "content": "You help to ensure an email address is correct. Email address correction. Fix email format. Fix obvious email mistype."}, {"role": "user", "content": "dev@honda.com"}, {"role": "assistant", "content": "dev@honda.com"}]}
{"messages": [{"role": "system", "content": "You help to ensure an email address is correct. Email address correction. Fix email format. Fix obvious email mistype."}, {"role": "user", "content": "trips@booking.com"}, {"role": "assistant", "content": "trips@booking.com"}]}
{"messages": [{"role": "system", "content": "You help to ensure an email address is correct. Email address correction. Fix email format. Fix obvious email mistype."}, {"role": "user", "content": "article@news.com"}, {"role": "assistant", "content": "article@news.com"}]}
{"messages": [{"role": "system", "content": "You help to ensure an email address is correct. Email address correction. Fix email format. Fix obvious email mistype."}, {"role": "user", "content": "u34582765639aol.com"}, {"role": "assistant", "content": "u34582765639@aol.com"}]}
{"messages": [{"role": "system", "content": "You help to ensure an email address is correct. Email address correction. Fix email format. Fix obvious email mistype."}, {"role": "user", "content": "kfnnwufnjte@gmil.com"}, {"role": "assistant", "content": "kfnnwufnjte@gmail.com"}]}
{"messages": [{"role": "system", "content": "You help to ensure an email address is correct. Email address correction. Fix email format. Fix obvious email mistype."}, {"role": "user", "content": "fnance@nyse.com"}, {"role": "assistant", "content": "finance@nyse.com"}]}
{"messages": [{"role": "system", "content": "You help to ensure an email address is correct. Email address correction. Fix email format. Fix obvious email mistype."}, {"role": "user", "content": "sue.waters@google.ir"}, {"role": "assistant", "content": "sue.waters@google.ie"}]}
{"messages": [{"role": "system", "content": "You help to ensure an email address is correct. Email address correction. Fix email format. Fix obvious email mistype."}, {"role": "user", "content": "mike.johnspn@gmail.com"}, {"role": "assistant", "content": "mike.johnson@gmail.com"}]}
{"messages": [{"role": "system", "content": "You help to ensure an email address is correct. Email address correction. Fix email format. Fix obvious email mistype."}, {"role": "user", "content": "yahoo@yaho.com"}, {"role": "assistant", "content": "yahoo@yahoo.com"}]}
{"messages": [{"role": "system", "content": "You help to ensure an email address is correct. Email address correction. Fix email format. Fix obvious email mistype."}, {"role": "user", "content": "sallyhome@mil.cm"}, {"role": "assistant", "content": "sallyhome@mail.com"}]}
{"messages": [{"role": "system", "content": "You help to ensure an email address is correct. Email address correction. Fix email format. Fix obvious email mistype."}, {"role": "user", "content": "yorwork@aol.com"}, {"role": "assistant", "content": "yourwork@aol.com"}]}
{"messages": [{"role": "system", "content": "You help to ensure an email address is correct. Email address correction. Fix email format. Fix obvious email mistype."}, {"role": "user", "content": "hi..there@gmail"}, {"role": "assistant", "content": "hi.there@gmail.com"}]}
{"messages": [{"role": "system", "content": "You help to ensure an email address is correct. Email address correction. Fix email format. Fix obvious email mistype."}, {"role": "user", "content": "john doe@aol.com"}, {"role": "assistant", "content": "johndoe@aol.com"}]}
{"messages": [{"role": "system", "content": "You help to ensure an email address is correct. Email address correction. Fix email format. Fix obvious email mistype."}, {"role": "user", "content": "pat@gmailcom"}, {"role": "assistant", "content": "pat@gmail.com"}]}
{"messages": [{"role": "system", "content": "You help to ensure an email address is correct. Email address correction. Fix email format. Fix obvious email mistype."}, {"role": "user", "content": "sales@rockin.i"}, {"role": "assistant", "content": "sales@rockin.ai"}]}
{"messages": [{"role": "system", "content": "You help to ensure an email address is correct. Email address correction. Fix email format. Fix obvious email mistype."}, {"role": "user", "content": "sam@aol.om"}, {"role": "assistant", "content": "sam@aol.com"}]}
{"messages": [{"role": "system", "content": "You help to ensure an email address is correct. Email address correction. Fix email format. Fix obvious email mistype."}, {"role": "user", "content": "johny@hmail.com"}, {"role": "assistant", "content": "johny@gmail.com"}]}
{"messages": [{"role": "system", "content": "You help to ensure an email address is correct. Email address correction. Fix email format. Fix obvious email mistype."}, {"role": "user", "content": "bookin@booking.com"}, {"role": "assistant", "content": "booking@booking.com"}]}
{"messages": [{"role": "system", "content": "You help to ensure an email address is correct. Email address correction. Fix email format. Fix obvious email mistype."}, {"role": "user", "content": "web3gmailcom"}, {"role": "assistant", "content": "web3@gmail.com"}]}
{"messages": [{"role": "system", "content": "You help to ensure an email address is correct. Email address correction. Fix email format. Fix obvious email mistype."}, {"role": "user", "content": "pstmaster@hello.com"}, {"role": "assistant", "content": "postmaster@hello.com"}]}
{"messages": [{"role": "system", "content": "You help to ensure an email address is correct. Email address correction. Fix email format. Fix obvious email mistype."}, {"role": "user", "content": "mkejames@yaho.com"}, {"role": "assistant", "content": "mikejames@yahoo.com"}]}
{"messages": [{"role": "system", "content": "You help to ensure an email address is correct. Email address correction. Fix email format. Fix obvious email mistype."}, {"role": "user", "content": "hithere#mail.com"}, {"role": "assistant", "content": "hithere@mail.com"}]}
{"messages": [{"role": "system", "content": "You help to ensure an email address is correct. Email address correction. Fix email format. Fix obvious email mistype."}, {"role": "user", "content": "mymail!gmail,ca"}, {"role": "assistant", "content": "mymail@gmail.com"}]}
{"messages": [{"role": "system", "content": "You help to ensure an email address is correct. Email address correction. Fix email format. Fix obvious email mistype."}, {"role": "user", "content": "juunkgmailom"}, {"role": "assistant", "content": "junk@gmail.com"}]}
{"messages": [{"role": "system", "content": "You help to ensure an email address is correct. Email address correction. Fix email format. Fix obvious email mistype."}, {"role": "user", "content": "jjohn@gmmail.com"}, {"role": "assistant", "content": "john@gmail.com"}]}
{"messages": [{"role": "system", "content": "You help to ensure an email address is correct. Email address correction. Fix email format. Fix obvious email mistype."}, {"role": "user", "content": "rchel@news.com"}, {"role": "assistant", "content": "rachel@news.com"}]}
{"messages": [{"role": "system", "content": "You help to ensure an email address is correct. Email address correction. Fix email format. Fix obvious email mistype."}, {"role": "user", "content": "o.miler@aol.ca"}, {"role": "assistant", "content": "o.miller@aol.ca"}]}
{"messages": [{"role": "system", "content": "You help to ensure an email address is correct. Email address correction. Fix email format. Fix obvious email mistype."}, {"role": "user", "content": "nicemail@ahoo..com"}, {"role": "assistant", "content": "nicemail@yahoo.com"}]}
How Many Examples Are Needed For Fine-Tuning OpenAI?
When training GPT, or any AI model really, the more data we have the better. If we had to develop a production-ready model then we would likely use at tens of thousands at least.
One of the main challenges of working with AI is not actually creating the model, but gathering the training data. This remains to be a major hurdle for any AI project.
Please remember to split the training data into two groups: one for training and a second one for validation of the model. Also, within each group, there should be a roughly equal number of cases for each kind of result we are looking for. For example, if we wanted to train vision system to recognize digits, it would make little sense to only train it on pictures of "6". It would never learn how to handle all of the other digits.
Likewise, in our exercise we train the model based on exactly 25 valid email addresses, and 25 invalid ones. We will also validate it against another group of 25 valid and invalid cases.
Training the GPT Model
Once we have a working OpenAI API client, and training data, the actual training part is easy. Below is all of the code needed to get this part done:
<?php
include_once('OpenAIFineTuneSimpleClient.php');
$trainingDataFile = __DIR__.'/train.jsonl';
//upload file to OpenAI APIs
$fileData = OpenAIFineTuneSimpleClient::uploadFile($trainingDataFile);
$fileId = $fileData['id'];
//create fine-tune based on uploaded file
$fineTune = OpenAIFineTuneSimpleClient::createFineTune($fileId);
//list our current existing fine-tunes
$fineTunes = OpenAIFineTuneSimpleClient::getFileTunes();
If we print the list of fine tunes, we will get something like the response below, which will just say that our fine-tune is queued for training:
(
[object] => list
[data] => Array
(
[0] => Array
(
[object] => fine_tuning.job
[id] => ftjob-sWfFKYBvGuxESHG5NGfUtaIB
[model] => gpt-3.5-turbo-1106
[created_at] => 1699403239
[finished_at] =>
[fine_tuned_model] =>
[organization_id] => org-JTVHhgoYtb2mHLDispep2oQB
[result_files] => Array
(
)
[status] => queued
[validation_file] =>
[training_file] => file-Lf7BL8hTqwkWrCNTsbjs9Yr3
[hyperparameters] => Array
(
[n_epochs] => 3
[batch_size] => 1
[learning_rate_multiplier] => 2
)
[trained_tokens] =>
[error] =>
)
)
[has_more] =>
)
Notice the default hyperparameters (epochs, batch size and learning rate) - these are all very familiar from the PyTorch world! We can also change these settings, but for most cases the default will be a good start.
How Long Did It Take To Create A Fine-Tuned GPT Model?
Sometimes, training a new model goes quickly, and sometimes not. This depends on how busy the OpenAI servers are at the moment, and how much training data we have. In our case this took 4 hours. After the four hours passed, there was an email notification from OpenAI, and also if we fetch the fine-tunes again, we can see that our model is ready for use!
(
[object] => list
[data] => Array
(
[0] => Array
(
[object] => fine_tuning.job
[id] => ftjob-sWfFKYBvGuxESHG5NGfUtaIB
[model] => gpt-3.5-turbo-1106
[created_at] => 1699403239
[finished_at] => 1699421550
[fine_tuned_model] => ft:gpt-3.5-turbo-1106:personal::8IVSxqAx
[organization_id] => org-JTVHhgoYtb2mHLDispep2oQB
[result_files] => Array
(
[0] => file-HUiHkG9h4iKFQz1Su2ntwV3p
)
[status] => succeeded
[validation_file] =>
[training_file] => file-Lf7BL8hTqwkWrCNTsbjs9Yr3
[hyperparameters] => Array
(
[n_epochs] => 3
[batch_size] => 1
[learning_rate_multiplier] => 2
)
[trained_tokens] => 7086
[error] =>
)
)
[has_more] =>
)
The most important detail of the above is that the training process succeeded. Also, we have a new fine-tuned model ID, and we also know how many tokens in total were consumed for training. Some people have issues here because they think the training data is limited to 2048 tokens - but this is not true!
Each training case is limited (gpt-3.5-turbo is 2048, and gpt-4 can support up to 128k tokens) - but we can use any number of cases we want. So, we can have thousands of small training cases, which should give the model deep learning capabilities.
Validating The Fine-Tuned GPT
When validating a fine-tuned GPT, it is important to not use any of the same cases we used for training. We expect the model to learn these, but we want to try new cases!
We created a test script, using a completely different set of data. Let's see how our model performs:
<!--?php
include_once('OpenAIFineTuneSimpleClient.php');
$model='ft:gpt-3.5-turbo-1106:personal::8IVSxqAx';
$input = [
//correct emails (AI no suggested change expected)
'eric.jones@google.com' => 'eric.jones@google.com',
'jones.eric@gmail.com' => 'jones.eric@gmail.com',
'finetune777@mail.com' => 'finetune777@mail.com',
'himom@yahoo.com' => 'himom@yahoo.com',
'abelincoln@aol.com' => 'abelincoln@aol.com',
'g.washington@google.com' => 'g.washington@google.com',
'george.w@gmail.com' => 'george.w@gmail.com',
'george_w@mail.com' => 'george_w@mail.com',
'elvis_theking@yahoo.com' => 'elvis_theking@yahoo.com',
'rocknroll@aol.com' => 'rocknroll@aol.com',
'ticketsales@google.com' => 'ticketsales@google.com',
'pieceofcake1991@gmail.com' => 'pieceofcake1991@gmail.com',
'iam.theone@mail.com' => 'iam.theone@mail.com',
'starfamous@yahoo.com' => 'starfamous@yahoo.com',
'john_richards@aol.com' => 'john_richards@aol.com',
'ceo@google.com' => 'ceo@google.com',
'cio@gmail.com' => 'cio@gmail.com',
'canadatest@mail.ca' => 'canadatest@mail.ca',
'test@yahoo.de' => 'test@yahoo.de',
'test88923@aol.fr' => 'test88923@aol.fr',
'mailme.here@google.com' => 'mailme.here@google.com',
'm.j@gmail.com' => 'm.j@gmail.com',
'ironman@mail.com' => 'ironman@mail.com',
'mr.johnson@yahoo.com' => 'mr.johnson@yahoo.com',
'no7way2@aol.com' => 'no7way2@aol.com',
//emails with typos (AI suggested change expected)
'jj@google.cm' => 'jj@google.com',
'workgmail..com' => 'work@gmail.com',
'.hr@mail.com' => 'hr@mail.com',
'suppor@yahoo.com' => 'support@yahoo.com',
'srvice@aol.com' => 'service@aol.com',
'woork4832943@google.com' => 'work4832943@google.com',
'bornlastcenury20@gmail.com' => 'bornlastcentury20@gmail.com',
'jake@ma3il.ccom' => 'jake@mail.com',
'jeff@amazn.com' => 'jeff@amazon.com',
'pony1234567aol.com' => 'pony1234567@aol.com',
'fidget_spinner@goggle.com' => 'fidget_spinner@google.com',
'ht..@gmail.com' => 'ht@gmail.com',
'joe@@mail.com' => 'joe@mail.com',
'care##yahoo/com' => 'care@yahoo.com',
'missing@nsdaq.om' => 'missing@nsdaq.com',
'way1975@google.co' => 'way1975@google.com',
't.biggs^gmailcom' => 't.biggs@gmail.com',
'vacationsmail.com' => 'vacationsmail.com',
'the dude@yahoo.com' => 'thedude@yahoo.com',
'g*o*o*d@aol.com' => 'good@aol.com',
'piceofcake@google.com' => 'pieceofcake@google.com',
'www555@gmailom' => 'www555@gmail.com',
'parse...dots@mail.com' => 'parse.dots@mail.com',
'true@yahoo.vcom' => 'true@yahoo.com',
'thebestaolcom' => 'thebest@aol.com',
];
$failCount = 0;
foreach($input as $inputEmail => $expectedOutputEmail) {
$messages = [
[
"role" => "user",
"content" => $inputEmail
]
];
$gptOutput = OpenAIFineTuneSimpleClient::chat($messages, 100, $model);
$aiSuggestedEmail = $gptOutput['choices'][0]['message']['content'];
if(strcasecmp($expectedOutputEmail, $aiSuggestedEmail) != 0) {
$failCount++;
echo 'Failed Test: (input: '.$inputEmail.') (expected: '.
$expectedOutputEmail.') (result: '.
$aiSuggestedEmail.')<br?>';
}
}
$totalTests = count($input);
echo 'Total Tests: '.$totalTests.'
';
echo 'Failed Tests: '.$failCount.'
';
echo 'Model Accuracy: '.(($totalTests - $failCount) / $totalTests * 100).'%
';
And below is the result of our tests.
We managed to achieve a model with 90% accuracy, and the performance is very good at just 0.2368 seconds per test!
Looking at the failed tests, we believe this is an issue of our data set being too small for the job, but the results are surpisingly good already.
How Much Did It Cost To Fine-Tune GPT?
During our training stage, we used up around 8k tokens. We also ran many tests (even repeating some of them to make sure the results were reliable). The total cost was roughly $1.50, which is a great deal considering we got to experiment with one of the most advanced AI models, and used some of the best infrastructure in the world. We ran our code from local dev systems, which have no cost.
Hint: since the dollar cost of producing a single fine-tune is so low, it's possible to generate several variations of the same model based on slightly different training datasets, and then use the one that has the best validation test results.
Testing Our Model In OpenAI Playground
OpenAI has a playground, where we can accomplish most of the above tasks by an application. It was nice to see that our new model was made available in this application, and we did some testing of our own model vs. GPT-3.5-Turbo.
Fine-tuned GPT model successfuly fixing a mistyped email address
Normal non-tuned GPT model failing to fix the same email address
Conclusion
We are very happy with the opportunity to train some of the most powerful AI models available. From our short test, we could definitely see the benefit of using GPT fine-tunes in certain cases where a generative AI would excel. We managed to create a new model, which is capable of doing a good job at recognizing mistyped email addresses.
In the future, it would be interesting to have some kind of a download option on the model, and run it by ourselves. While the whole experience was impressive, we have to note that OpenAI will forever own the model we created, and charge us every time we want to use it.
If you are interested in other ideas for training GPT models, we have a separate article about cleaning up real-world data with AI and another example for using embeddings to compare and search texts.
