Learn how to work with the ChatGPT and GPT-4 models (preview)

The models behave differently than the older GPT-3 models. Previous models were text-in and text-out, meaning they accepted a prompt string and returned a completion to append to the prompt. However, the ChatGPT and GPT-4 models are conversation-in and message-out. The models expect input formatted in a specific chat-like transcript format, and return a completion that represents a model-written message in the chat. While this format was designed specifically for multi-turn conversations, you'll find it can also work well for non-chat scenarios too.

In Azure OpenAI there are two different options for interacting with these type of models:

Chat Completion API.

Completion API with Chat Markup Language (ChatML).

The Chat Completion API is a new dedicated API for interacting with the ChatGPT and GPT-4 models. Both sets of models are currently in preview. This API is the preferred method for accessing these models. It is also the only way to access the new GPT-4 models.

ChatML uses the same completion API that you use for other models like text-davinci-002, it requires a unique token based prompt format known as Chat Markup Language (ChatML). This provides lower level access than the dedicated Chat Completion API, but also requires additional input validation, only supports ChatGPT (gpt-35-turbo) models, and the underlying format is more likely to change over time.

This article walks you through getting started with the new ChatGPT and GPT-4 models. It's important to use the techniques described here to get the best results. If you try to interact with the models the same way you did with the older model series, the models will often be verbose and provide less useful responses.

Working with the ChatGPT and GPT-4 models (preview)

The following code snippet shows the most basic way to use the ChatGPT and GPT-4 models with the Chat Completion API. If this is your first time using these models programmatically, we recommend starting with our ChatGPT & GPT-4 Quickstart.

GPT-4 models are currently in preview. Existing Azure OpenAI customers can apply for access by filling out this form.

import os import openai openai.api_type = "azure" openai.api_version = "2023-03-15-preview" openai.api_base = os.getenv("OPENAI_API_BASE") # Your Azure OpenAI resource's endpoint value. openai.api_key = os.getenv("OPENAI_API_KEY") response = openai.ChatCompletion.create( engine="gpt-35-turbo", # The deployment name you chose when you deployed the ChatGPT or GPT-4 model. messages=[ {"role": "system", "content": "Assistant is a large language model trained by OpenAI."}, {"role": "user", "content": "Who were the founders of Microsoft?"} ] ) print(response) print(response['choices'][0]['message']['content'])

{ "choices": [ { "finish_reason": "stop", "index": 0, "message": { "content": "The founders of Microsoft are Bill Gates and Paul Allen. They co-founded the company in 1975.", "role": "assistant" } } ], "created": 1679014551, "id": "chatcmpl-6usfn2yyjkbmESe3G4jaQR6bsScO1", "model": "gpt-3.5-turbo-0301", "object": "chat.completion", "usage": { "completion_tokens": 86, "prompt_tokens": 37, "total_tokens": 123

 Note

The following parameters aren't available with the new ChatGPT and GPT-4 models: logprobs, best_of, and echo. If you set any of these parameters, you'll get an error.

Every response includes a finish_reason. The possible values for finish_reason are:

stop: API returned complete model output.

length: Incomplete model output due to max_tokens parameter or token limit.

content_filter: Omitted content due to a flag from our content filters.

null:API response still in progress or incomplete.

Consider setting max_tokens to a slightly higher value than normal such as 300 or 500. This ensures that the model doesn't stop generating text before it reaches the end of the message.

Model versioning

gpt-35-turbo is equivalent to the gpt-3.5-turbo model from OpenAI.

Unlike previous GPT-3 and GPT-3.5 models, the gpt-35-turbo model as well as the gpt-4 and gpt-4-32k models will continue to be updated. When creating a deployment of these models, you'll also need to specify a model version.

Currently, only version 0301 is available for ChatGPT and 0314 for GPT-4 models. We'll continue to make updated versions available in the future. You can find model deprecation times on our models page.

Working with the Chat Completion API

OpenAI trained the ChatGPT and GPT-4 models to accept input formatted as a conversation. The messages parameter takes an array of dictionaries with a conversation organized by role.

The format of a basic Chat Completion is as follows:

{"role": "system", "content": "Provide some context and/or instructions to the model"}, {"role": "user", "content": "The users messages goes here"}

A conversation with one example answer followed by a question would look like:

"system", "content": "Provide some context and/or instructions to the model."}, {"role": "user", "content": "Example question goes here."}, {"role": "assistant", "content": "Example answer goes here."}, {"role": "user", "content": "First question/message for the model to actually respond to."}

System role

The system role also known as the system message is included at the beginning of the array. This message provides the initial instructions to the model. You can provide various information in the system role including:

A brief description of the assistant

Personality traits of the assistant

Instructions or rules you would like the assistant to follow

Data or information needed for the model, such as relevant questions from an FAQ

You can customize the system role for your use case or just include basic instructions. The system role/message is optional, but it's recommended to at least include a basic one to get the best results.

Messages

After the system role, you can include a series of messages between the user and the assistant.

Message prompt examples

The following section shows examples of different styles of prompts that you could use with the ChatGPT and GPT-4 models. These examples are just a starting point, and you can experiment with different prompts to customize the behavior for your own use cases.

Basic example

If you want the ChatGPT model to behave similarly to chat.openai.com, you can use a basic system message like "Assistant is a large language model trained by OpenAI."

": "system", "content": "Assistant is a large language model trained by OpenAI."}, {"role": "user", "content": "Who were the founders of Microsoft?"}

Example with instructions

For some scenarios, you may want to give additional instructions to the model to define guardrails for what the model is able to do.

{"role": "system", "content": "Assistant is an intelligent chatbot designed to help users answer their tax related questions. Instructions: - Only answer questions related to taxes. - If you're unsure of an answer, you can say "I don't know" or "I'm not sure" and recommend users go to the IRS website for more information. "}, {"role": "user", "content": "When are my taxes due?"}

Using data for grounding

You can also include relevant data or information in the system message to give the model extra context for the conversation. If you only need to include a small amount of information, you can hard code it in the system message. If you have a large amount of data that the model should be aware of, you can use embeddings or a product like Azure Cognitive Search to retrieve the most relevant information at query time.

{"role": "system", "content": "Assistant is an intelligent chatbot designed to help users answer techni

Few shot learning with Chat Completion

You can also give few shot examples to the model. The approach for few shot learning has changed slightly because of the new prompt format. You can now include a series of messages between the user and the assistant in the prompt as few shot examples. These examples can be used to seed answers to common questions to prime the model or teach particular behaviors to the model.

This is only one example of how you can use few shot learning with ChatGPT and GPT-4. You can experiment with different approaches to see what works best for your use case.

The Chat Completion API is designed to work with multi-turn conversations, but it also works well for non-chat scenarios.

For example, for an entity extraction scenario, you might use the following prompt:

Creating a basic conversation loop

The examples so far have shown you the basic mechanics of interacting with the Chat Completion API. This example shows you how to create a conversation loop that performs the following actions:

Continuously takes console input, and properly formats it as part of the messages array as user role content.

Outputs responses that are printed to the console and formatted and added to the messages array as assistant role content.

This means that every time a new question is asked, a running transcript of the conversation so far is sent along with the latest question. Since the model has no memory, you need to send an updated transcript with each new question or the model will lose context of the previous questions and answers.

import os import openai openai.api_type = "azure" openai.api_version = "2023-03-15-preview" openai.api_base = os.getenv("OPENAI_API_BASE") # Your Azure OpenAI resource's endpoint value . openai.api_key = os.getenv("OPENAI_API_KEY") conversation=[{"role": "system", "content": "You are a helpful assistant."}] while(True): user_input = input() conversation.append({"role": "user", "content": user_input}) response = openai.ChatCompletion.create( engine="gpt-3.5-turbo", # The deployment name you chose when you deployed the ChatGPT or GPT-4 model. messages = conversation ) conversation.append({"role": "assistant", "content": response['choices'][0]['message']['content']}) print("\n" + response['choices'][0]['message']['content'] + "\n")

When you run the code above you will get a blank console window. Enter your first question in the window and then hit enter. Once the response is returned, you can repeat the process and keep asking questions.

Managing conversations

The previous example will run until you hit the model's token limit. With each question asked, and answer received, the messages array grows in size. The token limit for gpt-35-turbo is 4096 tokens, whereas the token limits for gpt-4 and gpt-4-32k are 8192 and 32768 respectively. These limits include the token count from both the message array sent and the model response. The number of tokens in the messages array combined with the value of the max_tokens parameter must stay under these limits or you'll receive an error.

It's your responsibility to ensure the prompt and completion falls within the token limit. This means that for longer conversations, you need to keep track of the token count and only send the model a prompt that falls within the limit.

The following code sample shows a simple chat loop example with a technique for handling a 4096 token count using OpenAI's tiktoken library.

The code requires tiktoken 0.3.0. If you have an older version run pip install tiktoken --upgrade.

import tiktoken import openai import os openai.api_type = "azure" openai.api_version = "2023-03-15-preview" openai.api_base = os.getenv("OPENAI_API_BASE") # Your Azure OpenAI resource's endpoint value . openai.api_key = os.getenv("OPENAI_API_KEY") system_message = {"role": "system", "content": "You are a helpful assistant."} max_response_tokens = 250 token_limit= 4096 conversation=[] conversation.append(system_message) def num_tokens_from_messages(messages, model="gpt-3.5-turbo-0301"): encoding = tiktoken.encoding_for_model(model) num_tokens = 0 for message in messages: num_tokens += 4 # every message follows <im_start>{role/name}\n{content}<im_end>\n for key, value in message.items(): num_tokens += len(encoding.encode(value)) if key == "name": # if there's a name, the role is omitted num_tokens += -1 # role is always required and always 1 token num_tokens += 2 # every reply is primed with <im_start>assistant return num_tokens while(True): user_input = input("") conversation.append({"role": "user", "content": user_input}) conv_history_tokens = num_tokens_from_messages(conversation) while (conv_history_tokens+max_response_tokens >= token_limit): del conversation[1] conv_history_tokens = num_tokens_from_messages(conversation) response = openai.ChatCompletion.create( engine="gpt-35-turbo", # The deployment name you chose when you deployed the ChatGPT or GPT-4 model. messages = conversation, temperature=.7, max_tokens=max_response_tokens, ) conversation.append({"role": "assistant", "content": response['choices'][0]['message']['content']}) print("\n" + response['choices'][0]['message']['content'] + "\n")

In this example once the token count is reached the oldest messages in the conversation transcript will be removed. del is used instead of pop() for efficiency, and we start at index 1 so as to always preserve the system message and only remove user/assistant messages. Over time, this method of managing the conversation can cause the conversation quality to degrade as the model will gradually lose context of the earlier portions of the conversation.

An alternative approach is to limit the conversation duration to the max token length or a certain number of turns. Once the max token limit is reached and the model would lose context if you were to allow the conversation to continue, you can prompt the user that they need to begin a new conversation and clear the messages array to start a brand new conversation with the full token limit available.

The token counting portion of the code demonstrated previously, is a simplified version of one of OpenAI's cookbook examples.