According to Sam Altman, OpenAI’s CEO, when you reduce the Temperature and Top-P values, you not only reduce the creativity of the responses, you reduce the chances of error and hallucinations. This will be explained in the context of Sam Altman’s deep understanding of creativity and the role of mistakes in the creative process.
The insights and software control skills explained in this article can empower anyone to dial in the right balance of creativity and mistakes for a particular ChatGPT-4 assisted project. Lawyers, for instance, may, for most of their uses, want to lower the default settings, which are high on creativity. This should improve the probability of accurate, delusion free answers. That makes most lawyers and their bots very happy. Judges too. All of this, and more, will be explained.
Sam Altman on GPT Creativity
In the What is the Difference Between Human Intelligence and Machine Intelligence? blog I quoted portions of Sam Altman’s video interview at an event in India by the Economic Times to show his “tool not a creature” insight. There is another Q&A exchange in that same YouTube video starting at 1:09:05, that addresses creativity and mistakes.
Sam Altman’s answer here assumes you know about ChatGPT’s creativity volume controls, where you can, if you want, turn the creativity volume down to zero. In so doing, you will improve accuracy, but the response will often be boring. Boring, but accurate, may be just what you want sometimes, but that is not the default setting for ChatGPT, as will be explained and demonstrated.
ChatGPT Creativity SettingsThis section provides a technical explanation of these two settings. Much of this is difficult to understand, but worry not, and plough through it, because after this comes an easy to follow demonstration of what it all means. Multiple examples will be provided to allow you to see for yourself how the GPT controls work in practice. That is the hacker “hands on” e-Discovery Team way.
First, the technical explanation of the two volume controls for GPT creativity: Temperature and Nucleus Sampling (aka Top-P). Both typically have settings of between zero and one, 0.0 and 1.0.
TEMPERATURE: Technically temperature affects the probability distribution over the possible tokens at each step of the generation process. A temperature of 0 would make the model completely deterministic, always choosing the most likely token. The “temperature” setting in GPT and similar language models, such as ChatGPT, controls the randomness of the model’s responses. A higher temperature value makes the model’s responses more random, while a lower, cooler value, makes the responses more deterministic and focused. See eg. Cheat Sheet: Mastering Temperature and Top_p in ChatGPT API (OpenAI Forum).
Temperature values are said to produce a more focused, consistent, and deterministic output. It is like going from water, the higher 1.0 value, to ice, the colder, more probable value of 0.0.
All Images and Videos by Ralph Losey assisted by Midjourney
Typically, OpenAI experts say a higher temperature (e.g., 0.8) may be suitable when you want a range of ideas, brainstorming suggestions, or creative writing prompts. A lower temperature (e.g., 0.2) is more appropriate when you’re looking for a precise answer, a more formal response, or when the context demands consistency. The default setting for ChatGPT 3.5 and 4.0 is 0.7. That’s pretty hot, especially for most legal work. No doubt OpenAI have put a lot of research into that default setting, but I could not find it. Id., Also see: Prakash Selvakumar, Text Generation with Temperature and Top-p Sampling in GPT Models: An In-Depth Guide Open AI says that finding the right temperature setting may require experimentation to strike a balance between creativity and consistency that suits your specific needs. Id. This sounds like a good lawyer answer of “it depends.” There are a tremendous number of variables, different questions and needs, different circumstances. That is the same situation lawyers are in with many legal questions. Plus the OpenAI software itself is constantly being updated, even though the version number of 4.0 has not been changed since March 2023.
The Bottom line for lawyers is that the default setting of 0.7 is pretty high in the random predictions scale. Unless you are looking for clever, very creative language or legal imagination – off the wall ideas – lawyers and judges should use a lower setting. Maybe dial down the random creativeness to 0.2, or even zero – 0.0 – for maximum route parroting of the most probable information. You just want the cold truth.
As Sam Altman explained, lowering the temperature setting also makes it more likely that your answers will not have as many mistakes or hallucinations. Note that I did not say no mistakes, the software is too new, and life is too complex to say that. Human lawyers are still needed to verify the Ai. Just because it appears much smarter than you, it can still be wrong, no matter how conservative the temperature setting. Think of the brilliant, very creative, higher IQ than you, conservatively dressed, young associate with little or no actual legal experience.
Baby lawyer explains Top-P to skeptical partner, All Images and Videos by Ralph Losey assisted by Midjourney
LEUS SAMPLING (aka TOP-P): Top-P sampling is an alternative to temperature sampling. Technically, this means that instead of considering all probable tokens that are likely to come next, the Top-P parameter directs GPT to consider only a subset of all probable tokens (the nucleus) whose cumulative probability mass adds up to a certain threshold, the (Top-Probablity). For example, if Top_P is set to 0.1, GPT will consider only the tokens that make up the top 10% of the probability mass for the next token. This allows for dynamic vocabulary selection based on context. The setting values for Top-P are, like temperature, between 0.0 and 1.0.
another way, the Top-P sampling parameter maintains a balance between diversity and high-probability words by selecting tokens from the Top-P most probable tokens. They are the tokens whose collective probability is greater than or equal to a specified threshold p. The Top-P parameter helps ensure that the chatbot response is both diverse and relevant to the given context. Text Generation with Temperature and Top-p Sampling in GPT Models: An In-Depth Guide
All Images and Videos by Ralph Losey assisted by Midjourney
Baby lawyer explains Top-P to skeptical partner, All Images and Videos by Ralph Losey assisted by Midjourney
