Anthropic Claude 3.5: A More Powerful but Cheaper LLM

We look at Anthropic’s new Claude 3.5 Sonnet LLM, explore the API and create a simple Streamlit chatbot

---

A Streamlit chatbot using Claude

A quick question. This is a full article that will appear in your email. Is that OK? Or would you prefer a shorter email with a link to the full article?

Thank you - now on to the main topic.

On Jun 21, 2024, Anthropic announced Claude 3.5 Sonnet, the first in their new Claude 3.5 family. We are going to explore the Claude 3.5 API and create a simple chatbot with Streamlit.

According to Anthropic, Claude 3.5 Sonnet outperforms competitor models and, Anthropic’s previous top-of-the-range Claude 3 Opus, offering high intelligence at mid-tier speed and cost. You can get it for free on Claude.ai and the Claude iOS app — subscribers get higher rate limits. 

It’s also accessible via an API (which we will look at shortly) and cloud platforms, Amazon Bedrock, and Google Cloud’s Vertex AI.

I should explain that version 3 of the Anthropic offering consists of three different models, ranging from Haiku, the least powerful but fastest model, through Sonnet, the mid-range model, to the high-end Opus. So, if Sonnet 3.5 outperforms Opus 3, it will be interesting to see how the newer versions of Haiku and Opus shape up when they are released later in the year.

Anthropic has also introduced “Artifacts,” a new feature on Claude.ai for better content generation and collaboration. It’s an interesting approach to interactive development with an LLM where the output such as code, images and entire apps are displayed in a window at the side of the conventional chat interface. Take a look at the video below to see how the user begins with a simple image (a crab called “Claw’d” — what else!) and goes on to develop a old-school interactive game using simple English prompts.

I fully intend to explore this another day but, in this article, I want to look at the API and how to use it in a simple Streamlit app.

We’ll do this in Python, so before we start, we need to install the Anthropic library, e.g.

pip install anthropic

The API

The first thing to say is that it is simple. The code below is a plain vanilla call to Claude.

import anthropic

client = anthropic.Anthropic(api_key="YOUR API KEY HERE")

message = client.messages.create(
    model="claude-3-5-sonnet-20240620",
    max_tokens=1000,
    temperature=0,
    system="You are a helpful agent, please answer the queries as best you can",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "How were the Himalayas formed?"
                }
            ]
        }
    ]
)
print(message.content)

If you’ve used an LLM API before it will look familiar. First, we import the library (of course) and then you’ll need an API key from Anthropic to create a client. 

(For the API key, you will need to provide a credit card but to ensure that you don’t overspend, you can pre-pay only a small amount. You can also set a limit on your usage. For this experiment overspending is unlikely — you will only be charged a few cents for what we are about to do. Needless to say, you must not show your API key publically and you should keep an eye on your usage and spending via the Anthropic dashboard.)

So, having created the client, we can use it to call the client.messages.create() method to query the LLM and get a response.

The parameters are as follows:

1. max_tokens — This limits the output of the LLM. When the limit is reached the LLM will stop. The more tokens that are output, the more you will be charged, so start small.

2. temperature — This can range from 0.0 to 1.0 and defaults to 1.0. The closer the temperature is to 0.0 the more deterministic the response, the closer to 1.0, the more random.

3. system — This is the system prompt and is a way of providing context and instructions to Claude. Note that this is specified separately from the other prompts.

4. messages — This is a list of messages for input to the LLM. You can see that there is one message specified here, “How were the Himalayas formed?”. In later calls to Claude the list may contain prompts from the user and responses from the LLM.

The response from the LLM is a structure that contains the entire response — the content part contains the text, as can be seen below:

[TextBlock(text="The Himalayas were formed...", type='text')]

I’ve removed most of the response here because it is not very readable. To print a readable version of the response we can extract the text and print it like this:

print(message.content[0].text)

The result is presented below.

The Himalayas were formed through a process of continental collision that began approximately 50 million years ago and is still ongoing today. Here’s a brief overview of their formation: 

1. Plate Tectonics: The formation of the Himalayas is a result of the movement of tectonic plates, specifically the collision between the Indian Plate and the Eurasian Plate. 

2. Ancient Ocean: Before the collision, there was an ancient ocean called the Tethys Sea between the Indian and Eurasian plates. 

3. Northward Movement: The Indian Plate, which was once part of the southern supercontinent Gondwana, began moving northward about 200 million years ago. 

4. Collision: Around 50–55 million years ago, the Indian Plate collided with the Eurasian Plate. As both plates are composed of continental crust, neither could subduct fully beneath the other. 

5. Compression and Uplift: The collision caused intense compression of the Earth’s crust. The land at the collision zone was forced upwards, creating the Himalayan mountain range. 

6. Continued Pressure: The Indian Plate continues to move northward at about 1.5 inches (4 cm) per year, maintaining pressure on the Eurasian Plate and causing the Himalayas to grow taller by about 5 mm annually. 

7. Folding and Faulting: The intense pressure has resulted in complex folding and faulting of rock layers, contributing to the rugged terrain of the Himalayas. 

This ongoing process of mountain building (orogeny) makes the Himalayas one of the youngest and highest mountain ranges on Earth, with Mount Everest as its highest peak.

Pretty good and quite detailed, don’t you think?

OK, that’s all good but it isn’t an app. Let’s see how we can develop a Streamlit chatbot based on Claude.

A Streamlit Chatbot with Claude

We will use Streamlit’s very useful chat_message method to implement a simple chatbot that incorporates the code that we looked at above. 

The Streamlit chat_message method conveniently distinguishes between messages that are given to the LLM by the user and those that are generated by the LLM.

By default, a user prompt will be formatted like this:

And a response from the AI will look like this:

We will store user prompts and LLM-generated responses in a list of messages where each message is labelled as a ‘user’ or an ‘assistant’ message. These will be stored in the system state and new messages from the user and responses from the LLM will be appended to the list. This list is the ‘memory’ of the conversation that will be sent to the LLM with each new user prompt.

The main part of the code for the app is in three parts: get a prompt from the user; send the prompt to Claude; and display the resulting messages using the Streamlit UI components described above.

It’s worth noting, at this point, that we do not need to implement a loop to repeatedly get a prompt from the user. A Streamlit app always re-runs the entire code whenever a user input changes and so is a loop by default.

The code assumes that your API key is stored as a Streamlit secret. To run it locally, you will need to create a filesecrets.toml in a folder called .streamlit. In that file you should define your API key like this:

ANTHROPIC_API_KEY = "Type your key here"

Next, I’ll go through the main parts of the code but you can download the full code, including headers and the sidebar, from my GitHub repository.

Getting the user prompt

In the code below we prompt the user for an input. If one is entered we append it to the list of messages stored in the system state.

We also write the user prompt on the screen using the chat_message() method.

The next step is to call Claude with the new prompt and get its response. When that has been received the content of the response is added to the message list.

# Chat input
if prompt := st.chat_input("What would you like to ask Claude?"):
    # Add user message to chat history
    st.session_state.messages.append({"role": "user", "content": prompt})
    with st.chat_message("user"):
        st.write(prompt)

    # Get Claude's response
    with st.chat_message("assistant"):
        message_placeholder = st.empty()
        full_response = get_claude_response(prompt)
        message_placeholder.write(full_response)
    
    # Add Claude's response to chat history
    st.session_state.messages.append({"role": "assistant", "content": full_response})

Calling Claude

The function call Clause is a slightly modified version of the API call we saw earlier. We have not set a system prompt but the parameter is there should you wish to add something. For example, if you wanted Claude to give you ideas for a dinner you could set it to something like:

 “You are an expert chef and nutritionist with international knowledge. When asked for a recipe please also provide nutritional information.”

But by leaving it blank no specific instructions are given and you will have a general-purpose chatbot.

# Function to get response from Claude
def get_claude_response(prompt):
    try:
        response = anthropic.messages.create(
            model="claude-3-5-sonnet-20240620",
            max_tokens=1000,
            temperature=0,
            system = "",
            messages = st.session_state.messages
        )
        return response.content[0].text
    except Exception as e:
        return f"An error occurred: {str(e)}"

Displaying the result

It might seem a little unintuitive but, in the app, this code comes before the prompt. The thing to bear in mind is that because the code is re-run for every input, we are effectively in a loop and so the display code is run every time there is a new prompt.

You can see in the code below we loop through each message writing it to the screen with the chat_message() method.

# Display chat messages
for message in st.session_state.messages:
    with st.chat_message(message["role"]):
        st.write(message["content"])

Those are the main functional parts of the code.

Conclusion

We’ve seen how it is quite easy to use the new Claude 3.5 Sonnet LLM and how to incorporate it into a simple Streamlit app and create a general-purpose chatbot.

I recently wrote How to Build a ReAct AI Agent with Claude 3.5 and Python which is a slightly more involved use of Claude that creates an AI agent. The code for both this article and the previous one is in my GitHub repo (see below).

I expect to be writing more about Claude in the not-too-distant future.

---

Thanks for reading, I hope that this has been helpful.

All code and examples are available in my [Github repo](GitHub — alanjones2/claudeapps) — feel free to download them. The file for this particular app is called st-claude.py and the simple API code can be found in the Jupyter notebook c1.ipynb.

Disclaimer: I have no connection with Anthropic other than as a user of Claude.