Jupyter Notebook Intelligence

Bringing the power of Copilot to Jupyter Labs - full article

Appropriately, image generated with Microsoft Copilot in Edge

Is this the final feature that makes Jupyter Lab a complete IDE? Jupyter already has many features that make it a great IDE, but now it also has GitHub Copilot integration.

We’re going to explore Notebook Intelligence (NBI), a Jupyter Lab extension that brings the power of GitHub Copilot to Jupyter Lab. The author, Mehmet Bektas, has already written an introductory article, on Medium, which is well worth reading (Introducing Notebook Intelligence!). It shows you NBI’s main features and how to install the extension. Bektas also points out that, despite his close relationship with the Jupyter Project, NBI is not an official Jupyter extension. But I would like to point out that as a Jupyter Distinguished Contributor, this guy definitely knows his stuff.

Here I’m going to look at three ways we can get NBI to quickly create new code — both Notebooks and a Streamit program. I’ve published all of the code, including Notebooks that go step-by-step in an associated GitHub repo.

We can use NBI to create a new Notebook in two ways — by giving prompts from within an existing Notebook or by using the /newNotebook command. We can also use NBI to create a new Python program using the /newPythonFile command, which we will use to create a Streamlit app.

The NBI UI

After installation, NBI can be seen in the toolbar on the left of the Jupyter Lab window. But before you can use it you need to log in to a GitHub account to use Copilot. This is straightforward, to log in NBI will provide you with a code you can copy and a link to click on. The link will take you to GitHub where you can enter the code and log in.

Now NBI is available. You may notice that code completions pop up as you type — just like Copilot in VSCode — just hit <tab> to accept them or carry on typing to ignore them.

You can also hit <ctrl-G> to bring up an NBI prompt in a Notebook. You can ask it to modify the code in that cell or create new code. A nice feature is that it shows you a diff of the original and the suggested code before you accept it. You can also edit the suggested code — the screenshot below illustrates this.

That is very similar to the way Copilot works in VSCode. And I used this feature to create a Notebook where I wrote almost no code myself. I simply wrote the prompts and accepted the suggestions. Below, you can see the result of a code suggestion from a blank cell. NBI creates the code and adds the prompt as a comment.

(Note the circled icon in a Notebook cell where you can click instead of hitting <ctrl-G> to bring up the prompt.)

The step-by-step approach to creating a new Notebook

The Notebook that I created analyses climate change data since 1850. The raw data shows how temperatures have changed over that period and gives figures for each month, in each year. I wanted to see how the summer temperatures had changed over time so needed to reformat the data and average the summer months into a single figure. Then I wanted to draw a bar chart of the result.

Instead of writing the code to do this, I let NBI do all the work.

Well, most of it. I wrote a line to read in the raw data.

import pandas as pd
df = pd.read_csv('HadCRUT-global-monthly.csv')

This gave me a dataframe like this:

You can see that the date combines the year and the month but I want to separate those and create a table of temperatures in monthly columns with a row for each year. So I opened a new cell and prompted NBI to create new columns for year and month. You can see the result below — the comment is the prompt that I gave.

# nbi-prompt: in df there is a column 'Time' that contains a date in 
# YYYY-MM format. Split this into two columns, one for YYYY and one for MM 
df['Year'] = df['Time'].str.split('-').str[0]
df['Month'] = df['Time'].str.split('-').str[1]
df.head()

This gave me the extra columns I needed so the next thing was to reformat the table.

# nbi-prompt: create a new dataframe that uses the data from df. 
# The new dataframe should have columns for each of the months and rows for each year. 
#The cells should contain the Anomaly (deg C) values
new_df = df.pivot(index='Year', columns='Month', values='Anomaly (deg C)')
new_df.head()

And then I wanted to create the ‘summer’ column.

# nbi-prompt: Create a new column in new_df called summer that is the average of the months 06, 07 and 08
new_df['summer'] = new_df[['06', '07', '08']].mean(axis=1)
new_df.head()

Remember, all of this code was written by NBI — I just wrote the prompts — and I ended up with the table that I wanted

Now I wanted a chart and that was easily created with the following prompt:

Draw a matplotlib bar chart of 'summer' over 'Year'

This gave me a suitable chart but all in one colour, so I selected the code in the cell hit <Ctrl-G> and asked it to make the positive bars red and the negative ones blue. NBI came up with a diff pane.

You can see the original code was adequate and the new code added the colours that I wanted. But the x-axis is a bit of a mess so I went through the same procedure to ask it to rotate the labels by 45 deg. and to only mark every 10th year. The resulting chart was fine.

That’s good but how about using Plotly? Again, I selected the codel and went through a series of prompts. The first asked NBI to convert to Plotly. That gave a standard chart but I then asked it to use the ‘plotly_white’ template and the ‘inferno’ colour scheme. The resulting code was this:

colors = px.colors.sequential.Inferno
fig = px.bar(x=new_df.index, y=new_df['summer'], color=new_df['summer'], color_continuous_scale=colors)
fig.update_layout(xaxis_tickangle=45, xaxis_dtick=10)
fig.update_xaxes(title='Year')
fig.update_yaxes(title='Summer Anomaly (deg C)')
fig.update_layout(title='Summer Anomaly over Year')
fig.update_layout(template='plotly_white', width=800, height=600)

And that is pretty much the result that I wanted. I’ve skipped a few steps here but you can see the detailed version in the Notebook in the GitHub repo.

Creating a complete Notebook

As in VSCode NBI has a chat interface. You can use this as a way of generating code that you can insert into a Notebook and it is here that you can use the /newNotebook and /newPythonFile commands. These last two allow you to create entire Notebooks or Python programs.

To demonstrate this I concatenated prompts from the previous exercise to create a complete Notebook in one go.

The screenshot below shows the chat interface after I have prompted NBI to create a new Notebook.

The input for the chat interface is at the bottom and the textual response above it. The code response is in the new Notebook you see on the right of the window. If you zoom in on the image you can see the complete prompt (otherwise look in the repo) and the Notebook is a complete, nicely formatted and documented version of the previous code that produces the same Plotly chart.

Creating a complete Streamlit app

Creating a complete app was as simple as creating the last Notebook. Instead of /newNotebook the prompt is newPythonFile and I gave it the following prompt:

Create a Python Streamlit app in the file weather.py that does the following:
Create a new pandas dataframe with the following data:'HadCRUT-global-monthly.csv'
The dataframe contains the following columns: 'Time', 'Anomaly (deg C)', 'Uncertainty (deg C)', 'Lower bound (deg C)', 'Upper bound (deg C)'.
The column 'Time' that contains a date in YYYY-MM format. Split this into two columns, one for years and one for months. Convert the numbers to names, e.g. 01 becomes January, 02 becomes February, etc.
Create a new dataframe that uses the data from the current one. 
The new dataframe should have columns for each of the months and rows for each year. 
The cells should contain the 'Anomaly (deg C)' values.
Create a new column in the new dataframe for the summer temperature that is the average of the months June, July and August).
Draw a Plotly Express bar chart of summer temperatures over each year.
Use the plotly_white template, change the size to 800 x 600 and use the 'inferno' color scheme.
Turn the x-axis labels 45 degrees and only show a label for every 10th year.
Add a suitable Streamlit title and subtitle for the page.

NBI went on to produce a complete app in a Python file that is the equivalent of the Notebook.

Conclusion

This little experiment was done with v1.0.0; with this version, there was no <Ctrl-G> command when editing Python files. I suggested to Mehmet that this would be a good idea. Hey, Presto! There is now a v1.1.0 with that feature!

I’m seriously thinking that Jupyter Lab with NBI might replace my usual VSCode environment. I often find that I prototype stuff in a Notebook, first, and then move on to a Python program when I’m happy with the result. Jupyer Lab lets you do this as easily as VSCode and it supports Markdown, has a terminal and Git integration (via an extension) — what’s not to like?

Notes

The data used in this exercise was downloaded from Met Office Hadley Centre observations datasets and is used under the UK Open Government Licence for Public Sector Information v3.0