Part I · Ch. 1 — A Number with a Shape

Part I · Chapter 1 of 29

A Number with a Shape

Read the containers a model holds


What can twelve numbers look like?

Open a model’s saved file and you won’t find sentences explaining what it knows. You’ll find named collections of numbers. Before you can change those numbers you have to read the containers holding them: how many entries there are, what belongs together, and how to point at one entry without guessing.

Start with one number, then put a second beside it. A scalar is one number; a vector is an ordered list of numbers. Stack two of those rows and you have a matrix, a rectangular table.

A scalar, vector, and matrix contain 1, 2, and 4 entries. A separate twelve-entry row is rearranged into shapes (3,4), (2,2,3), and (1,2,2,3), keeping the same entry order.
A shape describes the container; reshaping preserves its entries and their count.

A tensor is a numeric container indexed along axes. An axis is one independent direction in its address: along a row, down a column, or between slabs. The shape lists the length of each axis in order, and the rank counts those axes. A single number has no axis, so we write its shape as the empty pair of parentheses, (). This use of rank differs from matrix rank, which measures independent directions in a transformation.

Our twelve entries are [0,1,2,3,4,5,0,1,2,3,4,5]. Each element is one entry, repeats included. The two copies of 2 are the same number sitting in different places. That difference matters the moment you select an entry and change its address.

Twelve entries, three containers

The row has shape (12). Three rows of four have shape (3,4), because 3×4=12. Two slabs, each with two rows of three, have shape (2,2,3), because 2×2×3=12. Read across each row before dropping to the next, and finish a slab before starting the next one.

The shape carries meaning. Twelve readings taken one after another and twelve readings from twelve sensors can hold identical numbers and still need different handling. A reshape changes the address system and nothing else. It doesn’t tell you which physical reading is the right one, and it doesn’t reorder the entries for you.

Every entry has an address

An index is an entry’s address. In the two-by-two table [[1,2],[3,4]], 3 is “second row, first column.” You need both halves, because a row holds several entries. “Second” on its own leaves open whether to count rows, columns, or every entry in one long line.

Two slabs each contain a 2 by 3 grid. The selected value 2 in the second slab has math address (2,1,3) and code address [1][0][2], ordered slab, row, column.
An index is an address, and its starting convention matters.

Take the two-by-three slab [[0,1,2],[3,4,5]] and call it the matrix W. Bold uppercase letters name matrices throughout the book, and the small letters below W give the row and the column. Math addresses on these pages start at one, so W₂₃ is the second row and third column, and its entry is 5.

$$ \mathbf{W}=\begin{bmatrix}0&1&2\\3&4&5\end{bmatrix},\quad W_{23}=5 $$

In words: the entry in the second row and third column is five.

A saved JSON list, Python and JavaScript all start counting at zero. The code address [1][2] points to that same 5. The table didn’t change; the counting did. Write the convention beside the address whenever you move between a diagram and code. Getting it wrong looks like bad arithmetic.

Addressing the second slab

Both slabs hold [[0,1,2],[3,4,5]]. The code address [1][0][2] reads: second slab, first row, third column. Its value is 2 and its mathematical address is (2,1,3). The first 2 in our original list is a different entry; this one sits at flat offset 8 counting from zero.

A shape such as (3,28,28) now reads plainly: three channels, each twenty-eight rows by twenty-eight columns. A channel is a separate plane of measurements over the same image locations. The three numbers alone don’t prove those are the axis meanings; the program has to tell you its convention. Another program might put channels last, and matching counts wouldn’t make the two layouts interchangeable.

The address follows the order of the axes. Channels first means the first index picks a channel; rows first means the same index picks a row. So a useful description of data gives axis names as well as lengths. Later computation diagrams carry those names along the arrows.

A picture is a tensor

These rules are enough to make a small picture. Take a four-by-four grid whose center four cells are bright and whose outer cells are dark. Each location stores a brightness: 0 for dark, 1 for bright. Writing the numbers beside the image doesn’t make it less of a picture. They are two views of the same data.

A 4 by 4 brightness grid contains 16 entries. Three channel grids of that size form shape (3,4,4), containing 48 entries.
A grayscale pixel is one brightness number; a color pixel has channel values.

The brightness rows are [0,0,0,0], [0,1,1,0], [0,1,1,0], and [0,0,0,0]. Four rows times four columns gives sixteen entries. A color image keeps three measurements per location, and RGB names those red, green, and blue channels. Three copies of our example form shape (3,4,4), with 3×4×4=48 entries. Three identical channels would still look gray.

The dtype says how a number is stored. A byte is eight binary digits, and float32 stores a floating-point number in four of them. Floating point covers fractions and a wide range of magnitudes, but only a finite set of values. Calculations shown on the page are rounded for reading; the interactives keep more precision inside.

Shape and storage answer different questions

Twelve float32 entries occupy 12×4=48 bytes of numeric payload. Shape (12), shape (3,4), and shape (2,2,3) all have that same payload. Software also stores information about the container, so the whole file, and the memory a running program uses, come to more than 48 bytes.

Display size is a third quantity. Zooming an illustration on your screen creates no new stored pixels. Going from a four-row grid to an eight-row grid does, and that needs a rule for inventing the new values. Keep the container, its storage representation, and its display size apart when you inspect an image dataset.

Rearrange the container

Before you change the shape below, find the selected 2 and predict its address in a three-by-four grid. Read across the original list: flat offsets zero through three fill row zero, four through seven fill row one, and eight through eleven fill row two. The selected entry becomes the first entry of row two under code indexing.

The same entries can have different shapes and addresses.

See it move

Enter (3,4), choose Reshape, and check the selected entry. Then go back to (2,2,3) and change only the index convention. The value stays 2 through both. One control changes the arrangement; the other changes what we call the addresses.

Ask for a shape with a different total count. The control reports the mismatch and offers a separate regeneration action. Reshaping keeps the entries; regeneration makes a new list. A larger shape draws a slice and states its full count; the entries it doesn’t draw are still there.

Where this shows up when you train

When a training program reports a shape mismatch, start with the named axes. A batch is a group of examples processed together. A batch axis puts several image tensors in one larger container without mixing their pixels. Its length says how many examples the operation gets at once, and each example keeps its own channels, rows, and columns.

That is the first habit of this workshop: make the stored object concrete. Count its entries, find one value, say what each axis means. Do those checks often enough and a model’s weight file reads as a collection of tables.

What you now know

  • A shape lists the lengths of the container’s axes.
  • Multiplying the axis lengths counts its entries.
  • Reshaping changes addresses while preserving entries.

Where we’re headed

You can read the numbers a model holds. Next a matrix moves an arrow, and then a whole grid. Continue to the next chapter.