Start with the examples
A model needs arrays. Data arrives as a file full of rows, images, or measurements. Which entries belong together, which one is the answer, and which information will exist at prediction time? Those decisions come before the first gradient. Keep each conversion small enough to inspect, with a route back to its source.
A sample is one example used as input or evaluation. A feature is a measured or encoded input value. A row can be a sample without every column being a feature: the target has to come out before the inputs are built. An identifier can be useful for splitting and useless as a prediction feature. Ask what each column means before converting everything that looks numeric into a tensor.
Read the axes aloud. A table X has shape (N,F): N samples, F features. Images have (N,C,H,W): samples, channels, height, width. Token IDs have (N,T), with T counting sequence positions. Numerical sequences have (N,T,F). Graphs combine node features X=(N,F) with adjacency A=(N,N). Here T is a sequence length, not softmax temperature. One letter can carry a local meaning as long as you say so.
A tiny table
Our CSV columns are length,color,target, with rows (1,small,0), (3,large,1), and (2,small,0). The strings “small” and “large” are category values, whatever the column name suggests. Take target out of the features. Fit a category vocabulary from training rows: small=(1,0), large=(0,1). Each row now has a numeric length plus two category indicators. Unknown categories become all zeros in our runner, and a missing cell is reported as an error.
Keep information in its proper split
The training split fits the weights. Validation guides development choices. A test set is held back for the final assessment once those choices are done. Leakage is evaluation information entering fitting or selection, preprocessing statistics as well as model updates. Compute a mean on the whole file and then split it, and the training representation already carries information from the evaluation rows.
Independent rows are an assumption, not a consequence of storing data in a CSV. Repeated observations from one person or device can be closely related. Keep those groups together when you evaluate transfer to new groups. For future prediction, split chronologically. Stratification preserves class representation in a random split, which helps, and it replaces neither grouping nor time order. The right split follows the question the evaluation is meant to answer.
Fit the scale on training only
Training values [1,3] have mean 2. Their population standard deviation is sqrt(((1−2)²+(3−2)²)/2)=1. Subtracting 2 and dividing by 1 gives [−1,1]. A held-out value 4 becomes (4−2)/1=2, using those same training statistics. Including 4 in the mean would leak evaluation information. A held-out value outside the transformed training range is fine; don’t refit the scale to make it look familiar.
Make the representation explicit
Standardization subtracts the training mean and divides by the training standard deviation. Normalization is the broader idea of moving values to a chosen numerical scale or convention. The two words cover different operations, so our export records the recipe it used. We write m for the training mean, s for the training standard deviation, x for a raw value, and z for the transformed value:
The numeric ledger keeps a feature’s name, whether it was treated as numeric, its fitted mean, and its fitted scale. Category columns keep their training vocabulary instead. The runner stores this preprocessing in model.json, because the weights expect that exact input representation later. Reorder the categories or change a numeric scale and every weighted sum can change, even with identical exported weights.
$$z=\frac{x-m}{s}$$In words: subtract the training mean and divide by the training standard deviation; handle constant features explicitly.
Images, text, and windows
A 2×2 grayscale image [[0,1],[1,0]] has shape (1,2,2) before the sample axis is added. A tokenizer is a rule converting text into coded items. In our hand vocabulary, a,b,a maps to [1,2,1]; a,b pads to [1,2,0] with mask [1,1,0]. These IDs come from our hand vocabulary; the transformer chapter’s model uses its own. For the ordered sequence [1,2,3] and window two, input [1,2] predicts target 3.
The mask says which positions are real inputs for the operation at hand. It is separate metadata, not a promise that a zero-valued feature can be ignored. A legitimate pixel can have brightness zero and still matter to a convolution. A graph needs its connectivity as well as its node features. Keep only the feature matrix and the message-passing neighborhood is gone, even though every node still has numbers.
Inspect each conversion
Start with the three-row CSV and use the first two rows for training. The length feature uses their mean and scale; the third row is transformed afterward. Select a numeric output and trace it back to its CSV row and column. Then change the training-row count and watch the numeric statistics and the category vocabulary both move. A split choice affects the representation, not only the rows behind the loss.
Where this shows up when you train
Augmentation makes modified training examples meant to keep the target’s meaning. A slight image shift fits when the class should survive that shift. It doesn’t fit when position defines the label. Define the allowed transformation from the task before you make extra examples. Keep every version of an original example inside its training group, so a transformed copy never leaks into validation.
More rows don’t always mean more independent evidence. A thousand near-duplicate frames can describe less variation than a smaller set from distinct conditions. A random split hides that by putting nearly identical records on both sides. Inspect group identifiers and time ranges, not only split percentages. No amount of model arithmetic repairs a misleading evaluation boundary chosen before training started.
What you now know
- Data axes have meanings as well as lengths.
- Fit preprocessing on training examples only.
- Splits must respect the independence and time structure of the task.
Where we’re headed
Next: choosing the objective, model, update settings and stopping rule for a concrete run. Continue the story.