Every part of the rosette pipeline, one piece at a time
Each section below covers one component. It says what the part does in plain words, why it was built that way, and what goes wrong without it. Most sections have a demo you can poke. The pictures are real images and real numbers from your project unless a demo is marked cartoon.
The whole system in one minute
A weeding robot needs two answers from a photo of the lawn: which pixels are each dandelion's leaves, and where the plant grows from. Two separate transformer networks answer those two questions.
What happens
- The frame is cut into overlapping squares so small plants stay big enough to see.
- Network 1 (RF-DETR Seg-Small) looks at each square and paints the pixels of each rosette.
- Plain code merges the same plant seen in two squares and cleans the masks.
- Network 2 (DINOv2 ViT-S/14) looks at a close-up of each plant and predicts the growth point.
Why two networks
"Which pixels are leaf" and "where is the crown" are different questions. The crown is often hidden under leaves or sits in a gap between them, so the middle of the mask is not a good answer. Section 11 shows this with a demo.
Words you will see a lot
- Mask: a yes/no answer for each pixel. Yes means "this pixel belongs to this plant".
- Transformer: a network where every piece of input can look at every other piece and decide what matters.
- Token: one small piece of the input, turned into a list of numbers.
- Fine-tuning: start from a network trained on something else, then keep training on your task.
The problem: a green plant on green grass
You can spot a dandelion by its shape: broad, jagged leaves fanning out from one point. The color is almost the same as the grass, so any rule based on color alone fails.
What it means
The input is a plain RGB photo taken looking straight down. There is no depth and no special light band. The network has to learn shape, leaf arrangement and texture, not just "green".
Design choice it forced
- A learned network (a transformer) instead of hand-written color rules.
- Training tiles that include plain turf, and other broadleaf weeds as "not a dandelion" examples (137 empty tiles in training).
Still a weak spot
On 8 crops of other look-alike weeds, the final model still drew 3 masks. Shape helps a lot but does not solve look-alikes completely.
Yellow = pixels the color rule calls "plant". Move the slider. Before it catches most of the rosette, it also flags a large share of the grass. The model's mask (cyan) follows the leaf shape instead.
Tiling: cut the frame into overlapping squares
The segmenter only accepts a 480 × 480 image. Squeezing a whole 4032 px photo down to that size would shrink a plant to a few dozen pixels, so the frame is cut into 640 × 640 tiles that step 480 px at a time.
What it does
- Slides a 640 px window across the frame, 480 px per step, so neighbouring tiles share a 160 px strip.
- The last row and column are pushed inward so the edge of the photo is still covered. Tiny photos are mirror-padded.
- Each tile remembers its top-left corner
(x, y)so its masks can be put back in the right place later. - All tiles of a frame go through the network together as one batch.
Why overlap
A plant sitting on a tile border would be cut in half. With a 160 px overlap, most plants appear whole in at least one tile. The price is that some plants appear twice, which is why section 10 exists.
Problem it solves
Small rosettes and thin leaf tips would vanish if the whole frame were shrunk. Tiling keeps the plant about 6× larger in pixels, as the comparison shows.
Patches become tokens
A transformer does not read pixels one by one. It cuts the 480 px tile into 12 × 12 pixel patches, giving a 40 × 40 grid of 1,600 patches, and turns each patch into a list of 384 numbers called a token.
What it does
- The patch embedding is one learned layer that maps each 12×12×3 patch to a 384-number vector.
- A position embedding is added to each token so the network knows where the patch came from. Without it, the tokens would be an unordered bag.
Why patches
Attention compares every token with every other token, so the cost grows with the square of the token count. 1,600 tokens is manageable. 230,400 pixels would not be. A 12 px patch is still small enough to describe a piece of a leaf edge.
Like cutting a photo into a jigsaw of small squares and writing a short description on the back of each piece, plus its row and column.
The encoder: patches look at each other
The encoder is 12 transformer blocks stacked. In each block, every token asks "which other patches help me understand what I am?" and mixes in information from them. This mixing is called self-attention.
What it does
A patch that shows a bit of jagged green edge cannot tell, alone, whether it is a dandelion leaf or a grass blade. After attention, it also carries information from the patches around it: are there broad leaves radiating from a center nearby? That context is what tells a rosette from turf.
Why "windowed" DINOv2
- DINOv2-style backbone: this design comes from a family that learned strong general visual features from a huge image set without labels, so it already "knows" textures and shapes.
- Windowed attention: in most blocks, tokens only attend within their window of the grid. That is much cheaper. Other blocks mix across windows so information still travels across the whole tile.
Problem it solves
Local shape is ambiguous on a lawn. Attention lets each piece use the whole neighbourhood, and windowing keeps the cost low enough for many tiles per frame.
Each cell here stands for a group of patches. Line thickness shows how much the chosen cell borrows from another. In this cartoon a leaf cell pays most attention to other leaf cells. In a windowed block, it can only see cells inside its own window (shaded).
Feature taps and the projector
Outputs are copied out of the encoder after blocks 3, 6, 9 and 12, joined, and compressed by a small projector into one feature map with 256 channels. That map is the "memory" the decoder reads from.
What it does
Early blocks are good at fine detail like edges. Late blocks are good at "what is this object". Tapping several depths gives the decoder both, which matters for drawing exact leaf outlines.
Why one scale (P4)
RF-DETR Seg-Small uses a single projected scale instead of a deep pyramid. It is faster and was enough here because tiling already keeps plants large.
Why 256 channels
It is the decoder's working width. Shrinking 384 to 256 makes every later step cheaper.
Darker blocks are the four tapped ones. Their outputs are combined into the memory F.
Object queries: 100 empty slots
The decoder works with up to 100 object queries. Think of each as a slot that asks: "is there a plant here for me to describe?" Each slot ends up as one possible plant, or as "nothing".
What it does
RF-DETR is two-stage: it first scores every location in the memory and starts the queries at the top-scoring spots (top-K), instead of starting all slots blank. Each query carries a rough box from the start.
Why queries instead of a grid of detectors
Older detectors made thousands of guesses and needed a cleanup step (NMS) to delete repeats. DETR-style models learn to give each plant to one query, so the network itself avoids most duplicates.
Problem it solves
Plants vary in count per tile, from zero to many. A fixed set of slots that can each say "no object" handles any count up to 100.
Left: 100 query slots. Right: the tile with each active query's box.
The decoder: four rounds of refinement
Each of the 4 decoder layers does three things to every query: compare notes with the other queries, look back at the image memory, then update its description and box.
The three steps in each layer
- Self-attention between queries (8 heads). Queries see each other. Two queries on the same plant can notice this and one backs off. This is how duplicates are avoided.
- Deformable cross-attention (16 heads, 2 sampling points). Each query looks at only a few learned points of the memory near its box, instead of all 1,600 positions. Fast, and it trains quickly.
- Feed-forward + box refinement. A small MLP updates the query, and the box is nudged a little closer to the plant.
Why repeat 4 times
Each round starts from a better box, so it looks at better places in the image. Four rounds is a speed/accuracy balance chosen by the RF-DETR Small design.
Three heads: score, box, mask
Every finished query is read by three small output parts called heads. One gives a score, one a box, and one a mask at a quarter of the input size (120 × 120), later resized to the tile.
How the mask head works
Each pixel of a 120 × 120 grid gets its own feature vector (a pixel embedding). The mask for a query is the dot product of the query vector with every pixel vector: pixels that "match" the query score high. Above zero means "in the mask".
Why a mask and not just a box
A box cannot show the gaps between leaves or their jagged edges. A robot that sprays or cuts needs to know where the leaves really are.
Why quarter resolution
Masks at full size cost 16× more memory. Quarter size keeps leaf shapes and is upscaled afterwards.
About the score
The score ranks proposals. It is not a calibrated probability. Videos and the gallery keep proposals with score ≥ 0.20. AP evaluation used 0.05 so that ranking quality could be measured.
Note on "pure transformer"
The backbone and decoder are transformers. Parts of the projector and mask head are convolutional, so "transformer-based" is the accurate description.
The mask reveal uses the model's real mask for this tile. The five proposals below it are example scores to show how the threshold works.
How the segmenter learned
Nothing in this section runs on the robot. It explains where the labels came from, what the network was punished for, and how training was steered.
Labels: boxes turned into masks
The public Weed-AI set only has boxes around dandelions. Each box was given to SAM 2 (a general "segment anything" model) as a prompt, and it drew a mask. These are pseudo masks: made by a model, not traced by a person. Your approved lawn masks from IMG_4847 and IMG_4848 were added.
Split by photo, before cutting tiles
60 / 9 / 9 public photos for train / validation / test. Lawn video IMG_4849 was held out. If two tiles of the same photo landed in train and test, the test would look easier than it is.
Loss: what counts as a mistake
- Matching first. Predictions are paired one-to-one with real plants. Unpaired predictions are taught to say "no object".
- Class loss (IA-BCE): score should be high for a matched query, and higher when its box overlaps the plant better.
- Box loss (L1 + GIoU): box edges in the right place, and good overlap.
- Mask loss: 5 × cross-entropy + 5 × Dice. Cross-entropy checks each pixel. Dice checks overall overlap, which stops a small plant from being ignored.
- 13 query groups during training only. Extra copies of the queries get extra matches, which speeds up learning. At inference only one group runs.
Steering choices
- Start from a COCO segmentation checkpoint, train on public tiles (stopped at epoch 14), then adapt to lawn tiles (best at epoch 3).
- Backbone learns 5× slower (2e-5 vs 1e-4), and early blocks slower still (layer decay 0.8). The new head changes fast, the pretrained knowledge changes gently.
- EMA weights (a smoothed running average of the weights) were kept, which gives steadier results than the last raw step.
- BF16 mixed precision, gradient checkpointing, batch 2 × 2 accumulation. These fit training on a 16 GB GPU.
Right panel shows the trained model's mask on this held-out tile, standing in for what a box-prompted mask looks like.
Gray: noisy weights step by step. Yellow: the running average. Higher decay is smoother but reacts more slowly. The project used 0.993.
Approximate: encoder rate 2e-5 shrunk by 0.8 for each block further from the output. Heads and decoder use 1e-4.
Cross-tile cleanup
Because tiles overlap, the same plant can come back two or three times, sometimes cut in half. Plain code (no learning) turns the per-tile answers into one clean list for the whole frame.
The five steps
- Shift each mask by its tile's corner
(x, y)into full-frame coordinates. - Drop masks under 180 pixels. They are almost always noise.
- Rank by
score0.2 × area0.7. Area counts far more than score, so a complete mask beats a half mask that happens to score slightly higher. - Merge: two masks are the same plant if their overlap (IoU) is ≥ 0.30, or if one is ≥ 70% inside the other and their centers are close. Keep the higher-ranked one.
- Open: a 3 × 3 morphological opening removes stray pixels. Tiny loose fragments are dropped.
Why favour area
The cut-in-half copy from a border tile can have a decent score. Ranking mostly by area keeps the whole plant.
Trade-off
These rules make the output cleaner, but two plants touching each other can be merged into one, and a thin real leaf can be trimmed.
Why the growth point needs its own network
The simple answer would be "take the middle of the mask". But one long leaf drags the middle away from the crown, and the crown itself is often hidden or sits in a gap with no mask pixels.
Three possible "centers"
- Mask centroid: the average position of all mask pixels. Easy, but pulled by big leaves.
- Box center: the middle of the rectangle. Pulled by whichever leaf reaches furthest.
- Learned growth point (this project): a network looks at the plant and predicts where the leaves meet.
Why separate weights
The center network gets only an RGB crop. It never sees the mask. That makes it a fair test: does learned geometry beat the box center? It did, 14.2 px vs 26.6 px error on held-out human-marked stems.
Still coupled
The crop is placed using the segmenter's box. A bad mask gives a bad crop, so errors can still flow from stage I into stage II.
The close-up crop
For each plant, a square crop 1.45× the longer side of its box is cut from the frame, limited to between 96 and 640 px, and resized to 224 × 224.
Why 1.45×
A tight crop can cut off leaf tips, and the leaf directions are the main clue to where the crown is. A margin of about 22% on each side keeps the whole rosette plus a little grass for context.
Why square, why 224
The DINOv2 ViT-S/14 was pretrained on 224 px squares. 224 ÷ 14 = 16, a clean 16 × 16 token grid.
Why mirror padding
Near the frame edge the crop runs off the image. Filling with black would add a fake hard edge the network never saw in training. Mirroring the image keeps the texture natural.
Why the 96–640 clip
Very small boxes would be blown up into mush. Very large ones would shrink leaves too much.
A DINOv2 ViT-S/14 that is mostly frozen
The crop is cut into 14 × 14 px patches (16 × 16 = 256 tokens of 384 numbers) and sent through 12 transformer blocks. Blocks 1–10 are frozen. Only blocks 11–12, the final norm and the new head are trained.
Why freeze most of it
There were only ~900 human-marked stem points for training. A 22-million-parameter network trained fully on that little data would memorise it. Freezing keeps DINOv2's general visual knowledge and lets only the top layers adapt.
Why DINOv2 weights
DINOv2 learned from 142 million images without labels. Its tokens already describe object parts and their layout well, which is what "where do these leaves meet" needs.
Learning rates
Head 1e-4, last two blocks 1e-5 (10× slower), cosine schedule.
The [CLS] token
ViTs add one extra summary token. It is thrown away here, because the task needs one answer per position, not one per image.
Parameter counts are approximate (ViT-S ≈ 1.8 M per block). The point is the ratio: fewer trainable weights per training example means less memorisation.
Heatmap and soft-argmax
A tiny head (LayerNorm → 384→128 → GELU → 128→1) gives each of the 256 tokens one score: "how likely is the growth point in my cell?" A softmax turns scores into weights, and the point is the weighted average of cell centers.
The math, in words
p_i = softmax(z)_i: scores become weights that add up to 1.ĉ = Σ p_i · g_i: the point is the average of the 256 cell centers, weighted byp.(u, v) = crop corner + crop side × ĉ: back to frame pixels.
Why soft-argmax and not argmax
Argmax picks the single best cell, so the answer jumps in steps of 14 px and gives no gradient to learn from. Soft-argmax can land between cells and is smooth, so training can push it a little at a time.
Why one shared head per token
The same tiny MLP is applied to every token. It has about 50 thousand weights, so it cannot memorise much. The ViT tokens do the hard work.
Watch out
If two places score high, the average can land in between, on a spot neither one chose. The code also saves the heatmap's entropy, a measure of how spread out it is, which can flag unsure points.
How the center model learned
Training needs to know where real crowns are. Real dandelion stem labels were rare, so two kinds of labels were mixed, and the more trustworthy kind counted almost 3× more.
Two kinds of labels
- Human-marked stems from the CropAndWeed dataset: 1,130 points on related rosette-like plants (904 train, 226 validation). Weight 1.0.
- Geometric proxies for dandelions: 1,444 points estimated from mask shape. The code tries to intersect the main leaf directions, and falls back to a geometric median. Weight 0.35. These are guesses, not observed stems.
The loss
- The label becomes a soft Gaussian blob on the 16 × 16 grid (σ = 0.055 of the crop). Cross-entropy pulls the heatmap toward that blob.
- 20 × Smooth-L1 on the final point pulls the answer itself toward the label.
- Each example's loss is multiplied by its label weight.
Why both loss terms
The heatmap term gives a stable signal everywhere on the grid. The point term cares about the exact final answer. Together they train faster and end up more precise than either alone.
Augmentation
Random flips, 90° turns and small brightness changes. A plant has no "up" when seen from above, so these create new valid examples. The label point is moved with the image.
The yellow blob is the Gaussian training target around the label. If the image is flipped but the label is not, the network would be taught the wrong answer.
What comes out for each frame
Every kept plant gets a mask, a score, a box and a growth point. They are saved in formats other software can read.
Files
- Overlay video or image: the mask painted in cyan, the point as a red cross.
- CSV: one row per plant with score, box and point.
- JSON Lines: masks stored as COCO RLE (lossless run-length encoding), plus the tile origin.
- 16-bit PNG (still images): each pixel holds its plant's ID number.
No tracking yet
Each frame is processed on its own. "Plant 7" in one frame can be "plant 9" in the next. Counts across a video are appearances, not unique plants.
Where the cross can land
The cross may sit outside the cyan area. That is allowed: the crown can be in the gap between leaves.
Reading the scores
Mask quality is measured with IoU: the overlap area divided by the combined area. AP50 counts a mask as correct when IoU ≥ 0.5. AP50:95 averages over ten stricter thresholds, so it rewards exact outlines.
| Held-out check | AP50:95 | AP50 | Reference masks |
|---|---|---|---|
| Weed-AI test · 81 tiles | 0.521 | 0.804 | SAM 2 pseudo masks |
| Lawn IMG_4849 · 38 tiles | 0.148 | 0.334 | approved drafts, not complete |
| AgriStress · 5 tiles, 1 photo | 0.151 | 0.514 | very small check |
| Growth point, mean error (px in 224 crop) | Model | Box center |
|---|---|---|
| Human stems, related plants (226) | 14.22 | 26.55 |
| Public dandelion proxies (92) | 7.05 | 8.75 |
| Held-out lawn proxies (100) | 23.46 | 24.86 |
How to read them
The scores measure agreement with references that are themselves imperfect. The lawn AP is low partly because the lawn reference masks are drafts and miss plants. Error is in crop pixels, not millimetres.
Each chip is one IoU threshold from 0.50 to 0.95. AP50 only uses the first chip. AP50:95 averages all ten, so a slightly-off mask loses points there even when it passes AP50.
Speed, and getting the point onto the robot
On an RTX 4060 Ti (16 GB) a full frame took 220–278 ms on average, about 3.6–4.5 frames per second. The videos play at 15 fps only because frames were sampled that way.
Pixel to 3-D point (planned with the D435i)
- Align depth to the RGB image and check the depth near
(u, v)is valid. X = (u − cx)·Z / fx,Y = (v − cy)·Z / fy, using the camera's real calibration.- Transform into the robot frame with the calibrated camera pose, then track the plant over several frames before acting.
Two separate speed limits
Blur: how far the ground moves during one exposure. Latency: how far the robot travels between processed frames. Both have to be checked on the real robot.
Not yet done
Live camera integration, depth checks, tracking and closed-loop robot trials have not been run. The 2-D point is not a tool-contact point or a safety decision.
blur ≈ fx × speed × exposure ÷ height. This ignores rolling-shutter skew and rotation.
What is not proven yet
- Labels are approximate. Public masks come from SAM 2, lawn masks are approved drafts, and most dandelion center labels are geometric guesses.
- Look-alike weeds still get masks sometimes (3 of 8 check crops).
- No tracking across frames, so no stable plant identity yet.
- No depth or robot loop has been tested. Accuracy in millimetres is unknown.
- Next best test: fully hand-traced masks and measured crown points on new lawn scenes, with an explicit "other weed" class.
The paper figure, now that you know the parts
Every box in the figure is one of the sections above. Scroll sideways on a small screen.
Glossary
A way for each token to take a weighted mix of information from other tokens.
The part of a network that turns an image into features. Here: windowed DINOv2-style ViT-S, and DINOv2 ViT-S/14.
Saved model weights from a moment in training.
A large public dataset of everyday objects. Used for the segmenter's starting weights.
A mask loss based on overlap. Good for small objects.
Exponential moving average of the weights: a smoothed copy that changes slowly.
One full pass through the training data.
Intersection over union. Overlap area ÷ combined area. 1.0 is perfect.
Shrink a mask then grow it back. Removes specks thinner than the brush.
A mask made by another model, not drawn by a person.
A learned slot in the decoder that becomes one predicted object or "nothing".
Run-length encoding: a compact, lossless way to store a mask.
A smooth, averaged version of "pick the highest cell".
One patch of the image turned into a vector of numbers.