Computer vision in operations

Computer vision turns images into decisions an operation can use: whether a part is defective, how many cartons are on a pallet, where a label sits, what a gauge reads. The decision this chapter supports comes before any model is chosen: what image data, labels, and quality requirements would make a vision system trustworthy for this task, and is vision the right tool at all?

In this guide's view, the hardest work in a vision project sits outside the model. Cameras, lighting, label definitions, and the cases a team chooses to photograph decide what any model can learn and what an evaluation can show. A model that performs well on a clean sample can fail when a supplier changes packaging or a camera is bumped a few degrees.

This chapter is written for operators and implementers. It draws on the sources cited; the data requirements, evaluation design, and controls are this guide's proposed practice. The worked numbers are invented for illustration. No vision system has been evaluated for this chapter. A dedicated workflow chapter on manufacturing visual inspection is planned but not yet written.

Name the task before choosing a method

"Vision" covers several tasks with different outputs and different checks. Writing the task down precisely is the first requirement, because each output is checked differently.

Reading text in images overlaps with document processing, which separates OCR, layout, and extraction. Microsoft Research In this guide's view, the same distinctions apply to labels and signs.

Multimodal models that accept an image and a question can produce fluent answers to open-ended prompts. That flexibility is useful for exploration and harder to evaluate: a free-text answer can mix a correct observation with an unsupported one, and its wording can vary between runs. For a repeatable operational decision, prefer a constrained output (a label from a fixed list, a box, a number) that can be scored against a reference. If a team uses a multimodal model, treat its output like any generated text: check claims against the image, constrain the answer format, and keep the model-selection discipline of recording the exact configuration.

A single project can combine tasks. A pallet check might detect cartons, read each label, and count units, and each step has its own failure modes. Evaluate each step and the combined result.

Image data requirements are the core of the project

For your technical lead

The images a system sees in production define what it must handle. The proposed practice is to write image requirements before collecting a training set, the same way the model-selection chapter writes task requirements before choosing candidates.

Capture conditions

Record, for each camera position:

• Lighting: source, direction, and how it changes across shifts, seasons, and windows. Glare and shadow can hide or imitate defects.
• Geometry: distance, angle, and field of view. A defect seen face-on may be invisible at a slant.
• Resolution: the smallest feature the decision depends on, in pixels. If a 0.5 mm crack must be found, the image must resolve it with margin.
• Motion: line speed and exposure time. Blur removes fine detail.
• Occlusion: hands, packaging, other parts, condensation, or dust that can cover the region of interest.
• Background and fixtures: what else appears in frame and whether it changes.

Fixing capture conditions can be cheaper than modeling around them. Constant lighting, a fixed mount, and a clear background can turn a hard vision problem into a simple one. In this guide's terms, capture design is part of the workflow, not a preliminary.

Camera placement and drift

Cameras move, lenses collect dirt, and lighting ages. Record a reference image for each position and compare periodically: if the view, brightness, or focus has shifted, results may change without any change to the model. Assign an owner for camera maintenance, as the support-reply chapter assigns an owner for knowledge articles.

Label definitions and agreement

A label is a decision someone made about an image. Before labeling at scale:

• Write the definition of each class with example images at the boundary: what counts as a scratch, how small a dent matters, whether a smudge is a defect.
• Have two or more people label the same sample independently and compare. Where they disagree, refine the definition or record the case as genuinely ambiguous. A model cannot be more consistent than the definition it learns from.
• Record who labeled what, under which definition version.

Label errors are not limited to small projects. Northcutt, Athalye, and Mueller estimate label errors in at least 6% of the ImageNet validation set and at least 3.3% on average across ten widely used benchmark test sets, and show that such errors can change which model appears best. Northcutt et al., 2021 Those are public benchmarks, not operational data, but they support a practical rule: audit a sample of your own labels before trusting an evaluation built on them.

Class balance and rare events

Defects that matter may be rare. A training set with thousands of acceptable parts and a handful of defective ones teaches a model little about defects, and an evaluation with few defects says little about how many it misses. Options include deliberately collecting defect examples (from returns, quality holds, or staged samples), labeling defect types separately, and reporting results for each type rather than only an overall figure.

Representative sampling

Collect images across the conditions production will see: every shift, site, line, product variant, supplier, and season that matters. Record where each image came from so evaluation can be sliced the same way. Hold back data from at least one condition (a site, a month, a supplier) as a test of how the system handles change.

Change in the input is the main operational risk

A model learns from the images it was given. When products, suppliers, packaging, cameras, or lighting change, the images change, and performance can change with them.

Recht, Roelofs, Schmidt, and Shankar built new test sets for two benchmarks by closely following the original collection process and found accuracy drops of 3% to 15% on CIFAR-10 and 11% to 14% on ImageNet across the models they evaluated; they attribute the drops to the models' difficulty generalizing to slightly harder images, not to overfitting from reusing the original test sets. Recht et al., 2019 That study used research benchmarks and models of its time. It supports a narrower point for operations: even a careful attempt to reproduce the same kind of data can produce a meaningful drop, so a change that is not careful, such as a new supplier's packaging, deserves its own test.

Proposed controls:

• Keep a list of changes that should trigger re-evaluation: new product variant, new supplier or packaging, camera replacement or move, lighting change, new site.
• Monitor the distribution of model outputs and confidence over time. A sudden rise in one class, or in uncertain cases, can signal a change in the input before quality data arrives.
• Sample production images for human review on a schedule, including images the system marked as acceptable, so misses are found.

Design the decision, not only the detector

A vision output feeds a decision: reject a part, hold a pallet, alert a person, update a count. The design questions are about that decision.

What does each error cost?

A missed defect (false negative) and a false alarm (false positive) have different costs. A missed crack in a safety-relevant part can be severe; a false alarm costs a reviewer's minute and perhaps line time. Write both costs down before choosing a threshold.

Base rates decide the review load

The share of items that are actually defective changes what a given error rate means. An illustrative shift, with invented numbers:

• 5,000 items are inspected, and 1% (50) are defective.
• The system flags 90% of defective items (45) and misses 5.
• It also flags 2% of the 4,950 acceptable items: 99 false alarms.

Reviewers see 144 flagged items, of which 45 are real: about 31% of flags are defects. Five defects pass unflagged. These rates are invented to show the arithmetic, not measured performance. If defects become rarer while the error rates stay the same, the share of flags that are real falls further, and reviewers may start to treat flags as noise. That is alert fatigue, the opposite failure to over-reliance: the reviewer under-trusts the system instead of over-trusting it. This is our editorial judgment, not a finding we cite; measure how often reviewers dismiss flags that later prove real.

Thresholds and confidence

Classification and detection models output a score, and a threshold turns it into a decision. Moving the threshold trades misses for false alarms. Choose it from the cost of each error and the review capacity, and record it as part of the configuration.

Do not read a score as a probability without checking. Guo and colleagues found that the modern neural networks they studied were poorly calibrated, with confidence that did not match how often predictions were correct, and that a simple post-processing step improved calibration on most of the image and document classification datasets they tested. Guo et al., 2017 Build a reliability table on local data, as the document-extraction chapter proposes for field confidence, before using a score to decide which items skip review.

Route uncertain and high-consequence cases to people

Use the human-review playbook's fixed routing order (rules, observable conditions, consequence, then a tested confidence estimate): items in a safety-relevant class, items near the threshold, and images the system cannot assess (blurred, obstructed, out of frame) go to a person. A clear "cannot assess" outcome is better than a forced guess.

Evaluate on the conditions that matter

For your analyst

Structure the evaluation

For each class or defect type, score a held-out set the model never saw during development. For classification, count true positives, false positives, false negatives, and true negatives; for detection and segmentation, count matched, missed, and spurious boxes or regions at a stated overlap criterion. Report results:

• By defect type and severity, not only overall.
• By condition: site, line, shift, product, supplier, camera.
• On a challenge set of deliberately difficult images (glare, partial occlusion, boundary cases), reported separately from a sample that reflects normal production.
• With denominators: "9 of 10 cracks found" says more than "90% recall," because it shows how little evidence 10 cracks provide.

For rare events, small samples prove little. The pilot-design playbook shows why observing no failures in a modest test does not establish a low failure rate; the same reasoning applies to "no missed defects in the trial."

Prevent leakage

Images from the same item, batch, or minute of production can be near-duplicates. If some go into training and others into testing, the test overstates performance. Split by item, batch, or time period, and keep a final test set untouched until the decision.

Compare against the current process

Measure the current inspection on the same images or the same period: what human inspectors catch and miss, at what effort. A vision system that matches human performance with less effort can be worthwhile; one that beats humans on a clean sample but misses a defect type inspectors catch may not be. Compare simpler options too, such as better lighting or a fixed gauge.

Operating the system

For your technical lead

Where inference runs

A model can run on a device near the camera (edge) or on remote servers (cloud). The choice follows from requirements:

Running models on your own infrastructure, at the edge or in a private environment, adds operating responsibilities. The open-weight and hosted deployment chapter lists the responsibilities of operating a model yourself; it does not address edge or on-device hardware.

Continuing cost

Budget for relabeling when definitions change, retraining when products or cameras change, camera maintenance, monitoring, and review labor. These recur. A project that budgets only for the first model will find its accuracy decaying as the operation changes around it. Use the economics chapter to keep setup and recurring costs separate.

People in the frame

If cameras capture workers, customers, or the public, the system processes images of people. Treat this as a gate before deployment: confirm with whoever owns privacy and employment obligations which notices, consents, retention limits, and access controls apply, and design capture to exclude people where the task does not require them. This guide does not set out legal requirements, which vary by jurisdiction.

Untrusted content in images

Images can contain text, and a multimodal model may read it. Text in an image, like text in a document, is content, not instruction. The prompt-injection risk described in the retrieval chapter applies when a model can act on what it reads. Keep vision outputs as proposals until checked, and do not give an image-reading system permission to act on instructions found in the image.

When something simpler is better

Vision is one way to sense the physical world. Before building it, check whether a simpler sensor or identifier answers the question:

• Identifiers: if the question is "which product is this?", a barcode or GS1 identifier answers it directly. GS1 defines the GTIN as an identifier a company can use to uniquely identify its trade items. GS1 In this guide's view, reading a code is more dependable than recognizing a product from its appearance, and it is easier to test.
• Physical sensors: weight, presence, temperature, or distance sensors can answer "is it there?" or "is it full?" without images.
• Fixed rules on a controlled image: with constant lighting and a fixed position, a simple measurement or threshold may suffice.
• Better capture: improved lighting or a fixture can remove the ambiguity a model was meant to resolve.

The Google guidance this guide cites elsewhere recommends starting with a simple system and clear metrics. Rules of Machine Learning Vision is worth its cost when the visual judgment is real, varied, and frequent enough to justify data collection, labeling, and maintenance.

Product images in the catalog enrichment workflow are a different use: images are attributes to manage and check, not inputs for an inspection decision, and any attribute read from an image needs the same source checks as one read from a spec sheet.

People and skills

• Operations and quality staff define what counts as a defect, set the cost of each error, and own the decision the system feeds. Their judgment is the reference standard.
• Labelers need written definitions, boundary examples, and feedback from agreement checks.
• Technicians maintain cameras, lighting, and reference images.
• Reviewers handle flagged and uncertain items and report patterns that suggest drift.
• Engineers manage data splits, configuration records, monitoring, and retraining.

Keep inspectors' skills in use. If people review only flagged items, they lose practice on the full range and may miss what the system misses. Rotating unflagged samples into review keeps both the people and the evaluation honest.

What external research establishes

For your analyst

Recht and colleagues show that classifiers can lose accuracy on new test sets built to match the original collection process, on two research benchmarks. Northcutt and colleagues show that label errors exist even in widely used benchmark test sets and can change model comparisons. Guo and colleagues show that confidence scores from the networks they studied were poorly calibrated without adjustment. Each finding comes from research datasets and models of its time; none measures performance on any operational inspection task. They support the practices above (testing under change, auditing labels, calibrating locally) rather than any expected accuracy.

Evidence still needed and related reading

This chapter has no operator account of an inspection project, no measured image set, and no evaluated model. Useful evidence would include an operator's record of capture conditions and their effect on results, label agreement rates on real defect definitions, and performance before and after a documented change in product or camera.

Related chapters: document processing and structured extraction, model selection, speech systems and voice interfaces, human review, pilot design, catalog enrichment, and, for sourcing decisions, build, buy, or configure and vendor diligence.