AI for engineering leaders

Roles covered: engineering director or VP, platform and infrastructure leads, engineering managers. Functions: engineering, with shared responsibility for systems used by support, operations, sales, and finance. Linked workflows: incident triage, support replies, onboarding guidance, knowledge retrieval, order intake.

The decisions you own

Your decision is whether the organization can operate a useful system with acceptable reliability, cost, and responsibility. Model selection is part of that decision. Data access, evaluation, integration, and recovery are equally concrete engineering work.

Engineering leaders hold two jobs that are easy to blur, and this chapter keeps them apart:

1. Building and operating AI systems other departments use. Support wants drafted replies; operations wants purchase orders extracted; a services team wants permission-filtered search. You decide what gets built, bought, or declined, what quality gate a release must pass, who funds the continuing work, and how model or provider changes reach production.
2. Adopting AI in your own team's work. Coding assistants, incident briefs, and review aids change how engineers spend their time. You decide where they are permitted, how their effect is measured, and how the team keeps the skills that assistance can erode.

In the first job, the measure that looks like progress but is not: the number of AI features shipped. A feature without an evaluation set, an owner for its sources, and a working fallback is a liability that has not yet surfaced. In the second job, the misleading measure is self-reported speed. The research section shows why perceived and measured effects can diverge.

Map the work you manage

The table below separates tasks by consequence and proposes a starting authority. The authority levels are this guide's framework from assistance, approval, and autonomy, not an industry standard.

Capability verbs in the table follow the capabilities chapter; starting authority uses the arrangements in assistance, approval, and autonomy.

Before any row becomes a project, identify the simpler baseline: search, rules, templates, an existing product feature, deploy annotations on dashboards, or conventional predictive methods. Google's machine-learning engineering guidance recommends establishing metrics and a simple initial system before adding learned components. Rules of Machine Learning A model-based system should earn its additional operational complexity against that baseline, measured on the same cases.

Turn the business request into a system boundary

Ask for the unit of work, acceptable output, consequences of error, permitted inputs, and authorized actions. "An agent that handles onboarding" leaves too much unspecified. "Draft setup guidance from approved documentation and verified account state, with no permission to change configuration" defines something a team can evaluate. Each workflow chapter in this guide opens with that kind of boundary; use them as models for the intake conversation.

Separate proposal from execution

A model may propose a structured action. Deterministic application code should check identity, permissions, required fields, and business constraints before execution. Customer, document, and log text is data, not permission to expand authority. The retrieval chapter cites the prompt-injection definition and mitigations; they justify designing the boundary into code rather than into a prompt.

For actions with side effects, plan retries, duplicate prevention, partial failure, and audit records. Writes need idempotent submission: an uncertain response does not establish that the original operation failed. The order-intake chapter cites the engineering source and works this through for purchase orders.

Who decides what

Name these responsibilities before a pilot grows into a production dependency. The table is a proposed allocation; adapt it to your organization.

Two lines deserve emphasis. Engineering can enforce permissions; it cannot decide whether a support customer received a useful answer. And a department that asks for an AI system must fund the curation and review it depends on, or those costs will be absorbed invisibly until quality degrades.

Build evaluation into the delivery process

This is the platform investment that makes every later request cheaper to decide. Maintain representative cases, difficult cases, and cases outside the allowed scope. Hold some cases back from tuning. Compare the complete system against the baseline, not a prompt in isolation. Record model identifier, prompt revision, retrieval method, source snapshot, tool behavior, latency, and cost for each evaluated configuration.

A release should meet agreed business quality requirements and technical operating limits. Re-run the relevant evaluations when models, prompts, sources, or permissions change. Do not assume a provider upgrade preserves accepted behavior; the model-selection chapter describes when to revisit a choice. Published specifications change within weeks, so recheck them at the provider's documentation rather than reusing an earlier reading.

Proposed release gate for any AI feature built by your team:

• A written system boundary: unit of work, inputs, permitted actions, excluded requests.
• An evaluation set with a held-out portion, including boundary and adversarial cases.
• Results reported with denominators, separated into ordinary and challenge cases.
• Access filtering tested with realistic test accounts, not administrator credentials.
• Configuration versioned; a rollback path tested; a manual fallback documented.
• Logging and retention agreed with the data owner and security.
• Cost per acceptable outcome measured, including retries and review.
• Named owners for sources, review, incidents, and the kill switch.

Costs that belong in the plan

Measure cost per acceptable completed outcome. A lower price per model call can be offset by more retries or additional review. Pair cost measurements with latency and quality so optimization stays tied to the product promise. Each workflow chapter shows an illustrative split of its recurring budget; map those buckets onto the components above when you build a plan. The economics chapter explains why capacity released is not the same as cash saved, and why accounting classification belongs to finance.

Plan a first quarter

The following is a fictional planning example. Quarrowby Software, its teams, requests, and numbers are invented. Nothing here describes an observed organization.

Quarrowby's engineering director receives three internal requests in the same month: support wants drafted replies, the on-call rotation wants incident briefs, and customer success wants onboarding guidance. Each request, built separately, would need its own document connector, its own access logic, and its own ad hoc test set.

Weeks 1–3: measure before selecting anything. The director asks each requesting team for a sample of work units and a baseline measure of effort, using the definitions in the relevant workflow chapter. Engineering inventories the sources the three requests share: product documentation, runbooks, and account context. Two of the three depend on account-scoped data.

Weeks 4–8: build the shared layer first. Rather than three one-off builds, the team builds a permission-filtered retrieval service with provenance on every passage, and an evaluation harness that replays versioned cases against a recorded configuration. The support request is the first consumer because its source documentation is the most maintained and its sending step already requires an agent.

Weeks 9–12: offline replay, then a decision. Support drafts run offline against held-out cases. Incident briefs run in shadow mode during real incidents without being shown to responders, as the incident-triage chapter proposes. Onboarding waits: its entitlement data lives in a system whose records the team has not yet verified.

At the end of the quarter, the director reports what each configuration achieved against its agreed gate, what the shared layer cost, and which request is deferred and why. A decision to simplify or defer is a valid engineering outcome.

Counterexample. Suppose instead the director approves a request from a platform team to let an agent restart services and apply configuration changes during incidents, because a demonstration looked impressive. No evaluation harness exists; no replay set covers misleading alerts or unrelated deploys; the command structure of the incident process has not been consulted. This should not proceed. Google's incident-response guidance has responders assess impact before choosing a mitigation such as a rollback, SRE Workbook, Incident Response and an agent with write access would act outside that process. The right first step is the read-only brief, evaluated against replayed incidents, with any action authority considered only after the evidence exists.

Adopting AI in your own team's work

The second job has its own evidence, and it is contested. Two recent sources measure different things in different populations:

Sources: Becker, Rush, Barnes, and Rein, arXiv 2507.09089; DORA 2025 report.

These findings are not contradictory so much as non-comparable: one is a small randomized trial in a specific setting, the other a large observational survey. Together they support two practical points. First, developers' perception of their own speed may not match measured time, so self-report alone is a weak basis for a tooling decision. Second, throughput and stability can move in different directions, so measure both.

Proposed practices for your own team:

• Measure on your work. Compare assisted and unassisted work on comparable tasks, recording elapsed time, review time, rework, and escaped defects, not only lines or pull requests produced.
• Keep review responsibility unchanged. Generated code passes the same review and tests as any other code. A pull-request summary helps a reviewer orient; the reviewer still reads the diff.
• Track stability alongside speed. DORA measures instability through change fail rate and rework rate, and throughput through lead time, deployment frequency, and failed-deployment recovery time. Watch whether instability moves when adoption does.
• Decide where code and secrets go. An editor assistant sends surrounding code, configuration, and sometimes terminal output to a provider as context. Before adoption, record which repositories may be used with which tools, whether customer contracts or data-handling terms restrict sending customer code or configuration to a third party, how secrets are kept out of context (excluded files, secret scanning before suggestions are accepted), and what the provider retains. Decide who checks license and provenance questions for generated code, and keep that decision in the same review policy as other third-party code. These are proposed practices; your security, legal, and procurement owners interpret the terms.
• Protect practice. Automation bias is a named risk (see the authority chapter). On-call engineers should keep investigating incidents without a brief; junior engineers need work where they build understanding, not only accept suggestions.

Measure the right things

Your team: work, judgment, and skills

The platform job shifts some engineering effort toward work that may be unfamiliar: building evaluation sets with domain experts, writing permission tests, tracing a bad answer back through retrieval to a source, and reading provider change notes. The judgment that remains is deciding whether a system is good enough to release and when to pull it.

Skills to plan for, as proposed development areas rather than observed trends:

• Evaluation design: representative and adversarial cases, held-out sets, denominators, and leakage prevention.
• Security for model-based systems: prompt injection, permission-aware retrieval, and least-privilege tool design.
• Cost engineering: measuring spending per acceptable outcome rather than per call.
• Working with domain owners: translating "good answer" into a rubric reviewers can apply consistently.

Whether these become new roles, responsibilities within existing roles, or work bought from vendors is a decision to plan and measure, not a trend to assume. Neuronify's AI skills glossary and hiring index describe what NYC AI companies post in job listings; they do not describe how your team's work will change.

Who to bring in. Your team is the builder, so the outside roles differ. A first pilot needs owners from the requesting department to define a useful output and set acceptance thresholds, source owners to label content, and security or risk to approve the access design. If the shared retrieval layer or evaluation harness exceeds your team's capacity, outside builders or integrators can extend it under your release process, and an evaluator independent of the builders can score the replay. Keep in-house the harness, release and rollback, and permission enforcement. See Staffing an AI pilot and Choosing and working with an AI partner.

Change management and governance

• Publish the intake path. Other departments need to know how to request an AI system and what they must bring: a unit of work, a baseline measure, a source owner, and a reviewer.
• Make declining legitimate. A written reason to defer, such as unreliable source data or unenforceable access boundaries, prevents repeated escalations.
• Keep the incident chain of command. A generated summary can help responders assemble context; it should not silently gain authority to deploy a fix.
• Give reviewers a route to report bad outputs without making them investigate on their own time, and feed confirmed failures into the evaluation set.
• Do not set usage targets. Targets for assistant adoption can reward unsafe reliance. Set targets on outcomes and quality.
• Treat provider changes as releases. A model version change goes through the same evaluation gate as a code change.

Warning signs

Questions to ask vendors and internal teams

• What exactly is the unit of work, and what does the system do when it cannot complete it?
• Where are permissions enforced: before retrieval, after generation, or only in the prompt?
• Which configuration produced the evaluation results you are showing, and can we rerun them on our cases?
• How are model version changes announced, and can we pin a version while we re-evaluate?
• What does the system log, for how long, and who can read it?
• What does cost look like per completed outcome, including retries and failed attempts?
• What is the manual fallback, and who can switch the system off?

When to wait

Defer a build when the requesting department cannot name a source owner or a reviewer, when access boundaries in the underlying data are unreliable, when eligible volume is too low to repay integration and evaluation, or when a simpler baseline has not been tried. Defer broad adoption of coding assistance when you have no way to measure task time, review load, and stability on your own work. In each case, the simpler investment, such as better search, documentation ownership, dashboard annotations, or a measured trial on a single team, is useful regardless of the later decision.

What external research establishes

For your analyst

The METR trial and the DORA survey, summarized above, are the attributable sources this chapter uses on developer adoption; they measure different populations with different designs, and neither establishes the effect of AI assistance on your team. Google's SRE and machine-learning guidance support specific design points here: incident coordination and simple baselines. The sources on untrusted input, retrieval access, and idempotent retries are cited in the linked retrieval, authority, and order-intake chapters. None of them validates the platform approach, release gate, or responsibility allocation proposed here. No engineering organization was interviewed and no deployment was observed for this chapter.

Related reading

Incident triage, support replies, model selection, retrieval, pilot design, support leaders, and finance leaders. Prepare a pilot with the pilot kit.