Engineering Productivity

AI Coding Assistants: Trust, Quality, and Governance

Published
Updated

Material update: 2 September 2026

Scale an AI coding assistant only after it improves the software delivery system, not merely the rate at which code appears. Accepted suggestions, generated lines and active licences are usage measures. They do not show whether valuable changes reach users sooner, fail less often or remain understandable to the people who must operate them.

This is not a case against AI-assisted development. The available evidence says that outcomes depend on the task, codebase, developer experience and surrounding engineering practices. DORA’s 2025 research characterises AI as an amplifier of an organisation’s existing strengths and weaknesses. A July 2025 METR experiment found a slowdown in a narrow population of experienced maintainers working in mature repositories they knew well. The defensible decision is therefore to adopt selectively, measure end-to-end outcomes and retain the ability to restrict uses that fail the quality test.

Define the outcome before measuring adoption

Choose one repeatable class of work first: fixing defects in a service, adding regression tests or migrating a routine API, for example. Establish a pre-adoption baseline for time from start to merge, review duration, revision rounds, post-release defects, reopened work and reversions. Without that reference, a dashboard can show activity while leaving the direction of change unknown.

The primary measure should connect speed with correctness. Time to a verified change is more useful than time to generated code: the change must pass tests and review, deploy successfully and survive an agreed observation period. Lead time alone can reward authors for shifting effort to reviewers and operators. Defect count alone can discourage small, frequent releases. Read the measures together and segment them by task type, repository risk and author experience.

Do not publish one productivity multiplier for the company. METR studied 16 developers completing 246 tasks with tools available in early 2025. The result is a valuable warning against relying on perception, but it is not a universal coefficient for every team, model or codebase.

Classify uses by repository and action risk

A binary policy of “AI allowed” or “AI prohibited” cannot guide real work. Build a matrix that combines repository class with the action being performed. Lower-risk uses may include explaining code, drafting documentation and generating synthetic test inputs that contain no sensitive data. Higher-risk uses include authentication, cryptography, billing, data migrations, infrastructure policy and regulated systems.

For each cell, specify approved tools, permissible data, required review depth, mandatory checks and the role that may accept an exception. Put the rule close to the developer workflow, ideally in repository guidance and pull-request checks, rather than only in a corporate policy portal.

Separate suggestions from actions. A completion tool proposing a local fragment has a different risk profile from an agent that edits many files, runs commands, accesses a network or opens a pull request. As capability grows, narrow the workspace, shorten credential lifetime, reduce network reach and make stop conditions explicit.

Require evidence, not an AI disclosure ritual

AI-assisted code needs a verifiable evidence package. The author remains accountable and must be able to explain the change’s assumptions, boundaries and side effects. A reviewer should see the behaviour being changed, the tests that support it and the conditions that would disprove the chosen solution.

The minimum package should follow risk. It may include a regression test that reproduces a defect, type and lint results, dependency and secret scans, security test output, a migration plan and a recovery path. NIST’s Secure Software Development Framework is a useful foundation because it defines practices across the development lifecycle without depending on whether a human, a model or both produced the code.

An AI-use label can support an experiment or audit, but it cannot replace evidence. A checkbox does not reveal an invalid assumption, a stale API or a dependency that was absent from the model’s context.

Protect code, context and execution

Before rollout, security and the developer platform team should agree on data boundaries. Secrets, production records, restricted customer code and contractually controlled information should not be sent to an external model. Verify retention, training use, processing region and administrative controls for the specific service and commercial plan; a vendor’s general product page is not enough.

An agent that executes commands should run with least privilege in an isolated environment, without standing production credentials. Network access, package registries and cloud tools need explicit allowlists and logging. Instructions embedded in untrusted files, issues, documentation or tool output deserve particular attention because a model may treat them as commands. Approval should evaluate the effect of an action rather than the persuasiveness of the generated explanation.

Repository context also requires ownership. Decide which documentation, architectural decisions and examples are authoritative, how they are updated and what must never enter an embedding index. More context is not automatically better; stale or conflicting context can make an answer more confident and less correct.

Run the rollout as a product experiment

The first cohort should contain representative teams and tasks, not only enthusiasts. Compare similar periods or groups using the same definition of done. Delivery telemetry needs qualitative investigation as well: where did the tool remove work, where did it create rework, when did missing context send an agent down the wrong path, and how did reviewer load change?

The 2025 Stack Overflow Developer Survey illustrates why adoption and confidence need separate measures. Eighty-four per cent of respondents used or planned to use AI tools, while 46% distrusted output accuracy and 33% expressed trust. This is not a product efficacy study. It is evidence that widespread use cannot stand in for verified value.

Write expansion, correction and stop criteria before the trial begins. If faster drafting increases the review queue, escape defects or maintenance effort, the next decision may be to narrow the use case, improve repository context or strengthen tests—not to buy more licences automatically.

Keep decision rights visible

The engineering leader owns delivery outcomes; the repository owner defines permitted uses; security owns data and privilege boundaries; procurement owns supplier terms; and the change author owns the submitted result. Grouping these duties under “AI governance” should not make any of them anonymous.

Review outcomes periodically by tool, model, task class and repository. Product capabilities and models change faster than conventional developer tooling, so an earlier result is not permanent approval for every new agent mode. Google Research describes the role of online experiments, workflow data and measurement of applied assistance. Apply the same discipline internally: evaluate usability, local speed and system-level performance as different questions.

Sources