Juan Vistro
Menu
← All work

Global biotechnology company · Agentic engineering

Building a delivery system while delivering the work.

Built an AI-assisted engineering harness while progressing 40 corrective tickets to PR within 30 days, combining isolated parallel execution, required validation, and explicit human recovery paths.

Software Engineer · Harness design and delivery

  • Engineering judgment
  • Systems architecture
  • Reliable delivery
Ticket backlog

Agentic delivery harness

  • Ticket A
  • Ticket B
  • Ticket C
Reviewable PR

Validate · Recover · Improve

Parallel work. Required checks. A recovery path.

Representative tickets—not a concurrency limit. Validation indicators mean checks are required before PR handoff, not that every ticket passed on its first attempt.

40 corrective tickets. A 30-day window. Human oversight retained.
Tickets reached PR
40Not necessarily merged or deployed
Required no manual code-writing
35/40Human oversight still required
Average human intervention per ticket
~2 hoursOversight and assistance continued
Delivery window
30 daysHarness built while delivering the work

01 / The Challenge

The deadline couldn't wait for the tooling.

A major platform migration had produced an urgent corrective backlog with a fixed 30-day window. The existing manual workflow would have required more engineering capacity than was available.

Building automation consumed time needed for delivery. The challenge was to create additional capacity without allowing speed to undermine correctness.

  • Time

    A fixed 30-day delivery window.

  • Uncertainty

    No established internal pattern for parallel agent execution.

  • Correctness

    Changes needed to preserve expected system behavior.

02 / The Approach

Build for parallelism. Prove it incrementally.

My previous behavioral-analysis harness provided a foundation, not proof of parallel code generation. I designed for concurrency but started with sequential tickets, establishing useful throughput before introducing parallel execution.

Each ticket received its own Git worktree and isolated test context. Required validation—including shadow-diff integration tests—had to pass before successful completion.

1 / Prove sequential execution

Complete actual tickets while developing the harness.

2 / Introduce parallel delivery

Unlock concurrency after proving working throughput, within the same 30 days.

  1. Ticket intake

    • Scoped corrective work
    • Required validation commands
    • Ready for implementation

  2. Parallel isolated execution

    Three representative lanes—not a maximum concurrency claim.

    • Ticket A
      1. Independent Git worktree

        Isolated test context

      2. Coding-agent implementation
      3. Required checks
        • Ticket-specific commands
        • Unit / integration tests
        • Shadow-diff integration tests where applicable
    • Ticket B
      1. Independent Git worktree

        Isolated test context

      2. Coding-agent implementation
      3. Required checks
        • Ticket-specific commands
        • Unit / integration tests
        • Shadow-diff integration tests where applicable
    • Ticket C
      1. Independent Git worktree

        Isolated test context

      2. Coding-agent implementation
      3. Required checks
        • Ticket-specific commands
        • Unit / integration tests
        • Shadow-diff integration tests where applicable

    Required validation Failure blocks successful completion → recovery path.

  3. PR handoff

    Reviewable changes with human oversight. Reached PR does not mean merged or deployed.

Failure recovery

  1. Failure
  2. Bounded retry
  3. Persistent failure
  4. Human handoff

Restore progress on one ticket: bounded retries distinguish transient anomalies from persistent failures. Persistent problems return to me for ad hoc agent-assisted completion. Retries do not guarantee success.

System improvement

  1. Langfuse traces, spans, artifacts and transcripts
  2. Debugging-agent investigation
  3. Targeted harness fixes

Improve the system across tickets: recurring failure evidence informed targeted fixes and subsequent executions. Not every failure could be corrected automatically.

From sequential proof to parallel delivery.

Conceptual architecture, not proprietary tooling. Tickets execute independently: each has its own worktree and isolated test context for implementation and required validation before PR handoff. Failed checks lead to bounded retry or human assistance, not PR. Separately, diagnostic evidence informs harness fixes for subsequent executions—not guaranteed automatic recovery.

03 / Key Decisions

Increase throughput without losing control.

The architecture evolved during delivery. I invested in leverage, proved new capabilities incrementally and used observed failures to decide which controls actually helped.

  1. Invest while meeting the deadline. I built the harness while delivering real tickets, not as a separate tooling project. Reusing the prior analysis harness and behavioral tests reduced the risk of investing within a fixed 30-day window.
  2. Prove before scaling. I designed for parallelism but proved useful sequential throughput first. One worktree and isolated test context per ticket then supported independent execution and validation during the same delivery window.
  3. Enforce what can be verified. I moved strict output requirements toward deterministic interfaces, while replacing brittle keyword scope checks with contextual agent judgment. Objective validation remained mandatory; stronger prompts were not a substitute for verifiable controls.
  4. Engineer recovery. Bounded retries and human escalation restored progress on individual tickets. Separately, Langfuse evidence, debugging agents and targeted fixes addressed recurring harness weaknesses—not an assumption of unattended recovery.

04 / Results and Reflection

More throughput. Less manual work. Continued oversight.

The corrective backlog progressed through implementation and validation within the delivery window. These results did not mean agents operated without supervision or that every change was correct on its first attempt. Some tickets required intervention or rework.

Delivery observations—not claims of unattended autonomy, merge or deployment.
Tickets reached PR
40Within 30 days; not necessarily merged or deployed
Required no manual code-writing
35/40Human oversight and intervention still required
Average human intervention per ticket
~2 hoursIncludes oversight and assistance
Average ticket cycle time
~1–1.5 daysApproximate observed cycle time

What changed in my engineering approach

During the 30-day period, I corrected two brittle controls. Prompting agents to produce structured files gave way to Codex's deterministic structured-output interface once its availability and suitability became clear. Rigid keyword checks could not reliably interpret scope intent, so I allowed contextual agent judgment while preserving deterministic checks for genuinely verifiable requirements.

In this organizational context, explicit controls helped establish confidence in an unfamiliar delivery model. But excessive deterministic checks became brittle when the underlying decision required interpreting intent.

My subsequent harness designs became thinner: preserve strict validation for genuinely verifiable requirements, while allowing agents more flexibility in implementation and reasoning. This later philosophy was a result of the experience, not the architecture I started with.

Leadership principle

Build systems that can demonstrate correctness, expose uncertainty, and recover when automation reaches its limits.

Explore projects →Get in touch →