CROOSTechnology
PT Talk
← All articles

Article

How to build a bot that survives UI changes

Why selector-based automation breaks in production, and three patterns — fingerprint, self-healing, vision LLM — that keep running after the UI changes.

  • automation
  • engineering
  • rpa

Ler em português →

Monday. The bot that pulls reports from the ERP runs flawlessly — like it has for six months. Tuesday morning: everything broken. No useful error — just ElementNotFound in a loop, and a queue of 200 stuck jobs.

The cause? A junior dev renamed a <div> the day before. The selector #download-btn-v2 became #download-button-final. Three days passed before anyone noticed, debugged, and fixed it.

The automation didn’t break. The selector broke. Everything else was working perfectly.

This scene plays out in every company that has automation in production for more than a year. And the blame almost never lies with the bot — it lies with how the bot was built.

Why selectors break (the real reasons)

Automation tutorials focus on “how to click the button.” Few cover why that button is going to change. But it will. In production, it always does.

The causes I see most in the field:

  • UI iteration — design systems shift, classes get renamed
  • Framework rewrites — jQuery to React migration, IDs become hashes
  • A/B testing — half the users see one structure, half see another
  • Internationalization — button text changes from “Download” to “Baixar”
  • Accessibility — someone adds aria-labels and refactors structure
  • Redesign cycles — every 18 to 24 months, the whole UI gets reconsidered

If your bot depends on a property that any of these can change, it’s going to break. The question isn’t if — it’s when.

The stability hierarchy

Every serious automation library (Playwright, Cypress, Selenium) reached the same conclusion: how you select an element matters more than which element you select.

Worst to best:

1. Position — button:nth-child(3)

// Breaks on the next feature that adds a button before it
await page.click("button:nth-child(3)");

Fragile in every direction. Any structural change breaks it.

2. Generated ID — #download-btn-v2-final

// Breaks the next time the team renames or regenerates it
await page.click("#download-btn-v2-final");

Looks stable, but IDs tend to be “undocumented documentation” — any dev can rename them because nobody told them it was a contract.

3. CSS class — .btn-download

// Breaks on any design-system refactor
await page.click(".btn-download");

Classes exist for styling, not automation. The designer is going to change them.

4. Dedicated attribute — [data-testid="download-receipt"]

// Stable IF the team maintains the convention
await page.click('[data-testid="download-receipt"]');

The pattern recommended by Playwright and Cypress. Requires cooperation from the product team — which, in legacy systems, you almost never have.

5. Semantic role — getByRole

// Survives CSS refactors, ID changes, visual redesigns
await page.getByRole("button", { name: /download receipt/i }).click();

The underrated one. ARIA roles describe what an element does, not how it looks. They survive most refactors because any accessible site has to preserve them.

6. Semantic + context combination — fingerprint

// Find the button inside the receipt card, by role + label
await page
  .locator("article", { hasText: "February receipt" })
  .getByRole("button", { name: /download/i })
  .click();

Multiple signals combined. For it to break, several would have to change at the same time — unlikely.

7. Visual detection / vision LLM

# The final frontier. Survives almost anything. Slow and expensive.
action = computer_use.find("the button that downloads February's receipt")

Models like Anthropic Computer Use see the screen like a human. They survive almost any change. But: ~10× slower, ~100× more expensive, and non-deterministic.

The three patterns that survive production

The hierarchy above is about individual selectors. But a good selector isn’t enough — you need an architecture that accepts failure.

Pattern 1: Fingerprint detection

Instead of one selector, multiple signals. Text + role + relative position + parent context. The function returns the element that matches most signals, not all of them.

function findDownloadButton(page) {
  return page
    .locator("*", { hasText: /download/i })
    .filter({ has: page.getByRole("button") })
    .filter({ hasText: /receipt|statement/ })
    .first();
}

When the site changes, usually one signal breaks. The others still match. The bot keeps running.

Pattern 2: Self-healing

Try the primary selector. If it fails, try a chain of fallbacks. When a fallback works, save it as the new primary.

const strategies = [
  () => page.getByTestId("download-receipt"),
  () => page.getByRole("button", { name: /download receipt/i }),
  () => page.locator("button", { hasText: /download/i }).first(),
];

for (const strategy of strategies) {
  try {
    await strategy().click();
    return;
  } catch {
    continue;
  }
}
throw new Error("No strategy worked — UI changed too much");

Bonus: log which strategy worked. After a week you know which selectors are “aging out.”

Pattern 3: Vision-based reasoning

Use vision-LLM as a last-resort fallback, not as the primary pattern. When the other patterns fail, send the screenshot to the model, let it identify the element, and learn the new selector from that.

Cost: high per call. Frequency: low. ROI: positive when the alternative is a human doing weekly maintenance.

The economics of maintenance

This is where most teams fool themselves. “I’ll build a quick bot, CSS selector, two hours. If it breaks, I’ll fix it.”

Let’s add it up:

  • 10 fragile bots × 30 min of maintenance per week = 5h/week
  • 5h × 52 weeks = 260h/year (≈ 1.5 months of work)
  • At $50/h loaded = $13,000/year in maintenance

Compare:

  • 3 robust bots × 10 min/month of monitoring = 6h/year
  • 6h × $50 = $300/year

The difference isn’t the cost of building. It’s the cost of maintaining, multiplied by 12 months.

Where AI helps (and where it gets in the way)

Vision-based agents (Computer Use, browser-use, Operator) are impressive. But they come with a hard trade-off:

  • Latency — 3 to 10 seconds per action, vs. 50 to 200ms for a selector
  • Cost — cents to $0.10 per action, vs. essentially zero
  • Determinism — the model can “see” the wrong element 1 to 2% of the time

Use when: the flow runs a few times a day, the site changes often, and the alternative is weekly human maintenance.

Don’t use when: the flow processes thousands of items, latency matters, or the aggregate cost exceeds an analyst’s salary.

Back to the real case

In the first article I mentioned “the clever bit” — the bot that detected the download pattern instead of hard-coding the selector. Now you have the vocabulary to understand what it was technically: fingerprint detection (Pattern 1) + self-healing (Pattern 2) combined.

There was no vision model back then. Today I’d add a third fallback using Computer Use for the cases where the first two fail — turns into a “final safety net” and frees the human team from any routine maintenance.

The rule that separates script from system

Automation that breaks is automation you keep paying for every month — in person-hours, in stalled queues, in irritated customers. Robust automation has a higher upfront cost and a near-zero ongoing cost.

If your bot has already broken twice this year due to UI changes, it’s not in production. It’s in “production until the product team ships its next feature.”

If you’d like to talk through turning a fragile bot into one that sleeps soundly, let’s have a quick chat. No cost, ~30 minutes.