Duration: 24:48
PART 1 — Analytical Summary 🚀
Context and why it matters 💼
This session, “Runbot knows what you broke: Behind testing infrastructure,” is led by Kristoff Monet from Odoo’s Runbot team. Born 11+ years ago, Runbot started as a way to spin up Odoo instances for manual testing and has evolved into a core, data-driven CI system powering Odoo’s multi-repository development at scale. It sits alongside the Merge Bot and orchestrates complex pipelines spanning the community and enterprise codebases. The talk explains how Runbot compensates for the limits of external CI tooling, keeps Odoo’s branches healthy, and helps engineers quickly pinpoint regressions—even when failures are nondeterministic.
Why not just use GitHub Actions? Three principal reasons emerged: the need to coordinate tests across multiple repositories, extremely long test suites (9 hours for integration/unit tests; ~14 hours including specialized suites), and the need to handle nondeterministic failures without blowing through hosted CI quotas. Self-hosting also gives Odoo freedom to customize deeply.
How Runbot is built and operated ⚙️
Runbot is itself an Odoo module, open-sourced on GitHub, and runs on a distributed fleet. It’s split into four parts: a front-end UI, a “leader” that dispatches work, “builders” that execute jobs, and a dedicated builder for Docker images. Builds run in containers tailored to mirror Odoo’s SaaS environment (browser versions, Python version, packages); images are built centrally and pulled by builders to ensure reproducibility and performance.
Runbot’s front-end is more than a dashboard—it’s an operational control room. Engineers can: - Navigate “bundles” (branches aligned across repos) and “batches” (sets of commits and builds tested together). - See health via color status, check how far a branch is behind its base (e.g., “62 commits behind 19.0”), and jump directly into running Odoo instances for manual verification. - Inspect logs quickly, with links to full text logs and the exact command lines used.
From commit to confidence: bundles, batches, and builds 🧠
When a pull request lands in GitHub, a webhook notifies Runbot. Runbot creates a bundle (branches of the same name across repos, including PRs) and a batch (the set of commits and builds to test). It then “fills missing commits,” automatically selecting the best-matching commits in other repositories so the whole stack is tested coherently. That cross-repo synchronization is central to Odoo’s workflow.
Builds are split into steps and commonly run in containers. For heavy jobs (like the long enterprise test suite), Runbot creates sub-builds: it pre-installs all modules into a database, then parallelizes post-install tests by restoring that DB across multiple child builds. This significantly reduces cycle time and spreads the load across builders.
What gets tested—and how failures are tamed 💬
The pipeline spans fast validations and deep suites: - Early gates like a “minimal check” validate commit formatting to avoid wasting compute. - Quick “community run” and “enterprise run” builds spin up working instances fast for manual testing. - The “enterprise test” encompasses the multi-hour suite. - Additional checks include upgrade tests and fast style/security checks.
To tackle nondeterministic failures, Runbot runs nightly special builds: - Multi-builds repeat identical runs many times; if any red appears, the failure is flagged as random. - Single-app builds install one application plus dependencies and run its tests to isolate regressions. - “Fake time” builds run with shifted system dates (into the future or edge cases) to catch time-related bugs earlier.
Failures detected overnight are parsed and indexed as Runbot errors in the database. Runbot extracts qualifiers (module name, file path, test class/method, etc.), tracks first/last seen timestamps, and supports assignment workflows. Teams can link a candidate fixing PR and validate the outcome via graphs showing the last failing nightlies and whether failures disappear after the merge—turning “it seems fixed” into measurable evidence.
When a test becomes too flaky and blocks productivity, Runbot allows disabling it per version range (e.g., 19.0 to master), unblocking merges while responsible teams stabilize the test.
Extensibility, limits, and practical realities 💼
Runbot is usable by partners or clients as it’s an Odoo module, but deployment isn’t trivial today; the team provides a README and can advise, yet acknowledges setup could be improved. It’s architected to support GitLab in the future, though there’s no active plan to switch. Runbot is independent of Odoo.sh (no integration; log access there is unrelated). The team does maintain internal vulnerability scanning configurations, but those won’t be released; users can define their own custom builds and rules. Beyond Odoo’s core, Runbot also tests other projects, including Runbot itself.
Impact and takeaways 🧠
Runbot’s design focuses on integration, speed, and accountability. It streamlines cross-repository testing, shortens feedback loops with quick instance spins, and tames flakiness with targeted nightlies and metadata-rich error tracking. Engineers get clarity—what failed, when it started, who should fix it—and confidence in fixes via post-merge validation. For an ecosystem as broad as Odoo’s, this infrastructure keeps branches green, accelerates merges, and preserves developer velocity without compromising test depth.
PART 2 — Viewpoint: Odoo Perspective
Disclaimer: AI-generated creative perspective inspired by Odoo’s vision.
The value of Runbot isn’t only in running tests; it’s in reducing uncertainty for contributors. When your stack spans multiple repositories, the hardest problem is coherence. Bundles and batches give us that coherence, and the nightly strategies help us isolate the rare, tricky failures that cost teams days. We want developers to spend more time building and less time guessing.
Our philosophy remains the same: keep it simple for the user, integrated for the platform, and open for the community. By making Runbot an Odoo module and mirroring SaaS environments in Docker images, we keep the feedback fast and relevant. The result is a more predictable release rhythm and a happier community of contributors.
PART 3 — Viewpoint: Competitors (SAP / Microsoft / Others)
Disclaimer: AI-generated fictional commentary. Not an official corporate statement.
Odoo’s Runbot shows a pragmatic approach to CI at scale for a multi-repo product with an active community. The nightly strategies for nondeterminism and the metadata-driven error indexing are strong practices that any engineering organization can learn from. We particularly respect the emphasis on short feedback cycles via quick-running instances.
The enterprise challenge remains: scaling compliance coverage, long-duration tests, and security workflows across global organizations while keeping UX friction low. Hosted CI suites offer governance and audit tools out of the box; Odoo’s self-hosted freedom is powerful but shifts operational responsibility in-house. The differentiation will continue around depth of compliance, traceability, and how seamlessly these pipelines support very large customer landscapes without increasing cycle time.
Disclaimer: This article contains AI-generated summaries and fictionalized commentaries for illustrative purposes. Viewpoints labeled as "Odoo Perspective" or "Competitors" are simulated and do not represent any real statements or positions. All product names and trademarks belong to their respective owners.