• Kyle Mann

As a designer in software, I’ve always admired how architects can see their designs realized in physical form—a tangible manifestation of their vision. But I don’t envy their mistakes, which stand out for years or require costly reconstruction. In contrast, in software, shipping something imperfect is a badge of honor—it provides the feedback we need to iterate, pivot, refine, or roll back quickly.

Given the iterative nature of software development—where your designs are constantly evolving with every sprint—it’s easy to look back and ask: “What was I thinking?” and “What did I get done? What went well/poorly?” That’s why I carve out time each quarter and year to reflect: documenting what we’ve accomplished, the decisions we made, the questions raised, what worked well, and where we can improve—all while seeking input from the team.

This review focuses on product and design-related changes—it doesn’t cover the incredible behind-the-scenes engineering work, like maintenance, bug fixes, or marketing efforts that help bring everything together.

Value People

Moving into 2024, as our team expanded, and as our product UX needed maturing, we needed someone to help scale our growing product design function. In December 2023, about a year ago today, Adalene Teh joined our product design team, bringing her expertise from Amazon (AWS).

Adalene hit the ground running, quickly grasping the complexities of our product. What set her apart, though, was her collaborative approach to learning. She organized sessions and workshops with the team, using them not just to gain knowledge but to ask insightful questions that prompted the entire team to reflect and uncover previously overlooked insights. Together, we worked closely to iterate on and establish processes that could scale and better support our counterparts.

A year later, her contributions to new feature development and iterations are too numerous to list. However, what I admire most is her relentless curiosity and pursuit to deeply understand things—and then her drive to take action. Given her impact—I can’t believe it’s only been a year!

Work In Progress

In 2024, we focused on two main themes: 1) core product improvements, and 2) going beyond coverage. These priorities often competed for attention, but through a game of Tetris each sprint and quarter—here’s where we landed:

Pixels Matter: Core Product Improvements

If you’ve ever been part of a migration or redesign, you know the mention of “front-end migration” can send developers running for the hills. While we completed our front-end migration nearly two years ago, the process left us with lingering debt: cut features, polishing needs, copy fixes—all the loose ends. On top of that, we faced a growing backlog of product improvements that had been deprioritized during the migration. Here’s some of the larger items and themes:

  • Experimenting with PR comment formats to simplify reports
  • Improved debuggability with our uploads/coverage processing history
  • Navigation updates to accommodate new feature experimentation
  • Configuration management at repo level
  • Aligning copy across the UI and documentation
  • Fixing behavior issues, like handling removed code
  • Making Codecov more open-source-friendly (uploads without tokens)
  • Improving onboarding and simplifying the GitHub app installation process to reduce friction for new users (and discovery around providing value without requirement of GitHub app)
  • Pricing updates, including the introduction of a team plan for smaller teams
  • Dark mode
  • OKTA integration to support customer onboarding
  • Slack integration
  • VS Code extension (with highlighting)
  • Last, but not least: browser extension

These efforts not only resolved migration debt but also laid the foundation for a best-in-class product that customers can trust and love. We achieved these foundational improvements while simultaneously delivering entirely new products…

For Every Developer: Going Beyond Coverage

Our core mission remains helping developers write better code with less risk, but in 2024 we expanded our offerings beyond testing coverage to support a broader range of developer needs. Here’s a breakdown of our progress:

Bundle Analysis: our first venture outside the realm of testing, helping front-end developers understand and optimize their bundle size.

  • Current status: adopted by 252 organizations. Working on enhancing data actionability to guide developers toward performance improvements.
  • Tradeoff review: while supporting JavaScript-related issues aligned with our strategy, this experimental product feels distinct from testing-related features. We’re asking, “What does success look like, and how does it fit into the pre-release story?” Another notable, is that in our latest iterations, large code changes / bundle gates, there is a performance aspect to this – some exploration needed, but could be a another opportunity for a Sentry integration.

Test Analytics: building on our testing suite, this feature provides insights into frequently failing tests, flaky tests, and framework health.

  • Current status: adopted by 703 organizations, making it our fastest-growing experimental feature. We’re doubling down in this direction, as it aligns strongly with customer needs for helping them understand test quality.
  • Tradeoff review: while extending testing relationships makes this path clear, it raises questions about how we prioritize improvements versus other experimental features. One example of this is automated test selection, a concept we shelved for the time being due to its time constraint and technical challenges.

AI Review: helping developers generate tests for uncovered code and reviewing code changes with actionable suggestions before merging.

  • Current status: Adopted by 157 organizations in a short time, providing feedback directly in the PR comment. While the early adopters are promising, at the moment it’s in BETA and very experimental. Looking to get a sense of its accuracy improvement in the wild.
  • Tradeoff review: the AI space is very competitive and evolving rapidly, however Codecov’s existing integration into developer workflows gives us a unique advantage. While results aren’t always perfect, the feature saves time by offering a strong starting draft.

Looking back, we started 2024 with a rough idea of “pre-release beyond coverage” and ended with a solid foundation for a suite of tools that integrate seamlessly into existing workflows. These tools leverage our core UX: 1) reporting in pull request comments, and 2) providing insights into code quality via the UI; while it broadens the scope of reporting for our customers

Work In Progress: Next Up in 2025

Reflecting on 2024, our focus was broad—improving the core product while casting a wide net with new features. In 2025, our theme shifts to going deep and expanding on these efforts.

Core Product Improvements

One core challenge has been how we process and display coverage reports. Currently, users may encounter incomplete or processing reports, leading to confusion, support tickets, and feedback that “Codecov is broken.” In 2025, we aim to address this by better contextualizing report statuses and ensuring only confirmed, accurate results are displayed—maintaining trust in our reports.

New Feature Development

For the new features launched in 2024—bundle analysis, test analytics, and AI review—2025 will be about observation, review, and tough questions. Early traction suggests test analytics is a standout, aligning closely with our existing offerings and appealing to a broad audience. Identifying flaky and failing tests resonates across teams, from startups to large enterprises, whereas coverage reporting often has more appeal to mid-to-large organizations.

Ongoing Discovery and Integration with Sentry

Sentry is the go-to debugging tool for developers. With this in mind, we’re exploring ways to support developers earlier in their workflow to help prevent errors before they occur. This includes researching pre-release workflows and identifying points where we can proactively assist developers in avoiding errors altogether.

In parallel, we’ll revisit our existing Sentry integration to revisit/enhance its UX and increase visibility for features like test analytics. By embedding these capabilities into key Sentry touchpoints, we can introduce Codecov’s features (such as test analytics) to a broader audience who may not yet use our platform.

Lastly, In late 2024 Adalene and I officially transitioned to the Sentry Design team, moving from reporting within Codecov engineering. This structural change reflects our growing collaboration with the wider Sentry teams and our commitment to building features directly under its umbrella. I’m excited about the opportunities these integrations present to better serve developers in the pre-release and error prevention spaces.

Feedback is priceless

At Sentry, we know feedback is priceless, therefore here is a rough recap of feedback we received across surveys regarding our product (Pro and Team plan) AND internal feedback from the team.

Paying Customer Survey Results: Codecov Pro Plan

Roles in Organizations

  • Responses:
    • 56.7%: Developer
    • 20.9%: Engineering Manager
    • Remaining roles included QA Engineers, CTOs, Platform Engineers, and other technical leadership roles.

Ease of Setup

  • Responses:

    • 60.7%: Somewhat easy
    • 16.4%: Very easy
    • 21.3%: Somewhat difficult
    • 1.6%: Very difficult

  • Challenges Encountered:

    • “Really hard to figure out how to add more team members and get comments added to their PRs.”
    • “The documentation feels disorganized, especially for configuration options like Component vs Flag.”
    • “Repo syncing is slow, and API changes break integrations.”
    • “Monorepo support required more effort than single repositories.”
    • “Login issues from GitHub links leading to 404 pages are frustrating.”

Testing Prioritization

  • Responses:
    • 80.0%: Essential – It’s a core part of our development process.
    • 13.8%: Important – We test when possible, but it’s not the top priority.
    • 6.2%: Moderate – Testing happens where necessary, but not heavily focused.

Code Coverage Prioritization

  • Responses:
    • 40.9%: Essential – It’s a core part of our development process.
    • 31.8%: Important – Actively maintained but not top priority.
    • 24.2%: Moderate – Used as needed but not heavily focused.
    • 3.1%: Limited – Minimal focus on code coverage.

Pain Points in Code Coverage and Testing

  • Recurring Themes:

    • Integration Challenges: “Getting coverage reports for monorepos or specific setups like Cypress was difficult.”
    • Login Issues: “GitHub links leading to 404 pages when not logged in is a major pain point.”
    • Unreliable Metrics: “Coverage regressions reported for untouched code or dependency updates are confusing.”
    • Flaky Reporting: “CI upload delays and intermittent failures disrupt the workflow.”

  • Direct Quotes:

    • “Coverage doesn’t update sometimes. ATS misses changes.”
    • “The report shows only files involved in tests. If a large part of the code is not covered, it is not visible in the report.”
    • “The login strategy is frustrating; I can’t log in directly from a 404 page.”
    • “Identifying flaky tests in Go is a challenge that Codecov is helping us address.”

Usefulness of Coverage Insights

  • Responses:

    • 41.5%: Moderately helpful
    • 38.5%: Very helpful
    • 10.8%: Extremely helpful
    • 9.2%: Not helpful

  • Examples of Impact:

    • “Integration with GitHub PRs provides visibility into coverage drops before merging.”
    • “Coverage insights helped us grow our coverage from 68% to over 80% without a big upfront effort.”
    • “Dead code identification through indirect changes is a valuable feature.”
    • “Knowing what’s covered gives us confidence when deploying.”

Overall Satisfaction with the Pro Plan

  • Responses:
    • 64.1%: Satisfied
    • 21.9%: Very satisfied
    • 14.0%: Unsatisfied
    • 6.3%: Very unsatisfied

Additional Feedback

  • Positive Notes:

    • “Coverage reports help enforce quality without being onerous.”
    • “The pull request integration is central to our workflow.”
    • “We’ve seen cultural change toward prioritizing testing and quality.”

  • Suggestions for Improvement:

    • “Fix login issues, especially links leading to 404 errors.”
    • “Improve monorepo support and documentation for complex setups.”
    • “Provide clearer insights and a dashboard to track overall coverage trends.”
    • “Make the UI more stable and intuitive, especially for navigating reports.”
    • “Lower pricing for smaller teams or offer more tailored plans for SMBs.”

Internal Feedback

This includes results from an internal survey from teammates across various functions about the product and design team. The goal was to understand how well our processes, communication, and priorities are supporting the broader team and where we can improve. Here are the results:

Visibility Into Discovery and Design Work

  • Responses:

    • 54.5%: Yes, I feel I have good visibility.
    • 45.5%: Somewhat, but I would like more context.
    • 0%: No, I feel there is limited visibility.

  • General:

    • “I feel like I have good visibility into active design/product work and have an avenue to provide feedback when I want to.”
    • “I’m not sure if I need more visibility of design. I have pretty good product visibility.”

  • Areas for Improvement:

    • “I’d be interested in learning more about how customer feedback are aggregated and translated to longer-term projects.”
    • “I’m not sure where to look for the next possible projects that are in discovery/design.”
    • “Maybe present mockups and solicit feedback in a wider audience?”
    • “Some form of dashboard (for example, GitHub projects with the right filters).”

Understanding Design Iterations

  • Responses:

    • 9.1%: Yes, I always understand the reasons behind changes.
    • 63.6%: Sometimes, but additional context would help.
    • 27.3%: No, I often don’t understand the reasons.

  • General:

    • “The process seems to work mostly well from my (engineer) perspective at this point.”

  • Areas for Improvement:

    • “I like it when designs have a little concise description of the problem and solution. I don’t want to have to go dig through the discussion on an issue just to get the little bit of basic context I’m looking for. That said, I do understand sometimes there is more than a little bit of context required (the appless designs, for example).”
    • “How does an issue tie in with the company long-term strategy? vs ‘we’re building this because customer BigMoney asked for it and it makes sense in the product.’”

Clarity of Prioritization

  • Responses:

    • 45.5%: Yes, the prioritization process is clear.
    • 54.5%: Somewhat, but it could be clearer.
    • 0%: No, the process is unclear.

  • General:

    • “Prioritization seems to balance features with customer needs reasonably well.”

  • Areas for Improvement:

    • “There have been instances where we prioritized the wording of errors without fixing the underlying bugs causing them first. While I appreciated the cleared messaging, the error should not have existed in the first place and it caused a poorer customer experience for customers trying to onboard, and the customer-facing teams assisting them.”
    • “The balance between building something new vs polishing what we have is a bit fuzzy and has to be explained more.”
    • “I would love to WSLR framework similar to product that I could see prioritized on quarterly basis.”

Channels for Sharing Suggestions

  • Responses:

    • 63.6%: Yes, I know where and how to share suggestions.
    • 36.4%: Somewhat, but I’m unsure of the best channels.
    • 0%: No, I don’t know where or how to share suggestions.

  • General:

    • “I feel like I know where and how to provide feedback and see it incorporated when appropriate.”

  • Areas for Improvement:

    • “Need one centralized place to raise issues, currently I know of two: Figma comments and Slack design channel threads.”

Effectiveness of Addressing Issues

  • Responses:

    • 54.5%: Yes, I feel my issues are heard and addressed.
    • 36.4%: Sometimes, but not consistently.
    • 0%: No, I don’t feel my issues are heard or addressed.
    • 9.1%: I have not raised any issues.

  • General:

    • “IMO the process seems to work mostly well from my (engineer) perspective at this point. I feel like I have good visibility into active design/product work and have an avenue to provide feedback when I want to.”

  • Areas for Improvement:

    • “There have been instances from design side where the broader team was asked for feedback, the feedback was consistent, but it was ignored.”

Understanding Vision and Direction

  • Responses:

    • 45.5%: Yes, I understand the vision and direction clearly.
    • 54.5%: Somewhat, but I would like more clarity.
    • 0%: No, I don’t understand the vision or direction.

  • General:

    • “The overall vision is clear, and quarterly priorities are generally well communicated.”

  • Areas for Improvement:

    • “There’s been this debate about ‘is codecov for enterprise or self-served?’ and it’s not clear for me the features that align more with one or the other, or why we would prioritize one over the other (sometimes you can appeal to both, but not always).”
    • “It might be good to know in which stage certain features and ideas are. Is it just an idea? Do we know there is (paying) customer demand for this? How do we know?”

Additional Feedback

  • General:

    • “Looking forward to a successful 2025!”
    • “Get deeper and deeper in with Sentry design. Let’s get features in the Sentry UI.”
    • “IMO the process seems to work mostly well from my (engineer) perspective at this point. I feel like I have good visibility into active design/product work and have an avenue to provide feedback when I want to.”

  • Areas for Improvement:

    • “We need a better process around ‘un-shipping’ features. It feels like there have been a couple of experimental features, or features purpose-built for single customers which are not widely adopted. Those features seem to be in ‘limbo’ and are lacking a clear decision of either ‘yes, this is a great feature that provides value, we will double down on it and polish it up,’ or ‘we can never get this to scale beyond just a prototype, and there are no customers willing to pay for this either, let’s cut our losses.’”
    • “Sometimes OKRs get blocked (ie not dev-ready) a few weeks/sprints into the new quarter. I’m hoping that engineering can start their work as soon as the quarter starts. If there are multiple OKRs for the same product then having just one to be ready when the quarter starts is fine. I just want to avoid the situation where a given engineer has no OKRs to work on when the quarter starts.”
Get me out of here
Send us a reply via email