How PMs use the Codex app

Key Concepts

• PR (Pull Request): A method of submitting changes to a code repository for review before merging.
• Buildkite: A continuous integration and continuous delivery (CI/CD) platform.
• Skill (within the App context): A pre-built function or command within an application (likely a coding assistant) that performs a specific task, like fetching logs or running tests.
• Codex: A language model (likely OpenAI’s Codex) used for code generation and understanding.
• Inductive Journey: A process of discovery and refinement through observation, questioning, and iterative improvement.

Understanding Code & Initial Investigation

The speaker, a Product Manager (PM) who infrequently codes, describes a scenario where they encountered a confusing button within the application. Rather than immediately making changes, they prioritized understanding the button’s functionality. This demonstrates a cautious approach to code modifications, particularly when lacking deep code understanding. The initial action was a simple question posed to the team: “What does this button even do?” This highlights the importance of clarifying functionality before making alterations. Confirmation from the team revealed the button was unnecessary, leading to a request for its removal. This illustrates a workflow of questioning, validating, and then proposing changes based on user feedback and team consensus.

Pull Request & Test Failure

Following the team’s confirmation, the PM created a Pull Request (PR) to delete the button. The PR was approved, but a subsequent test failure occurred. This is a common occurrence in software development, where changes can unintentionally introduce regressions. The speaker admits to initially being hesitant to manually investigate the test logs, demonstrating a reliance on automation and assistance tools.

Leveraging Skills & Buildkite Integration

Instead of manually debugging, the PM utilized a “skill” within their application – specifically, a “Buildkite” skill. This skill allowed them to directly access Buildkite, the CI/CD platform, and fetch the test logs with a simple command: “build kite fetch logs. Why failing?” The system responded, identifying the issue as a missing Buildkite token. This exemplifies the power of integrated tools and automated diagnostics. The PM then obtained the necessary token and instructed the application to “fix my PR.”

Iterative Skill Improvement & Codex Usage

The PM navigated to the top-level folder containing the skills and prompted the application to address the Buildkite token issue. Importantly, they didn’t simply ask to “update the skill,” but instead provided guidance on how to think through the problem, asking the application how to set the Buildkite token, reflect on potential fixes, and then update the skill. This deliberate approach, using Codex on a lower setting (described as “underrated”), is believed to result in higher quality solutions. The process involved a back-and-forth interaction, ultimately leading to a successful update to the relevant PR.

The Inductive Process & Long-Term Benefits

The speaker summarizes the experience as an “inductive journey.” Starting with user feedback (the confusing button), they progressed through questioning, change implementation (the PR), debugging (the test failure), and ultimately, skill improvement. During this process, not only was the initial problem resolved, but two skills were also refined. This highlights the iterative nature of software development and the potential for continuous improvement. The speaker notes that with each interaction, Codex becomes more proficient at working within the codebase.

Notable Quote

“I had to ask you how to set the build kite token. I don't want you to have to do that next time.” – This statement emphasizes the goal of improving the application’s ability to handle common tasks autonomously, reducing the need for repetitive instructions.

Technical Vocabulary

• Repo (Repository): A storage location for code and related files.
• CI/CD (Continuous Integration/Continuous Delivery): A set of practices designed to automate the software release process.
• Regression: An unintended consequence of a code change that introduces new bugs or breaks existing functionality.

Conclusion

The PM’s experience demonstrates a practical application of AI-powered coding assistance tools (specifically Codex and integrated “skills”) within a real-world software development workflow. The key takeaways are the importance of cautious code modification, leveraging automation for debugging, providing thoughtful guidance to AI models for better results, and recognizing the iterative nature of software improvement. The process highlights how even non-coding roles can effectively contribute to code maintenance and enhancement with the right tools and a deliberate approach.