How Log4Shell changed open source funding forever | Episode 6 | The GitHub Podcast

Key Concepts: Log4Shell, Open Source Funding, Open Source Maintenance, Sovereign Tech Fund (STF), Open Knowledge Foundation Germany, Sprint (German agency for disruptive innovation), Public Tender Rules, Bug Bounty, FASA (EU pilot project for open source security), XZ Utils vulnerability, Community Health, GitHub Secure Open Source Fund (SOS Fund), CVE (Common Vulnerabilities and Exposures), Cohort Model, Diversity of Funds, Prototype Fund (Germany), Open Technology Fund (US), NLnet, Industry Investment in Open Source, Drip (period tracking app), Ghost (open-source CMS), TTCmap.ca (Toronto transit visualization).

Introduction: The Aftermath of Log4Shell and Open Source Funding

The GitHub Podcast episode, hosted by Abby from GitHub's open source programs team, explores how the critical Log4Shell vulnerability in December 2021 fundamentally reshaped government and organizational approaches to open source funding. Log4Shell, a vulnerability in the open source Java logging library Log4j, allowed attackers remote control of affected systems with a single line of code, impacting millions of applications worldwide. The discussion features Felix Rea, Director of Developer Policy at GitHub, and Christian Grommyer, a Log4j maintainer.

Log4Shell's Impact and Government Response

Felix Rea, previously a board member of the Open Knowledge Foundation Germany, was lobbying the German government for open source maintenance investment when Log4Shell hit. He notes that Log4Shell was the first time the critical importance of open source maintenance became widely visible, comparing open source software to critical physical infrastructure like roads and bridges. In Germany, the cyber security agency issued a "red alert," leading many organizations, including the German health records system, to temporarily shut down services to prevent data leaks. This immediate, tangible impact garnered attention from politicians previously unconnected to digital policy.

Christian Grommyer, a self-employed trainer, software developer, and consultant involved with the Apache Log4j team, described the overwhelming experience of seeing his email inbox explode during the crisis, highlighting the immediate and intense pressure on maintainers.

The Genesis of the Sovereign Tech Fund (STF)

The proposal for the Sovereign Tech Fund (STF) in Germany predated Log4Shell, with feasibility studies already conducted. Felix Rea had a champion in Rafael Laguna de la Vera, head of Sprint, a German government agency for disruptive innovation. Despite initial disinterest from a journalist who found the topic "boring," Log4Shell's emergence changed everything. The journalist subsequently wrote a widely impactful article, "How do you put out a burning internet," which highlighted existing proposals for open source infrastructure support, referencing past incidents like Heartbleed. This article was instrumental in convincing the new German government to fund the STF, which was announced by the Ministry of Economy on Christmas Eve.

Christian Grommyer initially learned about the STF through a friend, unaware that Log4j's crisis was a direct catalyst for its creation. After applying, the Log4j project successfully received funding from the STF.

Challenges and Dynamics of Open Source Funding

Christian detailed the complexities of accepting STF funding for Log4j:

• Low Acceptance Rate: Only two out of ten Log4j maintainers accepted the funding. Reasons included new job commitments, aversion to tax complications, or a general lack of interest in working on open source for pay.
• Financial Management: Christian faced challenges in distributing the money, managing taxes, and dealing with other organizations in Germany demanding a share of the funds.
• Team Dynamics: Funding introduced tension within the previously all-volunteer team. Some members could now work full-time, while others, including the project's founder, could not accept the money, leading to feelings of exclusion. Christian emphasized the importance of constant communication to mitigate these emotional impacts and maintain team cohesion, especially given the team's global distribution (Netherlands, Poland) and varying costs of living. The Log4j team, being a 25-year-old project with established respect, was able to navigate these challenges.

Felix noted that funding, while crucial, is only one part of open source sustainability. He observed that projects can "implode" or "sit on that money for years" if they don't know how to manage it beyond "pizza party levels."

Lessons from Broader Funding Initiatives

Felix shared lessons from other funding efforts:

• FASA (EU Pilot Project for Open Source Security): This initiative faced government bureaucracy and public tender rules, preventing direct funding to developers. They resorted to a bug bounty project, but learned that bug bounties alone are often counterproductive, creating more work for maintainers if bug fixes aren't also incentivized.
• Community Health: The XZ Utils vulnerability, discovered shortly after Felix joined GitHub, underscored the importance of "community health." Open source development is a "human sport," requiring support for in-person meetings for globally distributed teams to prevent misunderstandings that arise from purely digital communication.
• Fragility of Quick Movements: Referencing the book "Twitter and Tear Gas," Felix highlighted that while digital tools enable quick assembly, they don't build the "hard work of working in community" or conflict resolution skills, making projects fragile. Long-standing projects like Log4j (25 years old) benefit from established respect and collaboration.

The GitHub Secure Open Source (SOS) Fund

Christian's experience with the GitHub SOS Fund was positive:

• Validation and Learning: Initially hesitant, not considering himself a "security person," Christian found validation that Log4j was already implementing many best practices. The program provided confirmation and identified areas for improvement, boosting his confidence and leading to the implementation of missing security pieces.
• Cohort Model and Diversity: The SOS Fund's cohort model fostered community among developers facing similar problems. Christian particularly valued the diversity of projects (Java, Python, UI), which exposed him to different challenges and broadened his perspective.
• Beyond Development Skills: Felix noted that even non-developers could benefit, citing a session on writing effective CVEs, which emphasized clear communication skills over coding.

Felix, involved in the SOS Fund's creation, expressed initial concerns about creating more work for maintainers but was pleased with the program's value and the effectiveness of the cohort model in building community.

Future of Open Source Funding and Developer Policy

Felix emphasized the need for a diversity of funds (e.g., public, private, industry-specific like GitHub SOS Fund, government-specific like STF, Prototype Fund, Open Technology Fund, NLnet) to cater to different project needs, from critical infrastructure maintenance to incubating new ideas or even winding down projects. He highlighted that while industry invests significantly in open source (developer time, foundation funding), this investment isn't always efficiently distributed to projects most in need. Felix is currently advocating for an EU fund, drawing lessons from the STF, and stresses the need for more research into effective funding models.

Christian, from a maintainer's perspective, expressed hope due to the emergence of funds and increased political awareness. However, he acknowledged significant remaining challenges:

• Long-term Sustainability: The temporary nature of funding streams creates instability for maintainers and their families.
• Defining "Criticality": Determining which projects are "critical" and deserve funding remains complex.
• Shifting Perception: Open source has evolved from a niche activity for "hackers at night" to an indispensable global infrastructure, driving everything from cars to financial transactions. This fundamental shift in perception gives him hope for future progress, though he estimates it will take "five or ten more years" to fully flesh out solutions.

Project Shout-outs

• Felix Rea: Drip, an open source period tracking app. He praised its privacy-preserving nature (sensitive medical data stored on-device) and inclusive design, avoiding gendered assumptions.
• Christian Grommyer: Ghost, an open source CMS. He appreciated its excellent documentation, ease of setup (with Docker support), and the company's transparency regarding financial numbers and community support.
• Abby: ttcmap.ca, an open source project visualizing Toronto's public transit delays and reroutes using open data. She found it very handy and noted it was created by a PhD student.

Conclusion

The Log4Shell crisis served as a pivotal moment, catalyzing significant shifts in how governments and organizations approach open source funding and sustainability. While initiatives like the Sovereign Tech Fund and GitHub Secure Open Source Fund offer promising models, the journey towards a truly sustainable and equitable open source ecosystem is ongoing. Key takeaways include the critical importance of diverse funding mechanisms, the necessity of supporting community health and conflict resolution within open source projects, and the ongoing challenge of ensuring long-term financial stability for maintainers. The conversation underscores that open source is no longer a niche hobby but a foundational pillar of modern society, demanding sustained investment and thoughtful policy.