Open Source Friday with Gunnar Morling with Hardwood

Key Concepts

• Apache Parquet: A columnar storage file format optimized for large-scale analytical queries.
• Hardwood: A new, high-performance, multi-threaded Java implementation of a Parquet parser with minimal dependencies.
• Columnar Storage: A data organization method where values of a specific column are stored consecutively, enabling efficient analytical processing and compression.
• JDK Flight Recorder (JFR) & Async Profiler: Java-based diagnostic and profiling tools used to identify performance bottlenecks and memory allocation hotspots.
• Agentic Coding: The practice of using AI agents (like Claude or GitHub Copilot) as collaborative tools in the software development lifecycle, rather than as autonomous code generators.
• Predicate Pushdown: An optimization technique where filters are applied at the storage layer to avoid reading unnecessary data.

---

1. Project Overview: Hardwood

Hardwood is a Java-based project designed to parse Apache Parquet files. Unlike existing implementations that often pull in the heavy Hadoop ecosystem, Hardwood is designed to be:

• Standalone: It has no mandatory transitive dependencies, reducing the "dependency baggage" and security risks associated with large classpaths.
• Multi-threaded: It leverages modern multi-core CPUs to process data in parallel.
• Performance-Oriented: It utilizes low-level optimizations, including native memory access and primitive data types, inspired by the "1 Billion Row Challenge."

2. Technical Principles and Methodologies

• Row-based vs. Columnar: The video contrasts row-based formats (like CSV, useful for operational workloads) with columnar formats (like Parquet, useful for analytics). Columnar storage allows for "projection" (reading only specific columns) and efficient compression (e.g., delta encoding for timestamps or dictionary encoding for categorical strings).
• Performance Analysis: Gunnar emphasizes the use of JDK Flight Recorder for event-based diagnostics and Async Profiler for identifying CPU and memory bottlenecks. These tools allow developers to visualize performance via flame graphs.
• Native Binaries: The project uses GraalVM to compile the Java CLI into a native binary, ensuring fast startup times, which addresses the common criticism that Java is too slow for CLI tools.

3. AI-Assisted Development Framework

Gunnar advocates for a "Built with AI, not by AI" philosophy:

• Design-First Approach: Before writing code, the project uses design documents stored in the repository. AI agents are tasked with drafting these designs, which are then reviewed and refined by the human maintainer.
• Test-Driven Iteration: The project relies on the official Apache Parquet test suite. When a feature is missing or a bug is found, the AI is tasked with making the implementation pass these specific, standardized test files.
• Human-in-the-loop: The maintainer retains ownership of the architecture, public APIs, and critical logic. AI is used to handle repetitive tasks or complex algorithms (like the S3 request signing algorithm) where the AI can be verified against existing specifications and test suites.

4. Real-World Applications

• 1 Billion Row Challenge: A viral coding challenge that served as the inspiration for Hardwood. It required processing 13GB of data to find min, max, and average values, teaching the importance of parallelization and memory management.
• Object Storage Integration: Hardwood implements its own S3 client using Java’s built-in HTTP client. By focusing only on the necessary subset of the S3 REST API, the project avoids the massive dependency footprint of the official AWS SDK.

5. Notable Quotes

• "I feel like those end-to-end tests work just like really well for this kind of implementation approach... I can compare what the Hardwood parsing result is to what the upstream Parquet Java parser emits." — Gunnar Morling
• "In the moment when you propose something, you have to own it. If you propose crap because AI told you so, this is totally on you." — Gunnar Morling

6. Synthesis and Conclusion

The development of Hardwood demonstrates a shift in open-source maintenance. By combining spec-driven development with AI-assisted implementation, maintainers can build high-performance, low-dependency libraries that were previously too costly to maintain from scratch. The project serves as a blueprint for how modern developers can use AI to handle low-level technical challenges while maintaining strict control over public APIs and architectural integrity. The key takeaway is that AI should act as a force multiplier for a human maintainer who remains the ultimate authority on the project's design and quality.