# parquet-go Project Context

## Overview
High-performance Go library for reading and writing Apache Parquet files. Originally developed by Twilio Segment, now community-maintained at github.com/parquet-go/parquet-go.

- **Go Version**: 1.22+
- **Current Version**: v0.26.0+ (pre-v1, breaking changes possible)
- **Codebase Size**: ~71,700 lines across 210 Go files

## Project Structure

```
parquet-go/
├── Root Package          # Core API: Reader/Writer/File/Schema
├── encoding/             # 7 encoding formats (plain, rle, delta, etc.)
├── compress/             # 6 compression codecs (snappy, gzip, zstd, etc.)
├── bloom/                # Split-block Bloom filter (SIMD-optimized)
├── sparse/               # Sparse array utilities with gather ops
├── hashprobe/            # Hash-based dictionary operations
├── format/               # Generated Thrift definitions (parquet spec)
└── internal/             # Memory mgmt, byte algorithms, unsafe casts
```

## Key Files by Function

| Purpose | Files |
|---------|-------|
| Core Types | `parquet.go`, `file.go`, `schema.go`, `node.go`, `type.go` |
| Reading | `reader.go`, `column.go`, `page.go` |
| Writing | `writer.go`, `column_buffer.go`, `buffer.go` |
| Logical Types | `type_boolean.go` through `type_variant.go` (40+ files) |
| Column Buffers | `column_buffer_*.go` (20+ files for each physical type) |
| Page Impl | `page_*.go` (15+ files) |
| Operations | `convert.go`, `merge.go`, `sorting.go` |
| Configuration | `config.go` (900+ lines) |

## Core Types & Interfaces

### Schema System
- `Schema` - Immutable, thread-safe parquet schema
- `Node` - Interface for schema nodes
- `Field` - Named schema node interface
- `Column` - Concrete column representation
- `Type` - Logical type interface
- `Kind` - Physical type enum (Boolean, Int32, Int64, Float, Double, ByteArray, FixedLenByteArray)

### Reader/Writer System
- `GenericReader[T]` - Type-safe reader (preferred)
- `GenericWriter[T]` - Type-safe writer (preferred)
- `SortingWriter[T]` - Writer with integrated sorting
- `GenericBuffer[T]` - In-memory row group buffer (implements sort.Interface)

### Row Group Abstractions
- `RowGroup` - Interface for row group collections
- `ColumnChunk` - Interface for column data
- `Page` - Interface for page data with stats
- `Pages` - Sequential page reader

## Common Patterns

### Writing Data
```go
// One-shot
parquet.WriteFile[T](path, rows, options...)

// Streaming
writer := parquet.NewGenericWriter[T](w, options...)
writer.Write(rows)
writer.Close()

// With sorting
sortWriter := parquet.NewSortingWriter[T](w, rowCount, options...)
```

### Reading Data
```go
// One-shot
rows, _ := parquet.ReadFile[T](path)

// Streaming
reader := parquet.NewGenericReader[T](r, options...)
n, _ := reader.Read(rows)

// Low-level
file, _ := parquet.OpenFile(r, size)
for _, rg := range file.RowGroups() { ... }
```

## Build Commands

```bash
make test     # Run tests with -race and coverage
make format   # Go fmt + modernize tool
make tools    # Install development tools
```

## Testing Patterns

- Unit tests: `*_test.go` throughout (100+ files)
- Examples: `example_test.go`
- Property tests: `internal/quick` package
- Key test files:
  - `parquet_test.go` - Core functionality
  - `writer_test.go` - Writer specifics (91KB)
  - `reader_test.go` - Reader specifics
  - `merge_test.go` - Merge operations
  - `convert_test.go` - Schema conversion

## Debugging

Set environment variable:
```bash
PARQUETGODEBUG=1
```

## Important Implementation Details

### Performance Optimizations
- SIMD assembly for AMD64 (dictionary ops, page bounds, ordering)
- Zero-copy via interface-based design
- Memory pooling with `BufferPool`
- Async reading with `ReadModeAsync`

### Schema Tags
Struct field tags control parquet behavior:
```go
type Record struct {
    ID   int64  `parquet:"id"`
    Name string `parquet:"name,optional"`
    Data []byte `parquet:"data,snappy"`
}
```

### Encoding Options
- plain, rle, delta (binary-packed, length byte array, byte array)
- bytestreamsplit (float optimization)
- dictionary encoding

### Compression Codecs
snappy, gzip, brotli, zstd, lz4, uncompressed

## Recent Development Focus
- Bug fixes: panic in Group.GoType(), json.RawMessage handling, repetition levels
- New features: Geometry/Geography types, VARIANT logical type
- Performance: GenericWriter optimizations
- Stability: SortingWriter improvements

## Common Bug Areas
- Schema conversion edge cases (`convert.go`)
- Nested structure handling (repetition/definition levels)
- Memory management in streaming writes
- Page statistics and index handling
- Dictionary encoding with nulls
- Interface type handling in dynamic value mapping (`value.go`, `row.go`)

## Dependencies (Runtime)
- Compression: brotli, gzip, lz4, zstd libraries
- Encoding: bitpack, jsonlite
- Types: google/uuid, go-geom (geometry)
- Serialization: protobuf
