io.Reader and io.Writer

Two interfaces — five lines of Go between them — underpin almost every Go I/O operation. Files, sockets, pipes, in-memory buffers, gzip streams, encryption wrappers, network responses — all read and written through the same shape. Master these once and the whole standard library composes.

1 · The intuition

Two interfaces, both with one method each:

go io · the entire contract

type Reader interface {
    Read(p []byte) (n int, err error)
}

type Writer interface {
    Write(p []byte) (n int, err error)
}

Read fills a buffer with up to len(p) bytes, returns how many. Write consumes a buffer, returns how many bytes were written. That's it. Everything else — files, sockets, in-memory readers, encryption, compression — is a type satisfying one or both of these.

Why this works. Single-method interfaces compose. A gzip.Reader wraps an underlying io.Reader and presents one too. bufio.Reader wraps an io.Reader and adds buffering. io.LimitReader wraps to cap how many bytes you can read. The whole I/O stack is a chain of these adapters.

2 · Try it — read a file

go main.go · the basics

123456789101112131415161718192021222324

package main

import (
    "fmt"
    "io"
    "strings"
)

func main() {
    var r io.Reader = strings.NewReader("hello, world")

    buf := make([]byte, 6)
    for {
        n, err := r.Read(buf)
        if n > 0 {
            fmt.Printf("read %d bytes: %q
", n, buf[:n])
        }
        if err == io.EOF {
            fmt.Println("done")
            break
        }
    }
}

Read returns (n, nil) when data arrives, (0, io.EOF) when there's no more. Always handle both n > 0 and err — Read may return both: "here's some bytes, and we're done".

3 · The helpers — io.ReadAll, io.Copy, io.LimitReader

go main.go · the everyday tools

123456789101112131415161718192021222324252627282930

package main

import (
    "fmt"
    "io"
    "os"
    "strings"
)

func main() {
    // Read everything into memory — careful with large inputs
    data, _ := io.ReadAll(strings.NewReader("hello"))
    fmt.Println(string(data))

    // Stream from reader to writer
    src := strings.NewReader("piped through io.Copy
")
    io.Copy(os.Stdout, src)

    // Bound the read — prevents an attacker uploading 1TB
    capped := io.LimitReader(strings.NewReader("long string"), 4)
    out, _ := io.ReadAll(capped)
    fmt.Println(string(out))  // "long"

    // TeeReader — splits the stream
    var copy strings.Builder
    tee := io.TeeReader(strings.NewReader("hi"), &copy)
    io.Copy(io.Discard, tee)
    fmt.Println("captured:", copy.String())
}

4 · bufio — buffer your I/O

Raw Read can return as little as 1 byte at a time. For text processing (line scanning, word splitting) wrap in bufio.Scanner. For high-throughput binary, bufio.Reader reduces syscall count.

go main.go · line scanning

1234567891011121314151617181920212223

package main

import (
    "bufio"
    "fmt"
    "strings"
)

func main() {
    input := "line one
line two
line three
"

    scanner := bufio.NewScanner(strings.NewReader(input))
    for scanner.Scan() {
        fmt.Printf("got: %q
", scanner.Text())
    }
    if err := scanner.Err(); err != nil {
        fmt.Println("scan error:", err)
    }
}

Scanner has a token-size limit. Default 64KB per line. If you might encounter longer lines, use scanner.Buffer to bump it.

5 · Writing — bufio.Writer and friends

go main.go · buffered writes

123456789101112131415161718

package main

import (
    "bufio"
    "fmt"
    "os"
)

func main() {
    // bufio.Writer reduces syscalls for many small writes
    w := bufio.NewWriter(os.Stdout)
    defer w.Flush()  // CRITICAL — without Flush, you lose buffered data

    for i := 0; i < 5; i++ {
        fmt.Fprintf(w, "line %d
", i)
    }
}

6 · The composable stack

Real Go I/O is a chain of adapters. Here's a typical pipeline:

go main.go · gzip + base64 + http

123456789101112131415

// Reading a gzip-compressed HTTP body and decoding it as JSON
resp, _ := http.Get("https://api.example.com/data.gz")
defer resp.Body.Close()

gz, _ := gzip.NewReader(resp.Body)
defer gz.Close()

var data MyStruct
if err := json.NewDecoder(gz).Decode(&data); err != nil { ... }

// resp.Body         is an io.ReadCloser (network)
// gz                wraps it as io.Reader (decompression)
// json.NewDecoder   wraps it once more (parsing)
// Each layer reads from the layer beneath. No buffering of the
// full response in memory.

7 · From the wild

go net/http · response body is an io.ReadCloser

1234567891011

type Response struct {
    Body io.ReadCloser   // not io.Reader — ReadCloser embeds Closer
    // ...
}

// Every HTTP client follows the pattern:
resp, err := http.Get(url)
if err != nil { return err }
defer resp.Body.Close()    // must close, or connection leaks

body, err := io.ReadAll(resp.Body)

From the wild: go standard library · BSD-3-Clause

8 · Common mistakes

Not closing. io.ReadCloser values (file, response.Body) must be closed. defer x.Close() right after acquisition.
io.ReadAll on untrusted input. Reads until EOF. An attacker can stream gigabytes. Use io.LimitReader first.
Forgetting Flush. Buffered writers lose data if you exit without flushing. Always defer w.Flush().
Not checking n. Read may return fewer bytes than requested. Always use buf[:n], not buf.
Confusing io.EOF as an error. EOF is normal end-of-stream. errors.Is(err, io.EOF) at loop end is the right pattern.

9 · Exercises (~10 min)

Word counter. Read a file with bufio.Scanner, split per word with scanner.Split(bufio.ScanWords), count occurrences in a map.
Pipe through gzip. Read a file, compress to another file via gzip.Writer. Verify with gunzip CLI.
HTTP body bounded. Make an HTTP request. Wrap resp.Body in io.LimitReader with 1MB cap. Verify you can't read more even from a 10MB response.
Implement io.Reader. Define type CountingReader struct { r io.Reader; n int }. Forward Read; count bytes. Use to measure how much was read.

10 · When it clicks

You see io.Reader in a function signature and know it could be a string, file, network, anything.
You stack adapters (gzip + json + buffer) without thinking about how the bytes flow.
You defer Close() on every ReadCloser as muscle memory.
You never call io.ReadAll on data you don't trust the size of.

Next · Day 6 / Concept 17

net/http

A full HTTP server in five lines. The middleware pattern is just functions.

Next concept

Found this useful?