The most expensive thing in a system I used to work on was 32 bytes long. Leak it before a certain moment and it would have taken millions ($) with it. Not the servers, not the data, not the code. Literally just the bytes. They sat in the memory of a Go process, waiting for the garbage collector to get around to them.
Eight years in tech, a good part of it around security, taught me enough to know my way around this problem, even if systems programming is a place I visit rather than call home. Still, the thought showed up uninvited: this secret is just sitting there. Shouldn’t somebody wipe it?
Go 1.26 finally made “somebody” the runtime itself. The experimental runtime/secret package is two functions. Getting them took over eight years, and using them correctly is harder than it looks.
How secrets leak from memory
Process memory is not supposed to be readable by anyone else. “Supposed” carries a lot of weight there. Memory ends up in swap, in core dumps (which plenty of infra helpfully ships to third-party crash services), in cloud VM snapshots, in panic tracebacks that print raw argument words into logs. The crash dump path has a famous casualty: a Microsoft signing key slipped past redaction into a crash dump, the dump left the isolated production network, and Storm-0558 ended with forged tokens and government mailboxes read. Sometimes the isolation itself breaks: Heartbleed let strangers read server memory over TLS, and Cloudbleed had Cloudflare’s edge leaking fragments of process memory into cached HTTP responses.
How much this matters depends on how long the secret lives. A session key that exists for 200 milliseconds has a tiny exposure window; a key that sits in memory for hours is in every dump and snapshot taken during those hours. Mine was the second kind.
Why a zeroing loop doesn’t work
The obvious fix: overwrite the buffer when done. for i := range key { key[i] = 0 }.
Except the secret doesn’t live in one buffer. It gets copied into function arguments, registers, and stack spill slots the compiler never told you about. When a goroutine stack grows, the runtime copies the whole thing and frees the old one without zeroing it, stale key included. Convert []byte to string and you’ve made an immutable copy nothing can scrub. Hand the key to an AEAD and the expanded key schedule lives in internal state you have no pointer to. Your loop cleans one copy out of many; an attacker reading memory can reassemble the rest.
C at least has explicit_bzero for the narrow problem of optimizers deleting “useless” writes. Go had nothing, so people coped with hacks. This is roughly what WireGuard’s Go implementation did to clear an AEAD’s key state:
func (s *safeAEAD) clear() {
if s.aead != nil {
v := reflect.ValueOf(s.aead).Elem()
v.Set(reflect.Zero(v.Type()))
s.aead = nil
}
}
If your security depends on reflect.Zero, something upstream has failed you.
Eight years of issue #21865
That snippet is not a random example. In September 2017 Jason A. Donenfeld, the author of WireGuard, opened golang/go#21865 showing exactly that hack, “unholy” by his own description, and asking for something better. WireGuard promises forward secrecy: past traffic stays safe even if the machine is compromised later. That only holds if ephemeral keys are actually gone from memory.
The thread ran for over eight years. Every simple idea (manual zeroing, mlock enclaves, Clear() methods on crypto APIs) broke against the same wall: in Go the runtime owns memory, so only the runtime knows where the copies went, and only the runtime can erase them. What finally landed, implemented by Daniel Morsing, might be the smallest API in the standard library:
func Do(f func())
func Enabled() bool
Behind it, the scheduler, the garbage collector and stack management all had to learn a new mode. Each of those sounds routine until you hit the details. One of them: when a signal arrives, the kernel dumps every register onto the signal stack, and you can’t erase that inside the handler, because the registers get restored from it on return. The fix waits: the thread gets flagged, and the runtime erases its signal stack later, from a safe point outside the handler. Smallest API, one of the hairiest implementations.
What Do promises
Read the package docs first. They’re short and good; this is the summary plus the parts I’d flag.
secret.Do(f) runs f, meaning the whole call tree under it, in secret mode. When Do returns, every register and stack byte that tree touched has been zeroed. Heap allocations made inside get erased too, but only once the GC notices they’re unreachable (more on this below). All of it survives panics and runtime.Goexit, though a panic will appear to originate from Do itself.
The fine print:
- Linux amd64/arm64 only. Everywhere else
Dosilently just callsf. Your Mac gives zero protection while behaving identically. - Global variables written by
fare not protected. - In Go 1.26, goroutines started inside
Doare not protected either. Own section below. - Don’t encode secrets into pointer values; addresses can linger in the GC’s own bookkeeping.
The package only exists under GOEXPERIMENT=runtimesecret, so the import fails on a normal build. Hide it behind the build tag the experiment defines:
// secret_on.go
//go:build goexperiment.runtimesecret
package sealed
import "runtime/secret"
func Do(f func()) { secret.Do(f) }
// secret_off.go
//go:build !goexperiment.runtimesecret
package sealed
func Do(f func()) { f() }
Do returns nothing; errors go out through the closure, a pattern Keith Randall showed in the thread:
var out [32]byte
var err error
secret.Do(func() {
err = deriveKey(&out)
})
if err != nil {
// handle it out here, in the ordinary world
}
runtime/secret, artist’s impression.
The heap asterisk
Stack and registers are wiped before Do returns. That’s a hard guarantee. Heap allocations are wiped when the GC notices they’re unreachable, which is two conditions: you drop every reference, and the GC actually runs. An idle service might not collect for a couple of minutes (the runtime forces a GC about that often), so a “wiped” secret can survive in dead memory for minutes. Keep any reference alive (a slice header in some struct, a captured variable) and it is never erased, and nothing warns you. The docs are also upfront that allocating inside Do costs tracking memory and longer GC sweeps.
The annoying part: you can’t check where a value went. Escape analysis is a compiler implementation detail, and there is no if onHeap(x). So the discipline is fixed-size arrays passed by pointer ([32]byte, not make([]byte, 32)), results copied into caller-created storage, and a test pinning it down:
func TestDeriveOutcomeZeroAlloc(t *testing.T) {
var seed, out [32]byte
msg := []byte("client-input:42")
allocs := testing.AllocsPerRun(1000, func() {
deriveOutcome(&seed, msg, &out)
})
if allocs != 0 {
t.Fatalf("expected 0 allocations, got %v", allocs)
}
}
The test fails in CI the day a refactor or a new compiler version starts leaking the derivation onto the heap.
The garbage collector, deciding when your allocation gets to die.
Stack-only code hurts
The natural way to derive an output from a secret seed:
func deriveOutcome(seed, msg []byte) []byte {
mac := hmac.New(sha256.New, seed)
mac.Write(msg)
return mac.Sum(nil)
}
Four lines, and it allocates: hmac.New builds two hash states on the heap, both full of key material. Inside secret.Do this works, the runtime eventually erases all of it. But “eventually” is the word we’re trying to delete.
The stack-only version builds HMAC by hand from sha256.Sum256, which doesn’t allocate:
const maxMsgLen = 64 // plenty for "client-input:nonce" style messages
// deriveOutcome computes HMAC-SHA256(seed, msg) with no heap allocations.
// Everything local stays on the stack, which is exactly the memory
// secret.Do wipes on return.
func deriveOutcome(seed *[32]byte, msg []byte, out *[32]byte) {
if len(msg) > maxMsgLen {
panic("deriveOutcome: message too long")
}
var inner [64 + maxMsgLen]byte
var outer [64 + sha256.Size]byte
for i := 0; i < 64; i++ {
inner[i] = 0x36
outer[i] = 0x5c
}
for i := 0; i < len(seed); i++ {
inner[i] ^= seed[i]
outer[i] ^= seed[i]
}
n := copy(inner[64:], msg)
innerSum := sha256.Sum256(inner[:64+n])
copy(outer[64:], innerSum[:])
*out = sha256.Sum256(outer[:])
}
It matches crypto/hmac (I checked it against the stdlib output), passes the zero-alloc test, and is worse code by every normal standard. I wrote it, it works, and I still feel like I should apologize to someone.
That’s the deal today. Tolerate “erased at the next GC cycle” and your code stays normal. Demand “erased before Do returns” and you’re writing C-flavored Go: fixed buffers, out-parameters, no fmt, no strings. Fine for one small derivation function, a serious project to retrofit through a big call tree.
My 32 bytes
I worked on a backend where certain outcomes are derived from a server-side seed under a commit-reveal scheme. Before a session starts, the server publishes a hash of the seed. Outcomes during the session are derived from the seed plus client input, an HMAC like the one above. When the session ends, the seed is revealed and anyone can re-derive the outcomes to check nothing was manipulated.
The scheme holds only if nobody learns the seed before the reveal, and the seed has to live from commit to reveal. Sometimes that’s hours; sometimes the reveal only comes with a rotation nobody is in a hurry to trigger, which makes it effectively open-ended. How to store and rotate such seeds is a big, opinionated topic for another day; in that system it sat encrypted at rest. What runtime/secret fixes is narrower: every touch of the plaintext smears copies across stacks and registers. So the I/O stays outside, and the plaintext touchpoints get wrapped:
func (s *Server) Outcome(ctx context.Context, sessionID string, clientInput []byte) ([32]byte, error) {
if len(clientInput) > maxMsgLen {
return [32]byte{}, errInputTooLong
}
encSeed, err := s.store.EncryptedSeed(ctx, sessionID) // fetch ciphertext, no secrets yet
if err != nil {
return [32]byte{}, err
}
var out [32]byte
secret.Do(func() {
var seed [32]byte
if err = decryptSeed(encSeed, &seed); err != nil {
return
}
deriveOutcome(&seed, clientInput, &out)
})
return out, err
}
out is created by the caller and survives. The seed and every intermediate exist only inside the wiped region. A core dump taken between requests now contains the encrypted seed and nothing else, as long as decryptSeed keeps to the same stack-only discipline.
Notice what this turns the problem into: the long-lived secret never exists in memory as plaintext, and what Do protects is ephemeral, the copies alive during one derivation. That framing matters, since even the maintainer of Go’s crypto libraries is on record in the thread as sceptical of wiping long-term secrets; ephemeral material is the case everyone agrees on.
Trust, but core-dump
Given the silent no-op on other platforms and the heap asterisk, don’t believe this package. Catch it in the act. In a test build, use a seed with a recognizable canary pattern instead of random bytes, run the workload, dump, grep:
$ gcore -o dump $(pgrep myapp)
$ grep -c --binary-files=text "CANARY-7f3a" dump.*
0
Run the same check on a build with the wrapper compiled out and you should get hits. That’s the control proving the test can find the secret at all.
A dump is a snapshot, though. Taken after the request, it proves the secret didn’t linger, but it can miss a transient leak that got overwritten before you dumped. For proper paranoia, scan /proc/<pid>/mem in a loop between requests. The runtime tests itself the same way from the inside; see secret_test.go.
The goroutine hole
In Go 1.26, secret mode does not extend to goroutines started inside Do. No panic, no vet warning, they just run unprotected. Since any library you call inside Do may use a goroutine internally, the protection can leak through a dependency you never see. Easy to check with Enabled():
secret.Do(func() {
fmt.Println("inside Do:", secret.Enabled())
done := make(chan struct{})
go func() {
defer close(done)
fmt.Println("inside goroutine:", secret.Enabled())
}()
<-done
})
On my linux/arm64 box:
$ GOEXPERIMENT=runtimesecret go run main.go # Go 1.26
inside Do: true
inside goroutine: false
$ GOEXPERIMENT=runtimesecret go1.27rc2 run main.go
inside Do: true
inside goroutine: true
The second run is the good news. Go 1.27, due out around when this post goes up, makes goroutines inherit secret mode (#76477). Is the 1.26 behavior scandalous? Not really, and it wasn’t an oversight: inheritance was debated in the thread and rejected, the argument being that implicit state makes it harder to audit which code actually runs protected. The direct call path is still wiped either way. But semantics reversing between releases is what “experimental” means.
What it costs
Two currencies. Per call: entering secret mode plus zeroing the used stack on exit. Deferred: every tracked heap allocation adds bookkeeping and sweep work, paid later during GC, easy to miss in a microbenchmark.
var sink [32]byte
func BenchmarkDirect(b *testing.B) {
var seed [32]byte
msg := []byte("client-input:42")
for b.Loop() {
deriveOutcome(&seed, msg, &sink)
}
}
func BenchmarkInsideDo(b *testing.B) {
var seed [32]byte
msg := []byte("client-input:42")
for b.Loop() {
secret.Do(func() {
deriveOutcome(&seed, msg, &sink)
})
}
}
func BenchmarkDoEmpty(b *testing.B) {
for b.Loop() {
secret.Do(func() {})
}
}
// BenchmarkInsideDoHeap is the allocating hmac.New version, for the
// "heap tax" comparison: watch allocs/op and B/op, and remember the
// erase cost is paid later, during GC sweeps.
func BenchmarkInsideDoHeap(b *testing.B) {
var seed [32]byte
msg := []byte("client-input:42")
for b.Loop() {
secret.Do(func() {
mac := hmac.New(sha256.New, seed[:])
mac.Write(msg)
mac.Sum(sink[:0])
})
}
}
$ GOEXPERIMENT=runtimesecret go test -bench . -benchmem ./derive/
goos: linux
goarch: arm64
pkg: go-runtime-secret-post/derive
BenchmarkDirect-4 5604813 226.9 ns/op 0 B/op 0 allocs/op
BenchmarkInsideDo-4 4328854 299.7 ns/op 0 B/op 0 allocs/op
BenchmarkDoEmpty-4 10411758 100.2 ns/op 0 B/op 0 allocs/op
BenchmarkInsideDoHeap-4 1000000 1846 ns/op 480 B/op 5 allocs/op
PASS
ok go-runtime-secret-post/derive 5.464s
Whatever your hardware says, the shape holds: one HMAC per request disappears against any workload with a network in it. Wrap loops in a single Do rather than each iteration, and don’t allocate inside.
Do you need this?
Almost certainly not. It’s aimed at crypto library authors, so that crypto/tls and the WireGuards of the world can make forward secrecy real while you inherit the protection without importing anything. The exception is an application that itself holds a long-lived, high-value secret: a commit-reveal seed, a master key that unwraps other keys. If you recognize yours there, you knew before reading this. If you’re unsure, regular hygiene (secrets out of logs, encrypted swap, core dumps disabled) buys far more per unit of effort.
And it’s experimental in the full sense: gated behind GOEXPERIMENT, outside the compatibility promise, Linux-only with a silent fallback, still moving. There’s already a proposal (#76795) for a secret.Exempt to mark allocations that should survive Do. Wrap it behind a build tag and expect to chase upstream.
A postscript for the maximalists: instead of shortening a secret’s lifetime you can attack its exposure, keeping it encrypted even while resident in memory and decrypting only at the moment of use. Against Cloudbleed-class adjacency leaks that’s arguably the only mitigation that helps. It’s also a post of its own.
Eight years, two functions, one very tired issue thread. And for the first time, a Go process that can honestly say the secret is gone.