The learning loop we lost

Writing code without agents took much longer. But that slower process also helped us learn. While implementing a feature, we had to follow the main flows, work across different parts of the codebase, and deal with edge cases. By the time we finished, we usually had a better understanding of how the system worked.

It was not typing the code that gave us this knowledge. It was the time we spent inside the system.

Agents have changed this relationship. They can generate and modify code much faster than we can absorb the changes. The speed of production has increased, but the speed at which our brains learn how a system works has not.

We may understand each change when we review it. But then we move on to the next task, the next plan, and the next implementation. Each change makes sense on its own, while our understanding of how all the pieces fit together slowly weakens or becomes outdated.

What planning and review cannot replace

An obvious counterargument is that we were already building software through small pull requests before agents. We were already reviewing changes made by other people and moving from one task to the next.

That’s true, but I think there’s still an important difference.

When colleagues left comments on our pull requests, we had to understand the feedback, think through the problem, discuss it, and change the code. Sometimes we disagreed. Sometimes the discussion showed that everyone had missed something. The process was slower, but it gave us time to sit with the problem and learn from it.

Now we can pass the same comment directly to an agent. The agent updates the code, we check that the issue is resolved, and everyone moves on. The feedback loop still exists, but we may no longer work through it ourselves. The problem gets fixed, but the learning opportunity is gone.

It doesn’t have to be this way. We can still stop and think through the feedback ourselves. But the easiest path is to let the agent deal with it, especially when many other tasks are waiting.

Another counterargument is that we now spend more time writing requirements, specifications, and architecture plans. If we design a change and review its implementation plan, shouldn’t we already understand the system?

That’s partly true, but planning, implementation, and real-world use give us different kinds of knowledge.

A specification tells us what we want to build. Implementation shows us how that idea fits into the existing codebase. Real-world use shows us whether it actually works.

During implementation, we often discover hidden dependencies, old assumptions, unexpected states, and edge cases that we missed during planning. It’s almost impossible to predict everything in advance.

An agent can find these problems too, but we may see only the final solution or a short summary. The information is available, but we may not spend enough time with it to fully understand or retain it.

Faster delivery, weaker judgement

Maybe we’re not only losing knowledge of a particular system but also getting less practice using our own technical judgement.

Judgement develops through making decisions, seeing their results, and fixing our mistakes. When we face a challenging problem and ask an agent for an answer before thinking it through ourselves, we skip part of that practice.

The agent may give us a good answer, and we may understand it when we read it. But understanding someone else’s reasoning isn’t always the same as doing the reasoning ourselves.

I’ve noticed this in myself. When I face a hard problem, my first reaction is usually to ask an agent. Thinking it through on my own feels slower and more uncomfortable. My brain naturally looks for a shortcut.

At the same time, agents don’t always reduce the number of decisions we need to make. Because they increase the speed of production, they can create many more decisions for us to review. We may write less code ourselves while having to approve more plans, changes, and technical choices throughout the day.

This creates a dangerous cycle. Faster production creates more decisions. More decisions create mental fatigue. Fatigue makes us delegate more thinking to agents. Less practice weakens our judgement, making us depend on agents even more.

I often feel more mentally drained at the end of a working day than I did when development moved more slowly. Constantly switching between tasks and reviewing so many decisions doesn’t feel sustainable. If we don’t intentionally slow down, it’s easy to keep generating, keep approving, and gradually lose the ability to judge the work properly.

Knowing when to slow down

The obvious way to break this cycle is to think first and ask the agent second. We need to form our own view, make some decisions ourselves, and then use the agent to challenge or improve our thinking.

But this is harder than it sounds.

Once agents make faster delivery possible, that pace can quickly become expected. Product teams and management then start planning around it, even though our ability to think, review, and make careful decisions hasn’t increased at the same rate.

An engineer who stops to think deeply can start to feel like the bottleneck. The pressure no longer comes only from our own desire to take the easier path. It also comes from an environment that rewards visible output more than careful thinking.

This creates a hidden risk for companies. The same process that increases a company’s output may weaken its ability to understand and control what it produces. In the short term, this looks like higher productivity. The cost may become visible only later, when the system behaves unexpectedly and too few people understand it well enough to question what was built or take control.

Maintaining control doesn’t mean thinking deeply about every change an agent makes. There are too many, and doing so would remove much of the value agents provide. The challenge is to know where speed is safe and where we still need to slow down and think for ourselves.

Some decisions are local, easy to reverse, and safe to delegate. Others affect the whole system and may be expensive to undo. Those decisions still require independent human judgement.

This is where system design becomes especially important. Decisions about how the parts fit together affect how much the system costs, how secure and reliable it is, how well it performs, and how difficult it will be to change.

The effects of these decisions are often difficult to see immediately. A design can seem reasonable today and still create serious problems months later. The code works, the tests pass, and the feature is released. That doesn’t mean the decision was good for the system as a whole.

Agents can help us explore options and identify tradeoffs. But we still need enough understanding to judge their suggestions. If we delegate both the reasoning and the final decision, we risk becoming a proxy between the agent and the codebase. We may approve changes without knowing whether they make sense at all.

The feedback we cannot speed up

Even when we give ourselves enough time to think before making an important decision, some answers only come from seeing the system in use.

We need to see how people use it, how it behaves under load, how much it costs, how it fails, and how difficult it is to change later. This is how we learn whether we made a good decision.

Agents can write tests, simulate users, and explore possible failures. All of this can speed up some feedback loops, but it can’t show us everything. Real users, production incidents, changing requirements, maintenance, and time still teach us things that are difficult to predict.

Agents can make implementation faster, but we still have to wait for real-world feedback.

If a system changes too quickly, we face two problems. First, we don’t have enough time to understand its current state. Second, by the time we learn whether an earlier decision was good, the system may already have changed several times. We may receive feedback about a version of the system that no longer exists.

We can now build systems faster than we can think about them, and change them faster than we can learn from them.

So, do we still understand the systems we build?

We can, but it no longer happens automatically. In the past, our understanding and judgement often developed as we implemented changes, responded to feedback, and saw the results of our decisions. Now we have to make sure those learning loops still happen.

The answer isn’t to stop using agents or to make every decision ourselves. We need to know what we can safely delegate. For important decisions, we need to stay involved long enough to learn from the results.

When code becomes easy to generate, producing more of it may no longer be the most valuable skill. What matters more is knowing when to slow down, when to think first, and when the agent’s answer isn’t enough.

That may be the real moat.

We require Codex to parse data shapes at the boundary, but are not prescriptive on how that happens (the model seems to like Zod, but we didn’t specify that specific library).

“Parse data shapes at the boundary” links to Alexis King’s Parse, don’t validate . A 2019 idea from Haskell turns out to be one of the rules they rely on to keep an agent-written codebase under control.

The whole idea is what you get back after the check:

Here’s what that looks like in Rust. Java gets the first example, because that’s where I learned to validate.

All the code in this post is in the rd-blog repository .

Validation in Java

If you’ve built Spring Boot applications, this should look familiar:

public record DeployRequest(
    @Pattern(regexp = "[a-z0-9-]{3,30}") String service,
    @Pattern(regexp = "dev|staging|production") String environment,
    @Positive int replicas
) {}

@PostMapping("/deploys")
public ResponseEntity<?> deploy(@Valid @RequestBody DeployRequest req) {
    deployService.submit(req);
    return ResponseEntity.accepted().build();
}

There’s nothing wrong with this code. The important part is that validation does not change the type.

req.service() returns a String before @Valid runs and a String after. A request that passed validation has the same type as one that has not been validated, so nothing in the value itself tells the compiler which one it has.

That tradeoff makes sense in Java, where introducing wrapper types adds more object types and framework friction. In Rust, newtypes are much cheaper, so you can keep the result of validation in the type instead.

Validation vs parsing

Compare these two signatures:

fn validate(req: &DeployRequest) -> Result<(), DeployError>

fn parse(req: DeployRequest) -> Result<Deploy, DeployError>

The return types differ on success, and that difference is the whole idea. Validation answers the question and then throws away what it learned. Parsing answers the same question and returns a value whose type represents data that passed those checks. Downstream code can require that parsed type instead of accepting the raw request.

Rust makes that loss explicit in the signature. () is the type with exactly one value, so it carries no information. A function returning Result<(), E> has nothing to return on success. If the function’s job is to check a value and it learns something useful about it, returning () throws that information away.

There is another Rust-specific difference in those signatures: validate borrows the request, while parse takes ownership of it. A successful parse consumes the raw DeployRequest and returns a Deploy, so the caller can’t accidentally keep using the original request afterward.

The standard library already does this

The standard library often keeps useful guarantees in the type it returns. Nobody writes is_utf8(&[u8]) -> bool, because std::str::from_utf8 returns a &str instead of a yes/no answer. &str is a &[u8] plus a proof that the bytes are valid UTF-8, so any function that takes &str can rely on that guarantee without checking the bytes again.

NonZeroU32 does the same thing for numbers. It’s a u32 plus a guarantee that it isn’t zero, and NonZeroU32::new checks that constraint when you construct one:

let workers = NonZeroU32::new(count)?;
let per_worker = total / workers;  // no division-by-zero case to handle

A function that takes NonZeroU32 no longer has to handle zero as an input.

The same code in Rust

Here’s the Java example in Rust:

pub struct DeployRequest {
    pub service: String,
    pub environment: String,
    pub replicas: u32,
}

pub fn validate(req: &DeployRequest) -> Result<(), DeployError> {
    if !is_service_name(&req.service) {
        return Err(DeployError::BadServiceName);
    }

    if !is_environment(&req.environment) {
        return Err(DeployError::UnknownEnvironment);
    }

    if req.replicas == 0 {
        return Err(DeployError::ZeroReplicas);
    }

    Ok(())
}

The function now knows that replicas is not zero, but the return type has nowhere to keep that information.

The problem appears when another function needs to rely on that check:

pub fn memory_per_replica_mb(req: &DeployRequest, total_mb: u32) -> u32 {
    total_mb / req.replicas
}

Nothing in the signature says that validate was called earlier, so this test compiles, type-checks, and panics:

#[test]
#[should_panic(expected = "attempt to divide by zero")]
fn nothing_stops_you_from_skipping_the_check() {
    let r = req("checkout-api", "production", 0);
    let _ = memory_per_replica_mb(&r, 1024);
}

Refining the type

The fix for DeployRequest is to give each field its own type, keep the inner value private, and only allow construction through a function that can fail.

The important boundary is the module. Code outside the module can’t construct these types directly, so it has to use their parse functions or constructors. That means each check only needs to be correct in one place.

All three types live in the same module and use the same error type:

pub mod domain {
    use std::num::NonZeroU32;

    #[derive(Debug, PartialEq, Eq)]
    pub enum ParseError {
        BadServiceName,
        UnknownEnvironment,
        ZeroReplicas,
    }

    // domain types below
}

Start with the service name:

#[derive(Debug, Clone, PartialEq, Eq, Hash)]
pub struct ServiceName(String);

impl ServiceName {
    pub fn parse(raw: &str) -> Result<Self, ParseError> {
        let canonical = raw.trim().to_ascii_lowercase();

        let len_ok = (3..=30).contains(&canonical.len());
        let chars_ok = canonical.chars().all(|c| {
            c.is_ascii_lowercase() || c.is_ascii_digit() || c == '-'
        });

        if len_ok && chars_ok {
            Ok(Self(canonical))
        } else {
            Err(ParseError::BadServiceName)
        }
    }

    pub fn as_str(&self) -> &str {
        &self.0
    }
}

The field is private, so code outside the module can’t create a ServiceName directly. It has to call ServiceName::parse.

The parse also canonicalises before checking. It trims and lowercases the input, validates that value, and stores the same value. If you validate first and normalise later, you may end up storing something different from what you actually checked.

as_str() makes the conversion back to a plain string explicit. I prefer that over Deref<Target = str>, which would let a ServiceName behave like a str implicitly in many places.

The environment uses the same pattern, but the valid values come from a fixed set:

const ENVIRONMENTS: [&str; 3] = ["dev", "staging", "production"];

#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
pub struct Environment(&'static str);

impl Environment {
    pub fn parse(raw: &str) -> Result<Self, ParseError> {
        let canonical = raw.trim().to_ascii_lowercase();

        ENVIRONMENTS
            .into_iter()
            .find(|name| *name == canonical)
            .map(Environment)
            .ok_or(ParseError::UnknownEnvironment)
    }

    pub fn name(self) -> &'static str {
        self.0
    }
}

The type is named after what the value means, not what it is made of. Environment isn’t “a string someone typed”. It’s an environment this system deploys to. Add another environment later and ENVIRONMENTS changes, while the meaning of Environment stays the same.

The replica count is slightly different. serde has already parsed it into a u32, so Replicas::new only has to check the remaining rule: the value must not be zero. For that, it can reuse a type from std:

#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)]
pub struct Replicas(NonZeroU32);

impl Replicas {
    pub fn new(raw: u32) -> Result<Self, ParseError> {
        let count = NonZeroU32::new(raw).ok_or(ParseError::ZeroReplicas)?;
        Ok(Self(count))
    }

    pub fn get(self) -> NonZeroU32 {
        self.0
    }
}

Replicas wraps NonZeroU32, so zero cannot be represented by the type. get returns that NonZeroU32 rather than a plain u32, so a function that receives it does not have to check for zero again.

These types should not derive Default unless the default value is valid. An empty String, for example, is not a valid ServiceName.

Now put those parsed fields together:

pub struct Deploy {
    pub service: ServiceName,
    pub environment: Environment,
    pub replicas: Replicas,
}

impl TryFrom<DeployRequest> for Deploy {
    type Error = ParseError;

    fn try_from(req: DeployRequest) -> Result<Self, Self::Error> {
        Ok(Self {
            service: ServiceName::parse(&req.service)?,
            environment: Environment::parse(&req.environment)?,
            replicas: Replicas::new(req.replicas)?,
        })
    }
}

Deploy checks the same three things as validate. The difference is that, on success, it returns a value that keeps what those checks learned in its types.

Before parsing:

pub fn memory_per_replica_mb(req: &DeployRequest, total_mb: u32) -> u32 {
    total_mb / req.replicas
}

After parsing:

pub fn memory_per_replica_mb(deploy: &Deploy, total_mb: u32) -> u32 {
    total_mb / deploy.replicas.get()
}

The function no longer depends on validation having happened somewhere earlier. Replicas already guarantees that the divisor is non-zero.

try_from also consumes the request. After the conversion, the raw DeployRequest has moved out of scope, so you can’t accidentally keep using the unchecked form. Neither Haskell nor Java gives you that by default, because in both the original value stays in scope.

Outside the domain module, constructing an invalid ServiceName is a compile error rather than a review comment:

let sneaky = ServiceName("Not A Service".to_string());
// error[E0603]: tuple struct constructor `ServiceName` is private

Refining is one way to put knowledge into a type. It stops invalid values from being constructed. Another is restructuring, which stops invalid combinations from being represented at all.

Removing invalid states

Here’s what you get by mapping a database row straight into a struct:

pub struct DeploymentRow {
    pub is_running: bool,
    pub is_finished: bool,
    pub finished_at_ms: Option<u64>,
    pub failure_reason: Option<String>,
}

4 fields, 16 combinations, and only a few that describe a real deployment. A deployment can be marked as both running and finished, or finished without a timestamp, and the type system accepts it. Any code that reads this struct then has to deal with those invalid combinations too.

An enum can encode only the states the system actually allows:

pub enum Deployment {
    Queued,
    RollingOut { started_at_ms: u64 },
    Live { since_ms: u64 },
    RolledBack { reason: String },
}

4 states, 4 variants, each carrying exactly the data that state needs. A queued deployment has no finished_at_ms, so code can’t try to use one. There are no invalid combinations to handle, and adding a fifth state later makes every incomplete match fail to compile.

This also removes the duplication between is_finished and finished_at_ms. Both represented the same fact, but they could disagree.

The standard library uses this idea too. In Java, a comparator returns an integer, even though only negative, zero, and positive matter. Rust returns Ordering instead, with exactly three variants: Less, Equal, and Greater.

Put the rule in the type

Both newtypes and enums move rules out of control flow and into the type itself.

With validation, a function can still accept the raw type after the check has run:

fn submit(req: &DeployRequest)

Nothing in that signature says whether req passed validation.

With parsed domain types, the requirement becomes part of the function signature:

fn submit(deploy: &Deploy)

Now callers have to provide a Deploy, and the function can rely on the guarantees that come with it.

That is the practical difference. With validation, the caller has to make sure the check happened first. With parsing, the function can require a type that could only be produced after those checks passed.

Shotgun parsing

So far, the benefit has been about types. The next problem is what happens when code starts changing state before validation is finished.

Shotgun parsing is when input checks are spread through the same code that acts on the input. The program starts doing work before it has finished deciding whether the input is valid.

The name comes from the 2016 LangSec paper The Seven Turrets of Babel : instead of parsing the input once at the start, checks are added wherever they happen to be needed.

Here’s a deploy handler that does it:

pub fn deploy_shotgun(
    world: &mut World,
    req: &DeployRequest,
) -> Result<(), DeployError> {

    if !is_service_name(&req.service) {
        return Err(DeployError::BadServiceName);
    }

    // effect 1
    world.deployments.push(req.service.clone());

    if !is_environment(&req.environment) {
        return Err(DeployError::UnknownEnvironment);
    }

    // effect 2
    let route = format!("{} -> {}", req.environment, req.service);
    world.traffic.push(route);

    if req.replicas == 0 {
        // too late
        return Err(DeployError::ZeroReplicas);
    }

    let entry = format!("deployed {} to {}", req.service, req.environment);
    world.audit.push(entry);
    Ok(())
}

Send it a valid service name, a real environment, and a replica count of 0:

$ cargo run
shotgun     -> rejected with ZeroReplicas
  deployments : ["checkout-api"]
  traffic     : ["production -> checkout-api"]
  audit       : []

parse-first -> rejected with ZeroReplicas
  deployments : []
  traffic     : []
  audit       : []

The deploy was rejected, but the program has already changed its state. The deployment was recorded and traffic was updated. The audit log is empty too, because the audit write came after the failing check, so the one record that would tell you what happened is the one record you don’t have.

The paper states the consequence directly:

Shotgun parsing necessarily deprives the program of the ability to reject invalid input instead of processing it. Late-discovered errors in an input stream will result in some portion of invalid input having been processed, with the consequence that program state is difficult to accurately predict.

The problem isn’t that any individual check is wrong. You can fix this example by moving every check before the first side effect, but once validation and processing are mixed together, later changes can easily break that ordering again.

The safer approach is to separate the two steps: first parse the input, then act on the parsed value:

pub fn deploy(world: &mut World, deploy: &Deploy) {
    world.deployments.push(deploy.service.as_str().to_owned());
    world.traffic.push(format!(
        "{} -> {}",
        deploy.environment.name(),
        deploy.service.as_str()
    ));
    world.audit.push(format!(
        "deployed {} to {}",
        deploy.service.as_str(),
        deploy.environment.name()
    ));
}

Then at the boundary:

let parsed = Deploy::try_from(req)?;
deploy(&mut world, &parsed);

The operation that changes state takes a Deploy, not a DeployRequest. Raw input therefore has to be parsed before it can be passed to that operation. The compiler can’t stop unrelated code from changing World, but it can make this deployment path require parsed input.

So “check before you act” becomes something the compiler enforces instead of a rule developers have to remember.

Why agents drift into shotgun parsing

Nobody sits down to write a shotgun parser. It happens gradually. A new requirement arrives, and its check gets added where the relevant data is already available, often somewhere in the middle of the processing. Over time, a codebase that relies on separate validation checks can drift into this pattern.

Coding agents often make changes where the relevant data is already available instead of restructuring the whole function. If you ask for “production deploys need an approver,” an agent may find the branch where the environment is checked and add another if there. That change makes sense on its own, but repeated changes like this are how shotgun parsing grows.

The way this happens hasn’t changed. The speed has. Two years of drift can now happen in an afternoon.

The same logic explains why types beat instructions here. A rule in your AGENTS.md costs context on every turn, competes with everything else in the file, and can’t be enforced. A type in the signature is in context whenever the agent reads the function, needs no reminding, and fails the build when violated.

Types can enforce structure, but they can’t tell whether the structure is a good one. An agent can still make a poor design compile, so you still need to read the diff. The difference is that you no longer have to check whether validation happened before the code changed state. You can focus on whether the types themselves represent the right rules.

Parsing at the boundary

The raw request type only needs to describe the input format. serde can then convert it into Deploy before the rest of the program sees it.

The TryFrom implementation from earlier already does the conversion. serde’s try_from attribute tells it to deserialise a DeployRequest first and then call Deploy::try_from:

#[derive(Deserialize)]
pub struct DeployRequest {
    pub service: String,
    pub environment: String,
    pub replicas: u32,
}

#[derive(Debug, Clone, PartialEq, Eq, Hash, Deserialize)]
#[serde(try_from = "DeployRequest")]
pub struct Deploy { /* unchanged */ }

DeployRequest derives Deserialize and mirrors the JSON, nothing more. Deploy is the domain type. serde first builds a DeployRequest, passes it to your TryFrom implementation, and returns either a Deploy or your own error. ParseError needs a Display impl for that last part, since serde reports the failure as a string:

let json = r#"{
    "service": "checkout-api",
    "environment": "production",
    "replicas": 0
}"#;
let err = serde_json::from_str::<Deploy>(json).unwrap_err();
// replicas must be at least 1

After from_str succeeds, the rest of the program only sees Deploy, not DeployRequest. That keeps the raw input type at the boundary and prevents handlers from accepting unparsed requests by accident.

That error message matters for agents too. If parsing fails during a build or test, the agent may see the error directly. A message like replicas must be at least 1 tells it what constraint was violated and what needs to change. An error with no useful message gives it much less to work with.

The derive that skips your constructor

This trap is Rust-specific, and it’s easy to miss.

#[derive(Debug, Deserialize, PartialEq)]
pub struct Replicas(u32); // private field

impl Replicas {
    pub fn new(raw: u32) -> Result<Self, &'static str> {
        if raw == 0 {
            return Err("replicas must be at least 1");
        }
        Ok(Self(raw))
    }

    pub fn get(&self) -> u32 {
        self.0
    }
}

It looks safe, but it isn’t. The generated Deserialize implementation can construct Replicas directly even though its field is private. That means it can bypass the constructor entirely:

#[test]
fn derive_bypasses_constructor() {
    assert!(Replicas::new(0).is_err());

    // ...and yet deserialisation accepts zero.
    let smuggled: Replicas = serde_json::from_str("0").unwrap();
    assert_eq!(smuggled.get(), 0);
}

serde has now created a Replicas(0) even though the constructor rejects zero. Private fields prevent code in other modules from constructing the type directly, but generated code in the same module can still do it. The fix is to tell serde to deserialise through TryFrom instead:

Note what makes this possible. The inner type is u32, which can hold zero, so the rule lives only in the constructor. The Replicas(NonZeroU32) from earlier has no such hole, because serde deserialises the inner NonZeroU32 and that rejects zero on its own. The further you push a rule into the types, the less there is left to bypass.

#[derive(Debug, Deserialize, PartialEq)]
#[serde(try_from = "u32")]
pub struct Replicas(u32);

impl TryFrom<u32> for Replicas {
    type Error = &'static str;

    fn try_from(raw: u32) -> Result<Self, Self::Error> {
        Self::new(raw)
    }
}

This is easy to miss in agent-written code. #[derive(Deserialize)] is the usual way to make a newtype deserialisable, so an agent may add it without noticing that it bypasses the constructor. In this case, invalid input never reaches Replicas::new, so the replicas must be at least 1 error is never produced.

This is exactly the kind of rule I would enforce rather than leave in AGENTS.md. In a larger codebase, domain newtypes should not be allowed to derive Deserialize directly. A CI check can reject that pattern unless deserialisation goes through TryFrom, and a test can verify that invalid values fail through both the constructor and serde.

Tradeoffs and limits

Using domain types adds more structs and conversion code. That is usually worth it for values that cross module boundaries or can cause real damage when they are wrong. It is probably not worth doing for every field in a small program.

Using ? reports the first error and stops, but that is just one implementation choice. If you need to show several validation errors at once, you can collect them first and only construct Deploy when all checks pass.

And Result<(), E> is not always a problem. Functions like fs::write return () because they do not learn anything new about an input value. The problem is specific to validation functions that check a value and then throw away what they learned.

Where to start

Start with places where the code already hints that validation happened earlier:

1. Search for comments, assertions, or panic messages like “validated earlier”, “checked above”, or “cannot fail here”. They often mark a fact the type does not record.
2. Find functions that return Result<(), E> and only inspect their input. Ask what the function knows after it succeeds, and whether it could return a type that represents that result instead.
3. Look at functions that accept raw input and also change state. If they can return an error after changing something, move all parsing before the first change.
4. Look for structs where fields can contradict each other. An enum may represent the valid states more directly.
5. Check domain newtypes for #[derive(Deserialize)] and #[derive(Default)]. Deserialize may bypass the parser, and Default is unsafe when the default inner value does not satisfy the type’s rules.
6. Consider #[serde(deny_unknown_fields)] for request and config types. serde ignores unknown fields by default, so a typo like “replcias” can otherwise be silently ignored.

Don’t try to change everything at once. Moving one value behind a parsed domain type at one boundary is already useful.

For coding agents, keep the rules short and mechanical:

- Parse external input into domain types at the boundary. Handlers
  take parsed types, not raw request or config structs.
- A function whose only job is checking input should return what it learned,
  not `Result<(), E>`.
- Domain types keep their fields private and are created through functions
  that can fail.
- Finish parsing before changing state.

I would enforce the serde rule separately in CI. Domain newtypes should not derive Deserialize directly unless deserialisation is explicitly routed through TryFrom.

Summary

Parsing forces you to decide what your types mean, and that’s real work. It also won’t stop an agent from making a poor design compile. What it gives you is a guarantee the compiler checks at every call site, including ones that haven’t been written yet.

That is why this matters more when agents are writing more of the code. When code arrives faster than anyone can review it, the best invariants are the ones reviewers don’t have to check manually.

So --dangerously-skip-permissions stops being scary, even when the next command Claude Code runs happens to be rm -rf on your home directory.

brew install docker/tap/sbx
cd ~/my-project && sbx run claude

The architecture

The sandbox has two ways to reach the host. A workspace folder for files, and a proxy for outbound network. Nothing else gets through.

Two things make this different from “Claude in a Docker container”:

1. Kernel isolation. A container escape gets you the host kernel. A microVM escape gets you nothing.
2. All outbound traffic goes through a host-side proxy. HTTP and HTTPS can only reach domains you allow. Raw TCP, UDP, and ICMP are blocked at the network layer.

That proxy is where the interesting work happens.

The proxy does two jobs

Credential substitution. You store a secret on the host:

sbx secret set -g github -t "$(gh auth token)"

Inside the sandbox, GITHUB_TOKEN holds a placeholder value like proxy-managed. The AI agent uses that env var as it normally would. On the way out, the proxy substitutes the real token for requests to api.github.com.

Custom services work the same way. Run sbx secret set-custom and the CLI generates a placeholder like sbx-cs-<rand>. The sandbox sees only the placeholder, and the proxy swaps in the real value on requests to the matching domain.

For SSH, SSH_AUTH_SOCK is forwarded into the sandbox, so private keys stay on the host.

Network policy. Three default modes (Open, Balanced, Locked Down) plus per-domain overrides. Denied requests come back with a clear reason:

HTTP/1.1 403 Forbidden
Blocked by network policy: domain api.mailgun.net:443
  detail: no matching allow rule — blocked by default deny policy

sbx policy allow network api.mailgun.net adds it. Or put it into a kit.

Kits

A kit is a YAML file that customizes a sandbox. The minimum useful one for a project:

schemaVersion: "1"
kind: mixin
name: my-project
displayName: My project dev stack
description: Toolchain + network allowlist for my project.

commands:
  install:
    - user: "1000"
      description: Install Rust stable
      command: |
        curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | \
          sh -s -- -y --default-toolchain stable --profile default

network:
  allowedDomains:
    - "crates.io:443"
    - "static.crates.io:443"
    - "index.crates.io:443"
    - "static.rust-lang.org:443"

The install commands run once when the sandbox is created. Here they bootstrap Rust stable as the agent user. The allowedDomains list lets rustup reach the domains it needs. Without it, the install would fail under the default-deny policy.

Load it with sbx run claude --kit ./.sbx/my-kit.

The dashboard

Running sbx with no arguments opens a dashboard listing every sandbox, its state, and resource usage. Useful when you’ve left things running.

Notice that even mcp-proxy.anthropic.com shows up as blocked. The proxy makes no exception for the agent’s own vendor. Anything not in your allowlist stays out.

Syncing your ~/.claude

Your host’s ~/.claude is not mounted. Slash commands, skills, hooks, and the global CLAUDE.md won’t be there unless you copy them in. A small wrapper handles create-or-reuse, sync, and attach in one go:

#!/usr/bin/env bash
set -euo pipefail

sandbox="claude-my-project"

sbx ls --quiet | grep -qx "$sandbox" || \
  sbx create claude --name "$sandbox" --kit ./.sbx/my-kit .

for item in commands skills agents hooks CLAUDE.md settings.json; do
  [ -e "$HOME/.claude/$item" ] || continue
  sbx cp "$HOME/.claude/$item" "$sandbox:/home/agent/.claude/"
done

sbx run "$sandbox"

The tool is still experimental, and a non-trivial project will need a custom kit. Docker keeps a community kit repo with reusable mixins worth borrowing from before writing your own.

Installation

brew install vhs

VHS depends on ttyd and ffmpeg, both installed automatically by Homebrew.

Writing a tape

Here’s the tape that produced Figure 1. It configures the terminal look (font, theme, window), hides shell setup from the viewer, then creates a markdown file and renders it with glow:

Output glow-render.mp4

Require glow

Set Shell "fish"
Set FontFamily "JetBrainsMono Nerd Font Mono"
Set FontSize 40
Set LineHeight 1.0
Set Width 2400
Set Height 1256
Set Theme "Catppuccin Mocha"
Set WindowBar Colorful
Set WindowBarSize 85
Set Padding 20
Set Margin 30
Set MarginFill "#f3f5f7"
Set BorderRadius 8
Set CursorBlink false
Set Framerate 30

Hide
Type "set fish_autosuggestion_enabled 0 && clear"
Enter
Sleep 500ms
Show

Sleep 500ms
Set TypingSpeed 35ms

Type 'echo "# VHS'
Enter
Sleep 150ms
Enter
Sleep 100ms
Type "`brew install vhs`"
Enter
Sleep 100ms
Enter
Sleep 100ms
Type "Write a **.tape** script. Run **vhs demo.tape**."
Enter
Sleep 150ms
Enter
Sleep 100ms
Type "> No screen recording. No trimming. Just code."
Enter
Sleep 200ms
Type '" > notes.md'
Sleep 300ms
Enter

Sleep 1s

Ctrl+L
Sleep 500ms
Set TypingSpeed 40ms
Type "glow notes.md"
Sleep 500ms
Enter
Sleep 3s

Hide
Type "rm notes.md"
Enter
Sleep 300ms
Show

Run it:

vhs glow-render.tape

Beyond what’s shown here, VHS supports Ctrl+C, arrow keys, and Wait (blocks until expected output appears). Set LoopOffset 60% starts the GIF loop partway through so the viewer sees the result first.

Font gotcha

VHS renders in headless Chromium, not your terminal emulator. It can only use system-installed fonts. If the font isn’t found, it falls back to a default monospace and produces broken letter-spacing with wide gaps between characters.

The fix: install the font system-wide. For JetBrains Mono Nerd Font:

brew install --cask font-jetbrains-mono-nerd-font

Output formats

VHS renders GIF, MP4, and WebM. Add multiple Output lines to generate all three in one run:

Output demo.gif
Output demo.mp4
Output demo.webm

MP4 and WebM support full color and compress well. For this recording, MP4 was 114 KB, WebM 113 KB, and GIF 194 KB, with the gap growing for longer or more colorful recordings. GIF is limited to 256 colors per frame, so rich themes like Catppuccin or Dracula show grainy artifacts.

GIF is the most portable format since it works as a plain image anywhere, but it’s heavier. MP4 and WebM are smaller and sharper, with autoplay support via the <video> tag. Use MP4 or WebM for blog posts and GIF for GitHub READMEs where a click to play is unavoidable.

The volume did exist. docker volume ls confirmed it. The data just wasn’t in it.

What changed in Postgres 18

PR #1259 changed two things in the official Postgres Docker image, starting with version 18:

From the PR description:

Concretely, this changes PGDATA to /var/lib/postgresql/MAJOR/docker, which matches the pre-existing convention/standard of the pg_ctlcluster/postgresql-common set of commands, and frankly is what we should’ve done to begin with.

This also changes the VOLUME to /var/lib/postgresql, which should be more reasonable, and make the upgrade constraints more obvious.

The new directory structure enables faster pg_upgrade --link between major versions.

Why it breaks

With Postgres 16, mounting a named volume at /var/lib/postgresql/data worked because that’s where PGDATA pointed. With Postgres 18, PGDATA moved to /var/lib/postgresql/18/docker. The old mount path and the new data path are siblings, not nested:

/var/lib/postgresql/
├── data/              ← named volume mounted here (empty)
└── 18/
    └── docker/        ← PGDATA lives here (anonymous volume)
        ├── base/
        ├── global/
        ├── pg_wal/
        └── ...

The image declares VOLUME /var/lib/postgresql, so Docker creates an anonymous volume there automatically. My database files ended up in that anonymous volume. The named volume sat empty at /var/lib/postgresql/data.

The container logs confirm it. On every startup:

$ docker logs pg
The files belonging to this database system will be owned by user "postgres".
...
fixing permissions on existing directory /var/lib/postgresql/18/docker ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
...
running bootstrap script ... ok
...
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

Success. You can now start the database server using:

    pg_ctl -D /var/lib/postgresql/18/docker -l logfile start

That’s initdb running from scratch. If data had been persisted, Postgres would skip initialization and just start the server.

Reproduce it

Start Postgres 18 with the old mount path:

docker run -d --name pg \
  -e POSTGRES_DB=myapp -e POSTGRES_USER=myapp -e POSTGRES_PASSWORD=testpass \
  -v pgdata:/var/lib/postgresql/data \
  postgres:18-alpine

Insert data and verify:

$ docker exec pg psql -U myapp -c \
  "CREATE TABLE test(id int); INSERT INTO test VALUES(1);"
CREATE TABLE
INSERT 0 1

$ docker exec pg psql -U myapp -c "SELECT * FROM test;"
 id
----
  1
(1 row)

Remove the container and start a new one:

$ docker rm -f pg

$ docker run -d --name pg \
  -e POSTGRES_DB=myapp -e POSTGRES_USER=myapp -e POSTGRES_PASSWORD=testpass \
  -v pgdata:/var/lib/postgresql/data \
  postgres:18-alpine

$ docker exec pg psql -U myapp -c "SELECT * FROM test;"
ERROR:  relation "test" does not exist
LINE 1: SELECT * FROM test;
                      ^

Data is gone. No error from Docker, no warning in the logs.

docker volume ls shows the evidence:

$ docker volume ls
DRIVER    VOLUME NAME
local     4dea31d05d27...   ← new anonymous volume (fresh initdb)
local     be27b5708be1...   ← old anonymous volume (your data is here, orphaned)
local     pgdata            ← named volume (empty the whole time)

A new anonymous volume appears after each remove/create cycle. The old one stays on disk until you run docker volume prune.

Two mounts confirm the mismatch:

$ docker inspect pg \
  --format '{{range .Mounts}}{{.Name}} -> {{.Destination}}{{"\n"}}{{end}}'
pgdata -> /var/lib/postgresql/data
4dea31d05d27... -> /var/lib/postgresql

$ docker exec pg sh -c 'echo "PGDATA=$PGDATA"'
PGDATA=/var/lib/postgresql/18/docker

PGDATA sits inside the anonymous volume, not the named one.

Why data sometimes survives

Restarting the same container preserves data. Removing it doesn’t:

docker stop / docker start keeps the container and its anonymous volume in place. docker rm destroys the container, orphaning the anonymous volume. The next docker run creates a fresh one, Postgres sees an empty PGDATA, and runs initdb.

This is the same reason docker compose up (foreground, Ctrl+C to stop) preserves data while docker compose down followed by docker compose up -d doesn’t. down removes the container. Ctrl+C just stops it.

The fix

Two things need to happen: back up your Postgres 16 data, then change the volume mount.

Back up first:

docker exec pg pg_dumpall -U myapp > backup.sql

Then recreate the container with the correct mount path at /var/lib/postgresql:

docker rm -f pg
docker volume rm pgdata

docker run -d --name pg \
  -e POSTGRES_DB=myapp -e POSTGRES_USER=myapp -e POSTGRES_PASSWORD=testpass \
  -v pgdata:/var/lib/postgresql \
  postgres:18-alpine

PGDATA (/var/lib/postgresql/18/docker) is now a subdirectory of the named volume mount. Data persists across container removal because the named volume has a stable name that Docker can find and reattach.

Restore the backup:

docker exec -i pg psql -U myapp < backup.sql

Verify a single mount:

$ docker inspect pg \
  --format '{{range .Mounts}}{{.Name}} -> {{.Destination}}{{"\n"}}{{end}}'
pgdata -> /var/lib/postgresql

One mount, one volume, data persists.

If you use Docker Compose, the equivalent mount path fix:

 volumes:
-  - postgres_data:/var/lib/postgresql/data
+  - postgres_data:/var/lib/postgresql

The wrong fix

My first instinct was to override PGDATA to force Postgres back into the old mount path:

docker run -d --name pg \
  -e POSTGRES_DB=myapp -e POSTGRES_USER=myapp -e POSTGRES_PASSWORD=testpass \
  -e PGDATA=/var/lib/postgresql/data/pgdata \
  -v pgdata:/var/lib/postgresql/data \
  postgres:18-alpine

This works in the sense that Postgres will store data inside the volume. But it fights the image’s intended design and breaks the version-specific directory structure (/var/lib/postgresql/MAJOR/docker) that enables pg_upgrade --link between major versions.

Who else got burned

The same issue has been reported across multiple open-source projects after they upgraded to Postgres 18:

• nginx-proxy-manager #4811
• OWASP-BLT #4717
• Hi.Events #877

The official Docker Hub documentation now includes this warning:

Important Change: the PGDATA environment variable of the image was changed to be version specific in PostgreSQL 18 and above. For 18 it is /var/lib/postgresql/18/docker. The defined VOLUME was changed in 18 and above to /var/lib/postgresql. Mounts and volumes should be targeted at the updated location.

Summary

If you’re upgrading to Postgres 18 in Docker: back up with pg_dumpall, change your volume mount, and restore.

- -v pgdata:/var/lib/postgresql/data
+ -v pgdata:/var/lib/postgresql

Don’t override PGDATA. The image default is correct.

The breaking change is documented but easy to miss. No error message will tell you your data is being written outside the volume. The only clue is initdb running on every startup in the container logs.

After each assistant message, Claude Code pipes session state to your script’s stdin as JSON: model, context usage, rate limits, cost, workspace path, and more. Whatever the script prints to stdout becomes the status line.

Configuration

Add a statusLine field to ~/.claude/settings.json:

"statusLine": {
  "type": "command",
  "command": "~/.claude/statusline.sh",
  "padding": 0
}

The script needs to be executable (chmod +x ~/.claude/statusline.sh).

The script

This is a two-line status bar with Catppuccin Macchiato colors (matched to my Kitty theme) and Nerd Font icons:

#!/bin/bash
input=$(cat)
NOW=$(date +%s)

# ─── Catppuccin Macchiato (true color) ─────────────────────────────
MAUVE=$'\033[38;2;198;160;246m'
BLUE=$'\033[38;2;138;173;244m'
TEAL=$'\033[38;2;139;213;202m'
GREEN=$'\033[38;2;166;218;149m'
YELLOW=$'\033[38;2;238;212;159m'
RED=$'\033[38;2;237;135;150m'
PEACH=$'\033[38;2;245;169;127m'
FLAMINGO=$'\033[38;2;240;198;198m'
LAVENDER=$'\033[38;2;183;189;248m'
SKY=$'\033[38;2;145;215;227m'
OVERLAY=$'\033[38;2;110;115;141m'
SUBTEXT=$'\033[38;2;165;173;203m'
RST=$'\033[0m'
SEP=" ${OVERLAY}│${RST} "

# ─── Parse JSON (single jq call) ──────────────────────────────────
{
  read -r MODEL
  read -r DIR
  read -r PCT_RAW
  read -r CTX_SIZE
  read -r FIVE_PCT
  read -r FIVE_RESET
  read -r SEVEN_PCT
  read -r COST
} < <(echo "$input" | jq -r '
  (.model.display_name // "—"),
  (.workspace.current_dir // ""),
  (.context_window.used_percentage // ""),
  (.context_window.context_window_size // 200000),
  (.rate_limits.five_hour.used_percentage // ""),
  (.rate_limits.five_hour.resets_at // ""),
  (.rate_limits.seven_day.used_percentage // ""),
  (.cost.total_cost_usd // 0)
')

# ─── Effort level ─────────────────────────────────────────────────
EFFORT=$(jq -r '.effortLevel // "medium"' ~/.claude/settings.json 2>/dev/null)
case "$EFFORT" in
  low)    EFFORT_SEG="${SUBTEXT}▽ low${RST}" ;;
  high)   EFFORT_SEG="${PEACH}▲ high${RST}" ;;
  max)    EFFORT_SEG="${FLAMINGO}⬆ max${RST}" ;;
  *)      EFFORT_SEG="${YELLOW}◆ med${RST}" ;;
esac

# ─── Directory (clickable OSC 8) ──────────────────────────────────
DNAME="${DIR##*/}"
DIR_SEG="${BLUE}󰉋 "$'\033]8;;file://'"${DIR}"$'\033\\'"${DNAME}"$'\033]8;;\033\\'"${RST}"

# ─── Git (cached 5s, keyed by dir) ────────────────────────────────
GIT=""
if [[ -n "$DIR" ]]; then
  CF="/tmp/claudeline-$(echo "$DIR" | cksum | cut -d' ' -f1)"
  BRANCH="" STAGED=0 MODIFIED=0
  if [[ -f "$CF" ]] && (( NOW - $(stat -f %m "$CF") < 5 )); then
    IFS=$'\t' read -r BRANCH STAGED MODIFIED < "$CF"
  elif git -C "$DIR" -c gc.auto=0 rev-parse --git-dir >/dev/null 2>&1; then
    BRANCH=$(git -C "$DIR" -c gc.auto=0 branch --show-current 2>/dev/null)
    while IFS= read -r l; do
      [[ "${l:0:1}" != " " && "${l:0:1}" != "?" ]] && ((STAGED++))
      [[ "${l:1:1}" != " " && "${l:1:1}" != "?" ]] && ((MODIFIED++))
    done < <(git -C "$DIR" -c gc.auto=0 status --porcelain 2>/dev/null)
    printf '%s\t%s\t%s' "$BRANCH" "$STAGED" "$MODIFIED" > "$CF"
  fi
  if [[ -n "$BRANCH" ]]; then
    GIT="${SEP}${TEAL}󰘬 ${BRANCH}${RST}"
    (( STAGED > 0 ))   && GIT+=" ${GREEN}+${STAGED}${RST}"
    (( MODIFIED > 0 )) && GIT+=" ${YELLOW}~${MODIFIED}${RST}"
  fi
fi

# ═══ LINE 1: model │ effort │ dir │ git ═══════════════════════════
printf '%s\n' "${MAUVE}✦ ${MODEL}${RST}${SEP}${EFFORT_SEG}${SEP}${DIR_SEG}${GIT}"

# ─── Context bar ──────────────────────────────────────────────────
PCT=0
if [[ -n "$PCT_RAW" && "$PCT_RAW" != "null" ]]; then
  PCT=${PCT_RAW%%.*}
  (( PCT < 0 )) && PCT=0; (( PCT > 100 )) && PCT=100
fi
if   (( PCT >= 60 )); then BC="$RED"
elif (( PCT >= 40 )); then BC="$YELLOW"
else BC="$GREEN"; fi

F=$((PCT / 10)); E=$((10 - F)); BAR=""
for ((i=0; i<F; i++)); do BAR+="█"; done
for ((i=0; i<E; i++)); do BAR+="░"; done

# ─── 5h rate limit with reset countdown ───────────────────────────
RLIM=""
if [[ -n "$FIVE_PCT" && "$FIVE_PCT" != "null" ]]; then
  FI=$(printf '%.0f' "$FIVE_PCT")
  if   (( FI >= 80 )); then RC="$RED"
  elif (( FI >= 50 )); then RC="$YELLOW"
  else RC="$LAVENDER"; fi
  RLIM="${SEP}${RC}󰥔 5h: ${FI}%${RST}"
  if [[ -n "$FIVE_RESET" && "$FIVE_RESET" != "null" ]]; then
    REM=$((FIVE_RESET - NOW))
    (( REM > 0 )) && RLIM+=" ${SUBTEXT}$((REM / 3600))h$(((REM % 3600) / 60))m${RST}"
  fi
fi

# ─── 7d rate limit ────────────────────────────────────────────────
if [[ -n "$SEVEN_PCT" && "$SEVEN_PCT" != "null" ]]; then
  SI=$(printf '%.0f' "$SEVEN_PCT")
  if   (( SI >= 80 )); then SC="$RED"
  elif (( SI >= 50 )); then SC="$YELLOW"
  else SC="$SUBTEXT"; fi
  RLIM+="${SEP}${SC}7d: ${SI}%${RST}"
fi

# ─── Cost ─────────────────────────────────────────────────────────
COST_SEG=""
if [[ -n "$COST" && "$COST" != "null" && "$COST" != "0" ]]; then
  COST_FMT=$(printf '$%.2f' "$COST")
  COST_SEG="${SEP}${OVERLAY}${COST_FMT}${RST}"
fi

# ═══ LINE 2: context │ 5h │ 7d │ cost ════════════════════════════
printf '%s\n' "${BC}${BAR}${RST} ${PCT}%${RLIM}${COST_SEG}"

Things worth noting:

• Single jq call extracts all fields at once instead of spawning a separate process per field. The script runs after every assistant message, so speed matters.
• Git status is cached for 5 seconds in /tmp/, keyed by directory hash. Without this, git status on large repos adds noticeable lag.
• Directory name is a clickable OSC 8 hyperlink that opens the folder in your file manager.
• Effort level isn’t in the JSON payload, so the script reads it directly from settings.json.
• Context thresholds: green under 40%, yellow 40-60%, red above 60%. Same color logic for rate limits at 50% and 80%.
• Segments hide when data isn’t available. Cost disappears at $0, rate limits disappear before the first API response, git disappears outside a repo.

Testing with mock data

Pipe a JSON object to the script to test without a running Claude Code session:

bash -c '
  RESET=$(($(date +%s) + 7920))
  cat <<EOF | ~/.claude/statusline.sh
{
  "model": {"display_name": "Opus 4.6 (1M context)"},
  "workspace": {"current_dir": "/Users/me/projects/myapp"},
  "context_window": {"used_percentage": 35, "context_window_size": 1000000},
  "rate_limits": {
    "five_hour": {"used_percentage": 23.5, "resets_at": $RESET},
    "seven_day": {"used_percentage": 12.8}
  },
  "cost": {"total_cost_usd": 1.47}
}
EOF
'

This produces two lines:

✦ Opus 4.6 (1M context) │ ▲ high │ ◫ myapp
███░░░░░░░ 35% │ ⏱ 5h: 24% 2h12m │ 7d: 13% │ $1.47

The git segment (branch, staged/modified counts) appears when workspace.current_dir points to a git repository. The reset countdown (2h12m) is calculated live from resets_at.

With minimal data (new session, no rate limits yet):

✦ Opus 4.6 (1M context) │ ▲ high │ ◫ tmp
░░░░░░░░░░ 0%

"attribution": {
  "commit": "",
  "pr": ""
} 

Setting both to empty strings disables the attribution entirely.

Before:

feat: add retry logic for failed API calls

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

After:

feat: add retry logic for failed API calls

I asked Claude Code (v2.1.74) to review a research document I was working on. The prompt: “deeply check the research.md and address all the notes I left. Do additional research if needed and update the doc.”

It found 11 notes, launched 5 background research agents, and got to work. Then it tried to check on their progress. By the time I noticed something was wrong, a single .output file had consumed 324 GB on my Mac.

Finding the damage

After the session, Disk Utility showed 21 GB free out of ~995 GB. That didn’t look right. I opened a new Claude Code session and asked it to help track down whatever was eating my storage. Standard suspects (home directory, Docker volumes, caches) all looked clean. Then I pointed it at the suspicious background task I had killed and asked it to check /private/tmp/:

$ du -sh /private/tmp/claude-501/
324G

Claude Code stores background task output in .output files under /private/tmp/claude-<uid>/<project-path>/tasks/. The base path is configurable via CLAUDE_CODE_TMPDIR.

$ du -sh .../tasks/*
324G    .../tasks/bday3zjvd.output          ← background bash task
  0B    .../tasks/ab82d716c855dd5ba.output  ← agent (completed normally)
  0B    .../tasks/a9bffb61f67ddceae.output
  0B    .../tasks/a4b988afd22aa8db1.output
  0B    .../tasks/a417efc2a4f409639.output
  0B    .../tasks/a2ded7593505f4524.output

A single file got all 324 GB. The 5 agent files were 0 bytes.

Claude Code handles bash tasks and agent tasks differently. For bash tasks, stdout and stderr go to the b*.output file with no size limit.

For agents, each subagent writes a JSONL (newline-delimited JSON) session log under ~/.claude/projects/, and the a*.output file is a symlink to it. When an agent finishes, Claude Code queues a task-notification message for the main session with the agent’s results. The agent files showed as 0 bytes, possibly because the symlink targets had been cleaned up by the time I checked.

A tail -5 loop can’t produce 324 GB of text. So what was writing to that file?

Reconstructing the timeline

To find out, I compared timestamps from the main session log at ~/.claude/projects/<project-path>/<session-id>.jsonl against the subagent logs:

Reconstructed from session 462e8822-5f7d-428d-a444-33602f3b8a92

20:45:26  Claude Code launches 4 research agents
20:46:00  5th agent launched (GitHub milestones)
20:46:05  ls .../tasks/*.output → 6 files found (5 agents + bash task's own file)
20:46:10  Bash: for loop with tail -5 on each .output file
20:46:51  Agent a417efc completes (rig-core)        ← notification queued
20:47:17  Agent a9bffb6 completes (Coolify)         ← notification queued
20:47:23  Agent ab82d71 completes (GitHub images)   ← notification queued
20:47:25  Agent a4b988a completes (milestones)      ← notification queued
20:47:48  Agent a2ded75 completes (Telegram)        ← notification queued
          All 5 agents done. For loop also likely done.
          But the Bash process stays alive for 2 more minutes.
20:48:13  Bash command backgrounded → task bday3zjvd
          stdout: "", assistantAutoBackgrounded: false
          .output FILE GROWTH STARTS HERE (~340 MB/s)
20:48:16  Claude Code fetches TaskOutput tool via ToolSearch
20:48:19  TaskOutput(ab82d71) → "No task found"     ← 1 ms response
20:48:20  TaskOutput(a2ded75) → "No task found"     ← 1 ms response
20:48:21  TaskOutput(a9bffb6) → "No task found"     ← 1 ms response
20:48:22  TaskOutput(a417efc) → "No task found"     ← 1 ms response
20:48:22  TaskOutput(a4b988a) → "No task found"     ← 1 ms response
20:48:25  TaskOutput(bday3zjvd, block=true, timeout=10s)
20:48:35  TaskOutput returns: timeout, status: running, ~30K chars
20:49:17  "agents running into permission issues with web search...
          enough domain knowledge" → starts document rewrite
20:57:59  All 5 agent notifications dequeued simultaneously
          (~10 min delay: agents finished at 20:47, consumed at 20:58)
          Claude Code adds extra findings via targeted edits
21:04:23  I killed bday3zjvd → 324 GB over ~16 min of growth

The agents finish within 2 minutes. Each completion queues a task-notification, but those notifications don’t arrive for another 10 minutes, not until the model finishes the document rewrite.

The innocent tail command

After launching the agents, Claude Code runs a for loop with tail -5 on each .output file to check on their progress:

⏺ Bash(for f in /private/tmp/claude-501/.../tasks/*.output;
    do echo "=== $(basename $f) ==="; tail -5 "$f"; echo; done)

tail -5 reads the last 5 lines and exits. Without the -f flag, it doesn’t follow or wait for new data. A for loop over 6 files should finish in under a second. But the Bash process doesn’t return for 2 full minutes.

I don’t have a good explanation for this. It could be something in Claude Code’s bash execution wrapper, or stdout buffering between the shell process and the .output file. The timing roughly lines up with the agents finishing their work (the last one completes at 20:47:48), but I can’t draw a causal link.

Then Claude Code backgrounds the command as task bday3zjvd:

{
  "toolUseResult": {
    "stdout": "",
    "stderr": "",
    "backgroundTaskId": "bday3zjvd",
    "assistantAutoBackgrounded": false
  }
}

A command can be backgrounded in 3 ways:

1. Claude Code passing run_in_background: true in the Bash tool call
2. A user pressing Ctrl+B
3. Automatic timeout. Since v2.0.19 , commands exceeding BASH_DEFAULT_TIMEOUT_MS ( default 120s ) are auto-backgrounded instead of killed

Claude Code didn’t request it (run_in_background was false). I didn’t press Ctrl+B. That leaves auto-timeout, and the ~2-minute runtime matches the 120s default exactly. The assistantAutoBackgrounded flag shows false, which appears to be a logging bug: the command was auto-backgrounded at the timeout, but the flag wasn’t set correctly.

Once backgrounded, Claude Code redirects stdout/stderr into the .output file. These files have no size limit, and are never cleaned up after sessions end .

Could the for loop be reading its own output file? The glob *.output expands in the same tasks/ directory where bday3zjvd.output is created. I reproduced the command in a fresh session: Claude Code creates the .output file before starting the bash process, so the glob does include it. tail -5 reads back its own earlier echo lines, roughly doubling the output. But it reads at most 5 lines and exits. The entire loop completes in under 30 milliseconds. Self-reference can’t explain 324 GB.

Before deleting the file, I checked the first 2000 bytes with head -c 2000. The beginning was normal: agent JSONL entries prefixed by === filename === headers from the echo commands. Exactly what the for/tail loop would produce. But 2000 bytes out of 324 GB tells me nothing about what filled the rest.

Since stdout is redirected directly to the file, writes should stop when the bash process exits. Yet the file grew for 16 minutes. Either the bash process itself stayed alive far longer than the for loop, or something else was writing to that file.

Growing in the background

With the command now running in the background, Claude Code tried to check on its agents via TaskOutput, but all five calls returned “No task found” in 1 ms. It also checked the bash task, calling TaskOutput with block: true and a 10-second timeout, and got back ~30,000 characters of output containing agent JSONL with MCP tool permission denials. Claude Code concluded the agents were “running into permission issues with web search” and decided it had “enough domain knowledge” to proceed without them.

It moved on to rewriting the document. The background bash task kept running.

The agents had delivered their results to the notification queue 2 minutes earlier. Those notifications sat for 10 minutes while Claude Code generated a 52 KB rewrite of the document, then were all dequeued simultaneously. It scanned the findings and applied a few extra details via targeted edits.

Meanwhile, bday3zjvd.output kept growing.

16 minutes after backgrounding, I noticed the task was still alive. I selected it in the task list, saw infinite scrolling output, and killed it:

⏺ Background command "Check last lines of each agent output" was stopped

No task found

Those five “No task found” errors deserve a closer look. They left Claude Code unable to poll for agent status, so it read the bash task’s output instead and mistakenly concluded the agents had failed:

⏺ Task Output(non-blocking) ab82d716c855dd5ba
  ⎿  Error: No task found with ID: ab82d716c855dd5ba

⏺ Task Output(non-blocking) a2ded7593505f4524
  ⎿  Error: No task found with ID: a2ded7593505f4524

⏺ Task Output(non-blocking) a9bffb61f67ddceae
  ⎿  Error: No task found with ID: a9bffb61f67ddceae

⏺ Task Output(non-blocking) a417efc2a4f409639
  ⎿  Error: No task found with ID: a417efc2a4f409639

⏺ Task Output(non-blocking) a4b988afd22aa8db1
  ⎿  Error: No task found with ID: a4b988afd22aa8db1

The agents had finished 30 seconds before these calls. There’s a known bug where Claude Code removes completed tasks from the registry too early, but I think something simpler is going on here. Every call returned in 1 ms, as if the IDs weren’t recognized at all.

Compare that to the bash task bday3zjvd, which TaskOutput found instantly. The IDs even look different: agents get an a prefix with long hex, bash tasks get a b prefix with short alphanumeric. TaskOutput doesn’t seem to recognize agent IDs. Others have reported a similar issue .

Cleanup and prevention

After killing the task, I needed to reclaim the space. Check how much background tasks are using:

$ du -sh /private/tmp/claude-*/
324G    /private/tmp/claude-501/

Then remove the project subdirectory that’s consuming space:

rm -rf /private/tmp/claude-501/<project-path>

Free space went from 21 GB to 344 GB.

On macOS, /private/tmp is not reliably cleaned on reboot. These files can persist indefinitely.

You may set CLAUDE_CODE_DISABLE_BACKGROUND_TASKS=1 to disable background tasks entirely. This prevents .output files from growing unchecked, but also means Claude Code can’t run long commands while continuing other work.

Reflections

This bug is quiet. No warnings, no errors, no disk space alerts. I caught it only because I happened to notice the background task indicator still blinking. If I hadn’t, it would have filled the disk.

Other users have reported similar .output file growth: 537 GB , 160 GB , 740 GB . The triggers vary (some involve failing commands dumping error output), but the underlying problem is the same: unbounded .output files with no cleanup. As of March 2026, no built-in fix exists. If your Mac is unexpectedly low on space, check /private/tmp/claude-*.

I checked the first 2000 bytes of the file before deleting it, but that only showed the normal for/tail output. I should have checked the middle or end to see what was filling the other 324 GB.

Claude Code ships daily. That pace is impressive, but 3,617 open bugs is the other side of it, and the quiet ones are the most expensive.

The best of the year

The Rust Programming Language by Steve Klabnik, Carol Nichols

This was my third or fourth attempt to get through it, and this time I made it. If you’re switching to Rust from another language, this is the book to start with. It has clear step-by-step explanations, tons of examples, and a few projects for hands-on practice.

Many times, when I had a question after reading a paragraph, the very next one answered it. Not many technical books have this quality. 5/5

How Not to Die by Michael Greger, Gene Stone

This book is packed with actionable nutrition advice, and it pushed me to change what I eat. I’m already seeing positive changes. 5/5

I May Be Wrong by Bjørn Natthiko Lindeblad

Deep reflections on a monk’s life. It shows how easily we get lost in plans, ideas, and thoughts about the past and future, while forgetting to live in the only moment we truly have: now.

Three ideas that stayed with me:

• “You won’t always have what you want, but you’ll always have what you need.”
• “This too shall pass.”
• “I may be wrong.” 5/5

The Wealth Ladder by Nick Maggiulli

The only financial book I read this year, and I enjoyed it a lot. It suggests a clear framework for thinking about wealth as a series of levels. The book shows which strategies work at certain levels but not at others, and explains the “why” behind them.

Many sources give you strategies without telling you that at certain levels they just don’t work, which leads to people getting stuck. This book helped me see wealth from a different angle and gave me clarity on what to stop doing and what to start doing. 5/5

The Insulted and Humiliated by Fyodor Dostoevsky

A treasure. It took me over a year to finish, but I didn’t push. I read it occasionally in the evenings.

The story is emotionally deep. It shows how people interact, what they feel, and what they do to each other. Dostoevsky highlights core society problems that remain relevant today. The book was published in 1861, but the human behaviors it captures feel like the author wrote it in 2025. 5/5

Doctor Zhivago by Boris Pasternak

Another 5-star novel, and one that feels larger than its characters. It’s about a whole generation, about a whole country that will never be the same again, about people who couldn’t accept and adapt to the changes brought by the 1917 Revolution in Russia. A story about love, identity, and survival. 5/5

Writing for Developers by Piotr Sarna, Cynthia Dunlop

A refreshing perspective on blogging. The first part walks through why and how to write a solid post, although I sometimes found it overwhelming, with too many details. The second part is the highlight: real posts, broken down and analysed. It was truly useful, and fun to read. 4/5

Docker Deep Dive by Nigel Poulton

I read the May 2025 edition. Despite the title, I’d call it a solid overview rather than a true deep dive. It was a great refresher that updated my rusty Docker knowledge and filled a few gaps in my understanding of the core concepts. The newer chapters, like running LLMs in containers and WebAssembly apps, were a nice bonus. 4/5

Flatland by Edwin A. Abbott

This book is often framed as a maths story. To me, it’s really about society, hierarchy, and what people do when their worldview gets challenged. It’s short, strange, and still relevant. 4/5

Doom Guy by John Romero

I read Masters of Doom by David Kushner last year and enjoyed it a lot. Doom Guy is a good companion, and it gives a different, mainly subjective, perspective on that period. Still, it’s full of hard lessons, and Romero’s reflections are the best part.

I think I’m finally done with this chapter of gaming history. If you enjoyed Masters of Doom, this is worth picking up. 4/5

AI Engineering by Chip Huyen

This was a tough one and took me a few months to finish. It gives a broad overview of the AI landscape. Some chapters go wide rather than deep, while others, like entropy and fine-tuning, go much deeper. Because of that, the target audience isn’t always clear. Some sections are simple enough for non-technical managers, while others may challenge even experienced engineers.

I don’t think this is a good first AI book. The more experience you have, the more you’ll get out of it. It reminded me of Designing Data-Intensive Applications by Kleppmann. When I first read it early in my career, much of it didn’t make sense, but years later it became far more valuable. I may revisit AI Engineering in a few years to close the knowledge gaps. 4/5

The Logic Manual by Volker Halbach

I started the year with this one, which was a slightly brutal way to begin. It’s hard to follow, and I wish it had more examples and practical applications. A few concepts, like the split between propositional and predicate logic, only clicked later when I started reading How to Prove It by Velleman. I’ll likely come back to it next year. 3/5

The worst of the year

How to Write a Lot by Paul J. Silvia

The most disappointing book I read this year. The core message is simple: schedule writing time and protect it. That’s good advice, but it’s delivered early, and the rest felt like filler, mostly aimed at academic publishing. 1/5

Next year

I have a few long reads in progress, some of which I mentioned in my post on How LLMs Sent Me Back to Logic and Linear Algebra . Next year I plan to focus on fundamentals: system architecture, more maths, and AI foundations rather than trends.

AI might replace some coding tasks. It won’t replace clear system thinking.

I went from “what the fck is this” to “ok, there is exactly one value of \(x\) such that \(P(x)\) is true”.

This is the first post in my learning journey series: From Logic to Transformers. I want to track my progress, so I can look back on how far I’ve come, and maybe help others on a similar path.

Why

I’ve been a software engineer for over a decade, mainly focused on backend systems. With all the recent AI developments, I became curious about Large Language Models (LLMs) and the hype around them. Naturally, I tried LLM-based tools in my pet projects and at work. My productivity went up. It felt like having a clever junior dev by my side, someone who can handle the heavy lifting but still needs guidance and direction.

The more I used these tools, the more I wanted to understand how and why they work under the hood. Understanding the why behind the tools has always led to deeper insights and helped me solve harder problems and design better systems.

But LLMs turned out to be a different beast. My attempts to understand them went nowhere. Books and blog posts only raised more questions. I even tried reading research papers, but the math did not click. Words like vector spaces and eigenvectors meant nothing to me. Matrix multiplications just worked in the code, but I had no idea why. I could follow the implementations line by line, but I could not see the reasoning behind them.

That frustration is what pushed me back to the basics.

Back to basics

When I turned to math for Machine Learning (ML), most books started with vectors and matrices, then moved on to vector spaces and linear algebra. I had studied linear algebra back in university, but no one explained why things worked. Or maybe I just was not paying attention because I did not see how it connected to real-world problems.

I sometimes envy folks who went into game development. They realized early on that linear algebra was essential for graphics and physics, and they built solid foundations. I skipped that step, which is why the math felt like a wall when I hit it later.

Linear algebra starts simple, with vectors and their geometric meaning. But it gets steep quickly. The deeper you go, the more theorems and proofs you meet, each one building on top of the previous one. That is where I got stuck again: why do these proofs work, and how did the author come up with them?

That search for answers is what led me to the missing piece: logic.

Finding the real gap

At that point I realized something important: all proofs are built on logic. Every “therefore” in a textbook assumed a foundation I didn’t have. I had never studied logic formally, beyond a few AND/OR/NOT gates in a hardware course at university. No wonder proofs didn’t make sense.

So I decided to start from scratch. Learn logic, and re-learn linear algebra in parallel. One day logic, the next day linear algebra. Repeat until things start to click.

The study routine

I ended up with two books: How to Prove It (3rd edition) by Daniel J. Velleman and Introduction to Linear Algebra (2nd edition) by Serge Lang. I tried watching videos and courses, but they distracted me and I lost focus, so I narrowed down to just these two resources.

The process is simple. I read a chapter and work through the exercises that have answers to check myself. I also attempt others, using ChatGPT to verify or clarify when needed.

While reading, I take quick notes, highlighting anything that stands out or concepts I don’t know. After finishing a chapter, I read it again, review my notes, and turn them into Anki cards . On the second read I almost always catch things I missed, and my understanding gets deeper.

I study in the mornings, usually for 90 to 120 minutes, when my mind is fresh and math is a good fit for that time. The first 30 minutes go to Anki reviews from both decks, no matter if it is a logic or a math day. If the review is quick, I use the extra time to turn more notes into cards.

The next 60 to 90 minutes are for reading: logic one day, math the next. If a subsection ends with exercises, I solve them right away. At first, I could not solve many, and that was fine. Over time, as I built and reviewed more cards, problems that once seemed impossible started to feel manageable, and my intuition grew.

Progress after 50 days

So, where am I after 50 days?

In logic, I have finished the Sentential and Quantificational logic chapters and I am now in the middle of Chapter 3 on Proofs. In linear algebra, I went through the chapters on Vectors, Matrices and Linear Equations, and Vector Spaces, and I am now in the middle of Chapter 4 on Linear Mappings.

Logic

The most memorable moment in logic was when I managed to derive \(A \backslash B = A\) from \(A \cap B = \emptyset\) and show they are equivalent.

Here is the full chain of reasoning (skip if you are not into the details):

• \(A \cap B = \emptyset\) is equivalent to \(\neg\exists x(x \in A \wedge x \in B)\) using the set definition
• which is equivalent to \(\forall x\neg(x \in A \wedge x \in B)\) using the quantifier negation law
• which is equivalent to \(\forall x(x \notin A ∨ x \notin B)\) using De Morgan’s law
• which is equivalent to \(\forall x(x \in A \rightarrow x \notin B)\) using the conditional law
• which is equivalent to \(\forall x((x \notin B \wedge x \in A) ↔︎ x \in A)\) using the biconditional law
• which is equivalent to \(\forall x(x \in A \backslash B ↔︎ x \in A)\) using the set definition
• which is equivalent to \(A \backslash B = A\).

Linear algebra

The highlight of linear algebra was this exercise:

Let \(V\), \(W\) be two vector spaces and let \(F: V \rightarrow W\) be a linear map. Let \(U\) be the subset of \(V\) consisting of all elements \(v\) such that \(F(v) = O\). Prove that \(U\) is a subspace of \(V\).

Here is my scratch work. We need to prove 3 things:

1. \(O \in U\)
2. if \(v, w \in U\), then \(v + w \in U\)
3. if \(v \in U\) and \(c\) is a scalar, then \(c * v \in U\)

Proof:

• Since \(F\) is linear, \(F(O) = O\). So \(O \in U\).
• Let \(v\) and \(w\) be arbitrary elements of \(U\). Using properties of linear map we have \(F(v + w) = F(v) + F(w) = O + O = O\). So \(v + w \in U\).
• Let \(v\) be an arbitrary element of \(U\) and \(c\) is a scalar. Using properties of linear map we have \(F(c * v) = c * F(v) = c * O = O\). So \(c * v \in U\).
• Therefore, \(U\) is a subspace of \(V\).

To my surprise, I knew exactly what to do and how to prove it. That was the best feeling in the world, and it motivates me to keep going.

Reflections so far

At first, it felt like a rabbit hole. I did not understand my gaps or where to begin building the foundation. I jumped from LLM blog posts and papers to ML, then to linear algebra, and finally to logic. Now, 50 days in, I feel like I have finally found the right starting point.

The path has not been easy. Some days I spend an hour wrestling with a single logic problem. Other days I manage 6 pages of linear algebra in 2 hours. Progress is uneven, but consistency wins.

Right now, my pace is about 5–6 pages in 90 minutes. That may sound slow, but the goal is not speed. It is building understanding that lasts. Focusing on the process instead of the result makes the work sustainable and even enjoyable.

This journey from Logic to Transformers will take time, but the connections are already becoming clearer. Logic helps me understand the structure of arguments and follow proofs step by step. With that foundation, concepts in linear algebra that once felt like rules to memorize, such as why a set of vectors forms a subspace, now start to make sense. Little by little, the pieces are coming together, and I can see how each layer of understanding builds on top of the previous one.

What’s next?

Over the next 50 days, the plan is to keep going with the books I am reading. I will probably finish linear algebra by then, but if not, that is fine. What matters is showing up every day and moving forward.

I will share updates along the way, including progress, lessons, and next steps. Maybe it will help someone else who feels lost in the why, like I was. I have found my starting point, and I hope you find yours too.

Sometimes the fastest way forward is realizing where you should have started all along. See you in the next post.

This is the story of my journey to debug a networking issue with my Synology NAS and bring it back to life. I’ll walk you through my troubleshooting process, starting with quick checks and gradually digging deeper to find the root cause. Even if you don’t own a NAS, I encourage you to read on. The approach I took and the tools I discovered along the way might give you a fresh perspective and help you solve entirely different technical challenges.

Basic troubleshooting: no quick fix in sight

I restart the NAS. No connection. My router shows the NAS in the list of disconnected clients but provides no additional details in the logs. I restart the router, try a new Ethernet cable, and use different LAN ports on both the NAS and the router. Same issue. The NAS power indicator is solid green. The LAN LEDs are green, indicating no issues with the physical connection, as shown below.

I didn’t set a static IP address when initially configuring my NAS. Typically, reconnecting to a different LAN port should trigger a DHCP request because the network interface initializes a new connection, potentially requiring a new IP address. But that doesn’t happen.

In rare cases, a NAS might fall back to a static IP configuration if the DHCP client fails to initialize. I’m wondering if the NAS switched to using a static IP. In this case, the router would have detected the expired lease and returned the IP to the pool of available addresses. Another client might have taken it, potentially causing an IP conflict.

I check all connected clients but don’t see the NAS IP assigned to any device. This makes me think that the NAS might be using a completely different IP, possibly outside the expected range on my local network. But I still want to be sure it’s not a router issue. Let’s connect the NAS directly to my MacBook.

Narrowing it down: direct connection and network scanning

I disable Wi-Fi and use a direct Ethernet connection between the NAS and my MacBook to isolate the issue. I want to be sure that no other network components are interfering with the connection. On the MacBook, I open the network settings and configure the Ethernet-to-Type-C adapter to use manual IPv4 settings, as shown below. This is the same range where the NAS IP was last seen.

Synology provides a tool called Web Assistant that helps discover NAS devices on the local network. I run the Web Assistant, but it detects nothing. Next, I decide to scan all available IPs in this isolated network, which consists of just the NAS and the MacBook.

I use a network scanning tool called nmap, which checks each IP in a specified range to see if it is reachable. The -sn flag skips port scanning, which speeds up the process since I only need to check for active hosts. I run the following command:

$ nmap -sn 192.168.4.0/24
Nmap scan report for 192.168.4.1 # MacBook IP address
Host is up (0.0013s latency).
Nmap done: 256 IP addresses (1 host up) scanned in 0.01 seconds

The output shows only the IP of my MacBook. For comparison, here’s the expected output when the NAS is visible on the network:

$ nmap -sn 192.168.4.0/24
Nmap scan report for 192.168.4.1 # MacBook IP address
Host is up (0.00056s latency).
Nmap scan report for 192.168.4.231 # NAS IP address
Host is up (0.0010s latency).
Nmap done: 256 IP addresses (2 hosts up) scanned in 21.58 seconds

This confirms that the router is not causing the issue. Something is wrong with the NAS itself. It’s time to reset it.

Resetting the NAS: soft vs. hard modes

Synology gives you two ways to reset a NAS :

• Soft reset restores the admin account to default and resets the network interfaces to use DHCP.
• Hard reset performs everything a soft reset does but also removes the Synology DiskStation Manager (DSM), the Linux-based operating system that the NAS runs on.

I press and hold the RESET button for about 4 seconds until I hear a beep. I release the button and launch the Web Assistant. Nothing happens. I restart the NAS, but the issue persists. The NAS remains undiscoverable. At this point, I suspect a problem with the DHCP client. If the DHCP client isn’t initializing correctly, a soft reset won’t resolve the issue.

I proceed with a hard reset. This action completely wipes out all system configurations but keeps my data on the hard drives intact. I restart the NAS. The hard drives spin up. The power LED starts blinking, indicating that the DSM is not installed. I launch the Synology Web Assistant again, but it fails to find the NAS. I repeat the same steps using a direct connection to my MacBook. Same result.

So here I am, sitting in front of this black box. It’s up and running, but it seems to live a life of its own. I recall that I have a travel router that might provide more detailed logs for connected clients.

Seeking more clarity: using a different router

I unpack my travel router, connect it to the WLAN, and then connect both the NAS and the MacBook to the router. I check the logs and see some activity from the NAS:

daemon.notice netifd: Network device 'eth2' link is up
kern.info kernel: [  219.955222] nss-dp 3a001400.dp3 eth2: PHY Link up speed: 1000
kern.info kernel: [  219.955300] br-lan: port 2(eth2) entered forwarding state

And here are the log entries from the MacBook:

daemon.notice netifd: Network device 'eth1' link is up
kern.info kernel: [   85.954650] nss-dp 3a001200.dp2 eth1: PHY Link up speed: 1000
kern.info kernel: [   85.954811] br-lan: port 1(eth1) entered forwarding state
daemon.info dnsmasq-dhcp[4707]: DHCPDISCOVER(br-lan) c8:a3:62:07:1b:37
daemon.info dnsmasq-dhcp[4707]: DHCPOFFER(br-lan) 192.168.8.202 c8:a3:62:07:1b:37
daemon.info dnsmasq-dhcp[4707]: DHCPREQUEST(br-lan) 192.168.8.202 c8:a3:62:07:1b:37
daemon.info dnsmasq-dhcp[4707]: DHCPACK(br-lan) 192.168.8.202 c8:a3:62:07:1b:37 MBP

The MacBook establishes a physical connection over the Ethernet cable and requests an IP address from the router. The router responds with a DHCP offer. The full DHCP handshake completes successfully. As a result, my MacBook successfully gets a new IP address. But there’s nothing similar for the NAS. The physical connection works fine (1 Gbps link from the logs), but the NAS isn’t making any DHCP requests.

I reset the NAS one more time, switch to a different LAN port on the NAS, and review the router logs:

daemon.notice netifd: Network device 'eth2' link is down
kern.info kernel: [  824.964258] nss-dp 3a001400.dp3 eth2: PHY Link is down
kern.info kernel: [  824.964680] br-lan: port 2(eth2) entered disabled state
daemon.notice netifd: Network device 'eth2' link is up
kern.info kernel: [  826.964295] nss-dp 3a001400.dp3 eth2: PHY Link up speed: 1000
kern.info kernel: [  826.964382] br-lan: port 2(eth2) entered forwarding state

The pattern remains consistent:

1. Link goes down (unplugging)
2. Link comes up (plugging in)
3. Port enters forwarding state
4. But no DHCP or higher-level network activity

NAS might be trying to use a static IP outside the local network range due to a DHCP client failure. Or its network stack isn’t initializing properly. But how could this happen after a hard reset?

A game-changer: discovering a hidden debug tool

I remove all the hard drives and the SSD cache from my NAS and power it up. Same issue. While removing the SSD cache modules, which are located at the bottom of the NAS, I notice a small cover that reveals part of the motherboard. I remove the cover and see a 6-pin connector. After researching online, I discover that this is a serial console port, also known as a UART debug port , as shown below.

The 6 pins are:

• TX (Transmit): Sends data from NAS to computer
• RX (Receive): Takes data from computer to NAS
• GND (Ground): Common reference point for electrical signals
• VCC (Power): Power supply (3.3V or 5V), not used as NAS has its own power source
• CTS (Clear to Send): Flow control pin, usually not needed
• RTS (Request to Send): Flow control pin, usually not needed

This port is commonly used in embedded systems and electronic devices for debugging, monitoring, and communication. In theory, I can use a USB-to-TTL adapter to connect my MacBook to the NAS via this port and access system logs.

The MacBook has the screen command, which can open a connection to the serial console and capture the output from the connected device. The command I run is screen -L /dev/tty.usbserial-* 115200, where:

• -L: Creates screenlog.0 in the current directory and logs all output to the file
• /dev/tty.usbserial-*: Path to the USB-serial device (* matches any serial device name). macOS creates this device when I plug in the USB-TTL adapter
• 115200: Baud rate, the speed of data transmission over the serial connection, which is a standard value for serial devices

Strangely, I find no reference to the serial console port in the Synology documentation. And I have no idea which pin is responsible for which function. I need to figure out the pinout.

Out of the 6 pins, I only need to locate 3: GND, RX, and TX. I take a multimeter and set it to continuity mode. I connect the black probe to the metal part on the NAS motherboard and touch the red probe to each pin one by one. Finally, I hear a beep. The ground pin is found.

I connect the GND pin to the USB-TTL adapter using a black jumper wire. Next, I need to find the TX pin. I connect the white jumper wire to the adapter’s RX pin and test each pin on the NAS UART port until I see messages appear on the MacBook. This is the best feeling I’ve had in over a decade in software engineering. I can finally see what’s happening inside the NAS:

Synology ready to boot Event() Start
Synology ready to boot Event() end
Initialize secure boot and secure flash related variables
Verify the signature of the image

GNU GRUB  version 2.02~beta3
   Press "CTRL-C" for boot menu or it boots automatically in 3s.
   Press "CTRL-C" for boot menu or it boots automatically in 2s.
   Press "CTRL-C" for boot menu or it boots automatically in 1s.
   Press "CTRL-C" for boot menu or it boots automatically in 0s.

Booting 'SYNOLOGY_2'
  Checking file [/zImage]...Passed.
  Checking file [/rd.gz]...Passed.
...

Since the RX pin is typically located near the TX pin, I test the adjacent pins. After a few attempts, I find the RX pin and connect it to the adapter’s TX pin using a red jumper wire, as shown below.

I now have low-level access to the NAS’s system logs and can control the device.

Digging deeper: why are DHCP requests not sent?

Synology’s DSM is based on Linux. At the end of the boot process, I see a login prompt that says SynologyNAS login:. I try different combinations of root and admin passwords but can’t get in. I restart the NAS and access the GRUB bootloader, which shows the following settings:

I add init=/bin/bash to the end of the line that starts with linux. This parameter tells the kernel to execute /bin/bash as the initial process instead of the default init system. This should allow me to bypass the usual user authentication. But it doesn’t work. I still see the Synology login prompt. I try a few other values for init, but without success.

It seems that something is either ignoring or overriding this parameter. I notice initrd /rd.gz in the GRUB bootloader, which could be the cause. But when I remove it, the system fails to boot. It turns out that /rd.gz is a compressed initial RAM disk image containing a minimal root filesystem and essential drivers needed to boot the system. GRUB loads this image into memory for the kernel’s use during the early stages of the boot process. So, this leaves me with limited options.

How come I haven’t checked the logs yet?

I go through the boot logs and find something strange. At the end of the boot process, only the loopback network interface is up. All the Ethernet interfaces are down. This explains why the NAS doesn’t send DHCP requests when connected to the router and is not visible on the local network.

I return to the GRUB and enable more detailed logging via debug loglevel=7 ignore_loglevel parameters. This provides more information but nothing directly relevant to the issue. I see that the network module is loaded, but no Ethernet interfaces are initialized:

:: Loading module r8168 ... [  OK  ]
... [  OK  ]
lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

I wonder if there’s a way to force the system to bring up the network interfaces. The ip=::::::dhcp parameter tells the kernel to use DHCP for all network interfaces. I update the GRUB configuration, press Ctrl-x to continue booting, and guess what, the Ethernet interfaces come up. The NAS successfully gets a new IP address from the router:

...
:: Loading module r8168[   12.555232] r8168 Gigabit Ethernet driver 8.045.12-NAPI loaded
...
udhcpc: started, v1.30.1
eth0      Link encap:Ethernet  HWaddr 90:09:D0:56:D7:2E  
          inet addr:169.254.183.221  Bcast:169.254.255.255  Mask:255.255.0.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:642 (642.0 B)
          Interrupt:77 Base address:0x8000 

eth1      Link encap:Ethernet  HWaddr 90:09:D0:56:D7:11  
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
          Interrupt:76 Base address:0xe000 

eth2      Link encap:Ethernet  HWaddr 90:09:D0:56:D7:32  
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
          Interrupt:75 Base address:0xa000 

eth3      Link encap:Ethernet  HWaddr 90:09:D0:56:D7:44  
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
          Interrupt:78 Base address:0xa000 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

I open the Synology Web Assistant, and it finds the NAS. It prompts me to reinstall the DSM, saying that I need to insert the hard drives.

Uncovering the root cause: PCIe initialization behavior

I put the hard drives back, start the NAS, and have the same issue. The NAS is no longer available on the local network, even when I use the ip=::::::dhcp kernel parameter. I remove the hard drives and use the same configuration that led to the previously successful run. But the issue persists.

Luckily, I saved the log from the successful run. I put the hard drives back and get a new log from the failed run. I compare both logs and notice a minor difference in the network module loading:

# Successful run
:: Loading module r8168[ 12.559868] r8168 Gigabit Ethernet driver 8.045.12-NAPI loaded

# Failed run
:: Loading module r8168 ... [  OK  ]

The failed run uses the default r8168 network driver, while the successful run loads the r8168 NAPI Ethernet driver, an improved version of the default one. I have no idea why it happens. I try to force the use of the NAPI driver by setting the r8168.use_napi=1 kernel parameter, but it has no effect.

I continue comparing the logs and notice a difference in PCI bridge initialization early in the boot process. During the successful run, the network interfaces are detected through PCIe enumeration as follows:

[    1.089159] pci 0000:00:01.2: PCI bridge to [bus 01]
[    1.096159] pci 0000:00:01.3: PCI bridge to [bus 02]
[    1.103160] pci 0000:00:01.4: PCI bridge to [bus 03-08]
[    1.108880] pci 0000:03:00.0: PCI bridge to [bus 04-08]
[    1.116168] pci 0000:04:00.0: PCI bridge to [bus 05]
[    1.123169] pci 0000:04:02.0: PCI bridge to [bus 06]
[    1.130170] pci 0000:04:06.0: PCI bridge to [bus 07]
[    1.137171] pci 0000:04:0e.0: PCI bridge to [bus 08]
[    1.144167] pci 0000:00:01.5: PCI bridge to [bus 09]
[    1.155454] pci 0000:00:08.1: PCI bridge to [bus 0a]
[    1.160473] pci 0000:00:08.2: PCI bridge to [bus 0b]

There is a PCIe switch at 0000:03:00.0, which serves as a root for the 4 LAN ports connected beneath it:

0000:00:01.4 (Root Bridge)
    └── 0000:03:00.0 (PCIe Switch)
        ├── 0000:04:00.0 (LAN Port)
        ├── 0000:04:02.0 (LAN Port)
        ├── 0000:04:06.0 (LAN Port)
        └── 0000:04:0e.0 (LAN Port)

However, in the failed run, the PCIe switch is missing:

[    1.086587] pci 0000:00:01.2: PCI bridge to [bus 01]
[    1.093587] pci 0000:00:01.3: PCI bridge to [bus 02]
[    1.098421] pci 0000:00:01.4: PCI bridge to [bus 03]
[    1.105589] pci 0000:00:01.5: PCI bridge to [bus 04]
[    1.116882] pci 0000:00:08.1: PCI bridge to [bus 05]
[    1.121898] pci 0000:00:08.2: PCI bridge to [bus 06]

When the switch fails to enumerate properly, all the network ports behind it become inaccessible. This explains why the default network module is loaded and no Ethernet interfaces are brought up.

There could be few reasons for PCIe switch initialization issues, such as Synology firmware problems, incorrect BIOS settings, or even power supply instability. I check the DSM release notes and see that an update was released about a week before the issue began. This leads me to think about the system as the following layers:

• Layer 1: DSM (Operating System & Settings)
• Layer 2: Firmware (Controls hardware initialization)
• Layer 3: Hardware (Physical components)

When I reset the NAS, it only affected Layer 1 (DSM), while the recent update could have modified Layer 2 (Firmware). Since firmware settings are stored in non-volatile memory, they persist through normal system resets until explicitly changed by another firmware update. This explains why the PCIe switch initialization issues continued even after resetting the system.

I restart the NAS few more times. During the last restart, the network PCIe bridges initialize correctly. This could be due to a timing issue or the proper initialization sequence. But I take advantage of this random event to run the Synology Web Assistant, discover the NAS, and install the DSM.

Reinstalling DSM and fixing network connectivity

After reinstalling DSM, the NAS works fine. I check the logs during the installation and see this:

------------upgrade
Begin upgrade procedure
Found an upgrade file on data volume. Begin upgrade
...
Found /tmpData/SynoUpgrade.tar.gz. Hide it.
'/tmpData/SynoUpgrade.tar.gz' -> '/tmpData/.SynoUpgrade.tar.gz'
Found /tmpData/SynoUpgradeIndexdb.txz. Hide it.
'/tmpData/SynoUpgradeIndexdb.txz' -> '/tmpData/.SynoUpgradeIndexdb.txz'
Found /tmpData/SynoUpgradeSynohdpackImg.txz. Hide it.
'/tmpData/SynoUpgradeSynohdpackImg.txz' -> '/tmpData/.SynoUpgradeSynohdpackImg.txz'
Do [/bin/tar xf /tmpData/.SynoUpgrade.tar.gz -C /tmpRoot]... try=1/2
Success to untar main tarball
Deploying default configurations
Untaring .SynoUpgradeIndexdb.txz
Untaring .SynoUpgradeSynohdpackImg.txz
DataMnt and RootMnt are same devices. No need to move packages
Touching /tmpRoot/var/.UpgradeBootup
...
[   78.624775] synobios open /dev/ttyS1 success
[   78.638434] correction with 0x00
[   78.648209] synobios: load, major number 201
[   78.652478] Brand: Synology
[   78.655273] Model: DS-1522+
[   78.658066] This is default settings: set group disks wakeup number to 1, spinup time deno 1
[   78.666517] synobios cpu_arch proc entry initialized
[   78.671478] synobios crypto_hw proc entry initialized
[   78.676524] synobios syno_platform proc entry initialized
mknod: /dev/synobios: File exists
Starting /usr/syno/bin/synohdcfgen...
/usr/syno/bin/synohdcfgen returns 0
[   78.685588] Module [r1000_synobios] is removed. 
[   78.696717] synobios: unload
Release upgrade preserved space temporarily in upgrade bootup.
End upgrade procedure
============upgrade

I restart the NAS few times. No issues. The update process extracts and applies firmware-related components, potentially overwriting any corrupted firmware settings with the correct ones. This could explain the resolution of the PCIe initialization problem.

I configure the NAS to use a static IP from a reserved range outside the DHCP pool. I do this to ensure consistent, reliable access to my storage and services regardless of network changes or reboots. Everything works as expected.

Reflections and takeaways

Low-Level Access is a Game-Changer

Accessing the serial console port showed me exactly what was going on during boot, turning the NAS from a black box into an open book. It let me see bootloader logs and kernel initialization, which helped me figure out why the network interfaces weren’t coming up. I learned that low-level access is essential when standard troubleshooting methods fall short.

Systematic Troubleshooting is Key

I started with the basics. Checked cables, ports, and DHCP settings before diving deeper. Isolating variables one by one, such as using direct connections and minimal network setups, helped me narrow down the cause. Documenting each step allowed me to save the log from a successful run, which played a key role in finding the core issue. This showed me the power of a structured and systematic approach.

Persistence and Curiosity Lead to Breakthroughs

I tested countless ideas, compared logs line by line, and kept digging even when nothing made sense. When I first found the UART port, I didn’t even know how use it. But I kept learning and experimenting until I uncovered the root cause. This reminded me that breakthroughs often result from persistence, curiosity, and a willingness to push beyond your current skill level.

Final thoughts and looking ahead

I still don’t know exactly what happened that morning when the NAS disappeared from my local network. I lost all system logs when I did the hard reset and only discovered the serial console port afterward. Based on the timing, I can only speculate that the latest Synology update corrupted the system configuration.

But I learned a great deal during this investigation, and I added a powerful tool under my belt—the UART debug port. If I’m lucky (or unlucky) and the issue occurs again, I won’t need to reset anything. I’ll just unwrap the USB-TTL adapter, connect my MacBook to the NAS, log in with my admin password, and check the system logs to get the answers I couldn’t get this time.

Happy debugging!

Implementing the core algorithm alone didn’t capture my interest. I wanted to see the algorithm’s results in action. Although I’ve spent over a decade building backend systems, there’s something satisfying about visual feedback. This led me to image compression.

While k-means clustering can be used to reduce image sizes by averaging similar colors, I’m aware there are more efficient algorithms for this purpose. However, for educational reasons, it’s an excellent way to see k-means in action.

Above is a comparison of an original image (left) and its compressed version (right), which uses only 16 colors and the Greedy K-means++ initialization method for k-means.

In the sections that follow, I’ll dive deeper into other initialization methods, their implementation, and their comparative effectiveness. But first, let’s explore the fundamental components of the k-means clustering.

K-Means clustering algorithm

K-means clustering consists of these 5 steps:

1. Initialization: Centroids are initialized using a chosen strategy. Common strategies include Forgy, MacQueen, Maximin, Bradley-Fayyad, and K-means++.
2. Main Loop: The algorithm iterates until it converges or reaches the maximum number of iterations.
3. Cluster Assignment: Each data point is assigned to its nearest centroid.
4. Centroid Update: New centroids are calculated as the mean of the points assigned to each cluster.
5. Convergence Check: The algorithm checks if the change in SSE (Sum of Squared Errors) is below a specified threshold.

function k_means_clustering(data, k, max_iterations, epsilon):
    // Initialize centroids using chosen strategy (e.g., k-means++)
    centroids = initialize_centroids(data, k)
    
    for iteration = 1 to max_iterations:
        // Assign each point to the nearest centroid
        clusters = assign_clusters(data, centroids)
        
        // Recompute centroids as the mean of assigned points
        centroids = compute_centroids(data, clusters, k)
        
        // Calculate the sum of squared errors (SSE)
        old_sse = sse
        sse = calculate_sse(data, clusters, centroids)
        
        // Check for convergence
        if (old_sse - sse) / sse < epsilon:
            break
    
    return centroids, clusters, sse, iteration

The pseudocode above captures the iterative process of refining cluster assignments and updating centroids to minimize the SSE, thus optimizing the grouping of data points.

Choosing initial centroids

Selecting initial centroids is a crucial step in the k-means algorithm because a good choice can lead to faster convergence and more effective clustering.

There has been significant research in this area, resulting in various methods for initialization . Here are some of the methods I explored for image compression, each characterized by its non-deterministic nature, meaning results may vary with each run due to their probabilistic approaches:

• Forgy: Randomly assigns each data point to one of the k clusters, using these assignments to compute initial centroids.
• MacQueen: Selects k unique data points randomly as initial centroids.
• Maximin: Iteratively selects the points that are farthest from existing centroids to ensure diversity.
• Bradley-Fayyad: Runs k-means on subsets of the data to identify effective initial centroids.
• K-means++: Chooses initial centroids with a probability proportional to their squared distance from the nearest existing centroid, enhancing cluster quality.
• Greedy K-means++: An improved version of K-means++ that evaluates multiple candidates at each step to optimize the selection.

Let’s wrap up the main k-means loop before we dive into these initialization methods in detail.

Assigning points to clusters

During each iteration of the k-means loop, the algorithm calculates the squared Euclidean distance between each point and every centroid. Each point is then assigned to the centroid it is closest to.

fn squared_euclidean_dist(a: &[f64], b: &[f64]) -> f64 {
    a.iter().zip(b.iter()).map(|(x, y)| (x - y).powi(2)).sum()
}

fn compute_clusters(data: &[Vec<f64>], centroids: &[Vec<f64>]) -> Vec<usize> {
    let mut result: Vec<usize> = vec![0; data.len()];
    for point in 0..data.len() {
        let mut min_dist = f64::MAX;
        for (idx, centroid) in centroids.iter().enumerate() {
            let dist = squared_euclidean_dist(&data[point], centroid);
            if dist < min_dist {
                min_dist = dist;
                result[point] = idx;
            }
        }
    }
    result
}

Updating centroids

After assigning points to clusters, the centroids are recalculated. This is achieved by computing the average of all points within each cluster.

fn compute_centroids(data: &[Vec<f64>], clusters: &[usize], k: usize)
    -> Vec<Vec<f64>> {
    let mut result = Vec::new();
    let feature_len = data.first().unwrap_or(&vec![]).len();
    for cluster in 0..k {
        let mut centroid = vec![0f64; feature_len];
        let mut cnt = 0;
        for point in 0..data.len() {
            if clusters[point] == cluster {
                for (idx, feature) in centroid.iter_mut().enumerate() {
                    *feature += data[point][idx];
                }
                cnt += 1;
            }
        }
        if cnt > 0 {
            for feature in centroid.iter_mut() {
                *feature /= cnt as f64;
            }
        }
        result.push(centroid);
    }
    result
}

Measuring progress with SSE

The algorithm measures its progress using the Sum of Squared Errors (SSE), which it aims to minimize. This sum is the total of the minimum squared distances between points and their respective centroids.

fn compute_sse(data: &[Vec<f64>], clusters: &[usize], centroids: &[Vec<f64>])
    -> f64 {
    clusters
        .iter()
        .zip(data.iter())
        .map(|(cluster, point)|
            squared_euclidean_dist(point, &centroids[*cluster]))
        .sum()
}

Convergence is reached when changes in SSE are zero or so small that further iterations do not significantly improve the clusters. This condition indicates that the centroids have stabilized, and the clusters are optimally formed.

Setting a maximum number of iterations prevents the algorithm from running too long, especially when changes in SSE are negligible to make a significant difference.

Putting it all together

Here is how the main k-means loop is implemented in Rust:

fn run_k_means(
    data: &[Vec<f64>],
    initial_centroids: &[Vec<f64>],
    max_iters: usize,
    eps: f64,
) -> (Vec<Vec<f64>>, Vec<usize>, f64, usize) {
    let k = initial_centroids.len();
    let mut centroids = initial_centroids.to_vec();
    let mut clusters = vec![0; data.len()];
    let mut sse = f64::MAX;
    let mut iters = 0;
    for _i in 0..max_iters {
        clusters = compute_clusters(data, &centroids);
        centroids = compute_centroids(data, &clusters, k);
        let prev_sse = sse;
        sse = compute_sse(data, &clusters, &centroids);
        iters += 1;

        if (prev_sse - sse) / sse < eps {
            break;
        }
    }
    (centroids, clusters, sse, iters)
}

Now, let’s revisit the various initialization methods mentioned earlier.

K-Means initialization methods

There’s a lot of confusion around k-means initialization methods in the literature and practice. For instance, some sources refer to K-means++ when they’re actually implementing the Maximin method. Others incorrectly describe MacQueen’s method, which uses random initial centers, as Forgy’s.

Let’s clear up these misconceptions and explore how each method functions and how to implement them.

Forgy: random assignment

The Forgy method randomly assigns each data point to one of k clusters. It then calculates the centroids of these initial clusters.

pub fn init_centroids(data: &[Vec<f64>], k: usize) -> Vec<Vec<f64>> {
    let range = Uniform::from(0..k);
    let mut rng = rand::thread_rng();

    let clusters: Vec<usize> = data.iter()
        .map(|_| range.sample(&mut rng)).collect();

    compute_centroids(data, &clusters, k)
}

MacQueen: random sampling

The MacQueen method, also known as “Random Sampling”, selects k random data points from the dataset to serve as initial centroids.

pub fn init_centroids(data: &[Vec<f64>], k: usize) -> Vec<Vec<f64>> {
    let mut rng = rand::thread_rng();
    data.choose_multiple(&mut rng, k).cloned().collect()
}