from zero to hero with rust: build a blazing-fast cli tool and crush i/o bottlenecks in minutes

@pilput

8 months ago0views

why rust for cli tools?

rust empowers beginners and seasoned engineers to build fast, safe, and reliable command-line tools. its zero-cost abstractions and fearless concurrency help you crush i/o bottlenecks without sacrificing readability. if you’re coming from full stack or devops, rust’s tooling, package manager (cargo), and strong community will feel refreshingly productive.

performance: native speed with zero-cost abstractions.
safety: compile-time guarantees prevent data races and many runtime crashes.
portability: generate a single static binary that’s easy to ship in ci/cd pipelines.
great tooling: cargo for build/test/publish, rustfmt for formatting, clippy for linting.

what we’ll build

a blazing-fast cli that scans files and directories, counts lines, words, and bytes (like a mini wc), and supports:

recursive directory traversal
parallel processing for i/o-bound workloads
globbing (file patterns)
colorful, human-friendly output

along the way, you’ll pick up skills useful for devops, full stack, and seo-minded content workflows (e.g., batch-analyzing content repositories, logs, or sitemap exports).

prerequisites

install rust and cargo: curl --proto '=https' --tlsv1.2 -ssf https://sh.rustup.rs | sh
familiarity with basic terminal usage
optional: ripgrep mindset—search large codebases and logs with speed

project setup

# create a new binary project
cargo new fastwc --bin
cd fastwc

# add helpful dependencies
# - clap: cli args
# - walkdir: directory traversal
# - rayon: parallelism
# - globset: fast glob matching
# - colored: colored output
# - anyhow: ergonomic error handling
cargo add clap --features derive
cargo add walkdir rayon globset colored anyhow

designing the cli

we’ll design a simple interface:

fastwc [options] <path>...

options:
  -r, --recursive     recurse into subdirectories
  -g, --glob <pat>    only include files matching glob (e.g., **/*.rs)
  -p, --parallel      enable parallel processing
  -q, --quiet         less verbose output
  -h, --help          show help

src/main.rs (minimal version)

use std::{fs::file, io::{self, bufread, bufreader}, path::path};
use clap::{arg, command};
use walkdir::walkdir;
use globset::{glob, globsetbuilder};
use rayon::prelude::*;
use colored::*;
use anyhow::{result, context};

#[derive(debug)]
struct counts {
    lines: u64,
    words: u64,
    bytes: u64,
}

fn count_file(path: &path) -> result<counts> {
    let file = file::open(path).with_context(|| format!("open {:?}", path))?;
    let mut reader = bufreader::new(file);
    let mut buf = string::new();
    let mut lines = 0u64;
    let mut words = 0u64;
    let mut bytes = 0u64;

    loop {
        buf.clear();
        let read = reader.read_line(&mut buf)?;
        if read == 0 { break; }
        lines += 1;
        words += buf.split_whitespace().count() as u64;
        bytes += read as u64;
    }

    ok(counts { lines, words, bytes })
}

fn build_globset(patterns: &vec<string>) -> result<globset::globset> {
    let mut builder = globsetbuilder::new();
    for p in patterns {
        builder.add(glob::new(p)?);
    }
    ok(builder.build()?)
}

fn main() -> result<()> {
    let matches = command::new("fastwc")
        .about("blazing-fast line/word/byte counter")
        .arg(arg::new("path").required(true).num_args(1..).help("files or directories"))
        .arg(arg::new("recursive").short('r').long("recursive"))
        .arg(arg::new("glob").short('g').long("glob").num_args(1..))
        .arg(arg::new("parallel").short('p').long("parallel"))
        .arg(arg::new("quiet").short('q').long("quiet"))
        .get_matches();

    let recursive = matches.get_flag("recursive");
    let parallel  = matches.get_flag("parallel");
    let quiet     = matches.get_flag("quiet");
    let globs: vec<string> = matches.get_many::("glob")
        .map(|vals| vals.cloned().collect()).unwrap_or_default();

    let globset = if globs.is_empty() { none } else { some(build_globset(&globs)?) };

    let paths: vec<string> = matches.get_many::("path")
        .unwrap().map(|s| s.to_string()).collect();

    // collect candidate files
    let mut files = vec::new();
    for p in paths {
        let meta = std::fs::metadata(&p).with_context(|| format!("metadata {}", p))?;
        if meta.is_file() {
            files.push(std::path::pathbuf::from(&p));
        } else if meta.is_dir() {
            if recursive {
                for entry in walkdir::new(&p).into_iter().filter_map(result::ok) {
                    if entry.file_type().is_file() {
                        files.push(entry.path().to_path_buf());
                    }
                }
            } else {
                for entry in std::fs::read_dir(&p)? {
                    let entry = entry?;
                    if entry.file_type()?.is_file() {
                        files.push(entry.path());
                    }
                }
            }
        }
    }

    // apply glob filtering
    if let some(gs) = &globset {
        files.retain(|f| {
            if let some(s) = f.to_str() {
                gs.is_match(s)
            } else {
                false
            }
        });
    }

    if files.is_empty() {
        eprintln!("{}", "no files matched.".yellow());
        return ok(());
    }

    let process = |f: &std::path::pathbuf| -> option<(std::path::pathbuf, counts)> {
        match count_file(f) {
            ok(c) => some((f.clone(), c)),
            err(e) => {
                eprintln!("{} {:?}", "skip (error):".red(), e);
                none
            }
        }
    };

    let results: vec<(std::path::pathbuf, counts)> = if parallel {
        files.par_iter().filter_map(process).collect()
    } else {
        files.iter().filter_map(process).collect()
    };

    let mut total = counts { lines: 0, words: 0, bytes: 0 };
    for (path, c) in &results {
        total.lines += c.lines;
        total.words += c.words;
        total.bytes += c.bytes;
        if !quiet {
            println!("{}\t{}\t{}\t{}",
                c.lines.to_string().cyan(),
                c.words.to_string().green(),
                c.bytes.to_string().blue(),
                path.display()
            );
        }
    }

    println!("{}\t{}\t{}\t{}",
        total.lines.to_string().bold(),
        total.words.to_string().bold(),
        total.bytes.to_string().bold(),
        "total".bold()
    );

    ok(())
}

understanding the performance

i/o-bound speedups: enabling --parallel uses rayon to process files concurrently, ideal when reading many small files.
reduced allocations: reuse a buffer (buf) while reading line by line.
zero-cost abstractions: iterators compile down efficiently; no garbage collector pauses.

try it out

# count rust files recursively, parallel
cargo run -- -r -p -g "**/*.rs" .

# count everything in current directory (non-recursive)
cargo run -- .

# quiet mode for scripts
cargo run -- -r -p -q .

devops integrations

this binary is ideal for ci/cd pipelines where you need fast content or log analysis.

pre-merge checks: ensure repo size or line counts stay within limits.
build metrics: track lines changed per build.
container-friendly: ship a single static binary in lightweight images.

# example github actions step
- name: fast line/word/byte metrics
  run: |
    cargo build --release
    ./target/release/fastwc -r -p -g "**/*.{rs,md,html}" . > fastwc.txt
    echo "metrics saved to fastwc.txt"

full stack and seo use cases

full stack: measure template sizes, api response samples, or content payloads in development.
seo: analyze large content repositories, count words per article, or validate sitemap exports quickly.
content ops: batch-inspect markdown/html files for word counts before publishing.

# seo example: count words in content folder
fastwc -r -p -g "content/**/*.md" content/

benchmarking tips

use hyperfine to compare against other tools.
warm up the disk cache by running once before measuring.
benchmark different buffer strategies (e.g., read_to_end vs line-by-line) for your dataset.

cargo install hyperfine
hyperfine './target/release/fastwc -r -p -g "**/*.rs" .' 'wc -l $(git ls-files "*.rs")'

quality and safety

clippy: cargo clippy -- -d warnings to keep code idiomatic.
formatting: cargo fmt for consistent style.
testing: add unit tests for counting logic and integration tests for cli behavior.

example unit test

// add to src/lib.rs if you extract logic there
#[cfg(test)]
mod tests {
    use super::*;
    use std::io::write;
    use tempfile::namedtempfile;

    #[test]
    fn counts_basic() {
        let mut f = namedtempfile::new().unwrap();
        writeln!(f, "hello world").unwrap();
        writeln!(f, "rust rocks").unwrap();

        let c = super::count_file(f.path()).unwrap();
        assert_eq!(c.lines, 2);
        assert_eq!(c.words, 4);
        assert!(c.bytes >= 22);
    }
}

packaging and distribution

release: cargo build --release for optimized binaries.
cross-compile: use cross to build for linux, macos, windows in ci.
versioning: tag releases and attach binaries for easy downloads.

cargo install cross
cross build --release --target x86_64-unknown-linux-musl

common pitfalls and fixes

unicode counts: the example uses byte length and split_whitespace. if you need grapheme clusters, use unicode-segmentation.
large files: line-by-line iteration is memory-friendly; avoid reading entire files unless necessary.
symlinks: decide whether to follow them; walkdir has options to control this.
error handling: we skip unreadable files but log errors; adjust for stricter behavior if needed.

next steps

add json/csv output for dashboards.
expose a library crate for reuse in other rust services.
add progress bars with indicatif.
implement async i/o for network-bound tasks using tokio.

key takeaways

rust lets you go from zero to a production-grade cli quickly.
parallel file processing and careful i/o dramatically reduce bottlenecks.
great fit for devops, full stack, and seo workloads that need speed and reliability.

cheat sheet

build: cargo build --release
run recursive + parallel: fastwc -r -p .
filter by glob: fastwc -g "**/*.md" content/
ci step: compile, run, save report