Checker

node.js, axios, cheerio, commander, p-limit

A CLI tool that crawls any website and identifies broken links, redirects, and dead pages.

view project

Challenge

The goal was to learn how web crawlers and link checkers work.

Approach

I built a BFS crawler with concurrent HEAD/GET requests, robots.txt compliance, and binary asset detection, wrapped in a terminal interface with live output and JSON reporting.

Outcome

The result is a fast, scriptable broken link checker that works on localhost before you push or as a CI/CD step to block deploys with dead links.

Highlights

  • BFS crawl with configurable depth and concurrency
  • works on localhost and live sites
  • CI/CD ready with --exit-code flag
NextBleed