🧱 Building Infrastructure with Docker — Part 0

The Journey to Reusable, Independent Infrastructure Modules

When you start a new project, how many times do you rebuild the same supporting infrastructure — a database, a message broker, a cache, a monitoring stack — before writing your first line of application logic?

Most engineers do it every single time.

This series aims to end that cycle.

💡 Why This Series Exists

We’re going to create a library of self-contained, production-grade infrastructure projects, each running in Docker and built to be reused across any system.

Each infrastructure component — PostgreSQL, Kafka, Mosquitto (MQTT), Keycloak, Prometheus, Grafana, Jaeger, MongoDB, Ignite, and Debezium — will live in its own independent project, with deterministic builds and persistent local volumes.

By the end of this series, you’ll be able to:

• Run any of these infrastructures independently.
• Combine them into a working data flow (for example: MQTT → Kafka → PostgreSQL → Debezium → Grafana).
• Understand how each component fits into real-world enterprise systems.

⚙️ Our Philosophy

This is not another “run this Docker command” tutorial. We’ll treat each infrastructure component with the same engineering discipline used in production environments:

If you follow along, you’ll learn not just how to run infra, but how to engineer it right.

• 🧩 The Architecture We’ll Build (Concept)

  At a high level, we are building an event telemetry pipeline that passes through several layers of infrastructure.

  Each component will be implemented and validated one at a time. Later in the series, we will show small, optional integration examples that stitch these modules together — without breaking their independence.

  ---

  🧱 Series Roadmap

  Part	Topic	Focus
  0	Introduction & Project Layout	(this post) – goals, repo structure, workflow
  1	PostgreSQL	persistent DB, healthcheck, seed DDL
  2	Kafka + Zookeeper	message backbone
  3	Mosquitto MQTT	lightweight telemetry broker
  4	Debezium	change-data-capture from Postgres
  5	Keycloak	authentication: UI login + API tests via curl
  6	Prometheus	scraping metrics and targets
  7	Grafana	dashboards and provisioning
  8	Jaeger	distributed tracing
  9	MongoDB	document store basics
  10	Apache Ignite	in-memory data grid with persistence
  11	Node Exporter + cAdvisor	system and container metrics
  12	Putting It All Together	simple telemetry-to-dashboard demo

  Each part will come with runnable code and configuration inside a single Git repository.

  ---

  📂 How the Git Repository Is Organised

  All modules in this series live in one repo, but each module stays isolated inside its own folder under modules/.

  A simplified view of the parent project structure:

  infra-docker-series/
    README.md
    Makefile
    docs/
      part-0.md
      roadmap.md
    tooling/
      mk/
        common.mk
      scripts/
        subtree-export.sh
    modules/
      postgres/
        ... PostgreSQL-specific content (Part 1) ...
      kafka/
        ... Kafka-specific content (Part 2) ...
      mosquitto/
        ... Mosquitto-specific content (Part 3) ...
      keycloak/
        ... Keycloak module (Part 5) ...
      debezium/
        ... Debezium module (Part 4) ...
      prometheus/
        ... Prometheus module (Part 6) ...
      grafana/
        ... Grafana module (Part 7) ...
      jaeger/
        ... Jaeger module (Part 8) ...
      mongodb/
        ... MongoDB module (Part 9) ...
      ignite/
        ... Ignite module (Part 10) ...
      node-exporter-cadvisor/
        ... metrics helper module (Part 11) ...
    orchestration/
      README.md
      snippets/
        ... small integration examples (later in the series) ...

  What you need to know as a reader

  • You only need this one repository for the entire series.
  • Every infrastructure piece lives in a separate folder under modules/.
  • Each module will have its own:
    • infra/docker-compose.yml
    • docker-volume/ directory for data
    • Makefile with standard targets (init, up, test, down, clean)
    • docs/ with module-specific notes

  We will design and explain each module’s inner structure in its own part of the series.

  ---

  🧰 Root Makefile: A Simple Entry Point

  The root Makefile exists only to make your life easier. It does not run any combined stack on its own; it just forwards commands to individual modules.

  Typical usage from the root of the repo:

  # List available modules
  make list
  
  # Bring up a specific module (for example, PostgreSQL)
  make up MODULE=postgres
  
  # Run its smoke tests
  make test MODULE=postgres
  
  # Tear it down
  make down MODULE=postgres

  Internally, this just runs make inside modules/postgres/ (or whatever module you choose). Each module still has its own Makefile and can be used directly from its folder if you prefer.

  ---

  🧱 Inside a Module (High-Level View)

  In later parts, we will deep dive into each infrastructure module and define its full structure. For now, this is the high-level pattern each module will follow:

  modules//
    docs/
      README.md
      requirements.md
      design-intent.md
    infra/
      docker-compose.yml
    docker-volume/
      ... data, bind-mounted ...
    scripts/
      test_health.sh
    .env.example
    Makefile
    Jenkinsfile
    .gitignore

  You will see concrete content starting from Part 1 (PostgreSQL). The key point now: all modules share the same shape and workflow, so once you learn one, the others feel familiar.

  ---

  🧑‍💻 How You Will Work With This Repo

  1. Clone the repository once:

     git clone https://github.com//infra-docker-series.git
     cd infra-docker-series

  2. See what modules exist:

     make list

  3. Pick a module and work in isolation:

     make init MODULE=postgres
     make up MODULE=postgres
     make test MODULE=postgres
     make down MODULE=postgres

  4. When we introduce new modules (Kafka, Mosquitto, Keycloak, etc.), you will follow the same pattern, simply changing MODULE=<name>.

  Later in the series, we will add optional orchestration snippets under orchestration/ that show how to connect multiple modules together. These will be off by default so that modules remain clean and independent.

  ---

  🧭 Learning Philosophy and Workflow

  For every module, we follow the same three-phase approach:

  1. Design
     • Explain the purpose of the module.
     • Decide image versions, ports, environment variables.
     • Document everything in docs/requirements.md and docs/design-intent.md.
  2. Build
     • Write infra/docker-compose.yml with pinned image tags.
     • Add a Makefile with init, up, test, down, clean.
     • Prepare .env.example for configuration.
  3. Test
     • Implement scripts/test_health.sh.
     • Run a small, meaningful smoke test:
       • PostgreSQL: SELECT 1
       • Kafka: produce/consume a message
       • Mosquitto: mosquitto_pub / mosquitto_sub
       • Keycloak: Login and call a protected API
     • Validate logs and cleanup.

  If you follow these steps with each part, you will build the same discipline in your own projects.

  ---

  ⚠️ Pitfalls We Intentionally Avoid

  • Putting everything into one giant docker-compose file → Hard to reason about, hard to test in isolation. We keep each infrastructure module separate.
  • Using latest tags → Leads to surprising upgrades and broken setups. We always pin explicit versions per module.
  • Mixing app code with infra → Infrastructure from this repo can be reused in multiple applications. Applications live elsewhere; this repository focuses on infra only.
  • Committing secrets → We version .env.example only. Your real .env stays on your machine and is .gitignored.

🔜 Coming Up Next — Part 1: PostgreSQL

We’ll begin with the most fundamental component — PostgreSQL. You’ll learn how to:

• Pin a specific image version.
• Bind a local persistent volume under ./docker-volume/postgres/data.
• Add a startup healthcheck.
• Seed a schema and validate connectivity using psql.
• Clean up safely with make down.

📘 Final Thoughts

This series isn’t about copying commands — it’s about thinking like an infrastructure engineer. By the end, you won’t just know how to “run Docker,” you’ll understand how to build and maintain reliable infrastructure stacks that scale from your laptop to production.

Let’s begin.

GitHub Repository Link

🔗 Project Repo: https://github.com/KathiravanMuthaiah/infrastructureWithDocker

Building Infrastructure with Docker Series: post links

🔗 Part1:

🔗 Part2:

“Technically authored by me, accelerated with insights from ChatGPT by OpenAI.” Refer: Leverage ChatGPT

Happy Learning