DevOps has revolutionized the way software development and deployment are approached, enabling organizations to achieve faster, more reliable, and scalable workflows with declarative tooling. Given the opportunity to architect a new suite of applications hosted with CloudNative solutions, I envision a setup that combines the power of Terraform, GitOps, and AWS (typically I use AWS, but this workflow can be applied to most cloud providers). In short: a foundational "infrastructure" repository is used alongside individual "infrastructure" folders for each application or service, leveraging the strengths of these tools to create a seamless workflow.
Components
Terraform
Terraform, an open-source infrastructure as code (IaC) tool, serves as the foundation of this architecture. It allows us to define and provision infrastructure resources across various cloud providers in a declarative and consistent manner. By utilizing Terraform, we gain the ability to create, modify, and manage infrastructure as code, enabling easy collaboration and version control.
GitOps
GitOps, an operating model that applies the principles of Git version control to infrastructure management, plays a crucial role in this workflow. Infrastructure changes are driven through a single source of truth: version-controlled Git repositories, which enhances visibility, traceability, and collaboration, as well as simplified rollbacks when necessary. It also allows any engineer, not just the DevOps team, to propose changes to the infrastructure without explicit cloud write access.
My DevOps Dream
Foundational Infrastructure
At the center of my DevOps dream is the "Infrastructure" repository. This repository serves as a hub for managing the overall infrastructure configuration. It contains the Terraform code that defines foundational components, such as networking, security groups, EKS (Elastic Kubernetes Service) clusters, etc.
📁 Infrastructure Repository
├── 📄 README.md
├── 📄 main.tf
├── 📄 variables.tf
├── 📁 modules/
│ ├── 📁 networking/
│ │ ├── 📄 main.tf
│ │ ├── 📄 variables.tf
│ │ └── ...
│ ├── 📁 compute/
│ │ ├── 📄 main.tf
│ │ ├── 📄 variables.tf
│ │ └── ...
│ └── ...
├── 📁 environments/
│ ├── 📁 dev/
│ │ ├── 📄 main.tfvars
│ │ └── ...
│ ├── 📁 staging/
│ │ ├── 📄 main.tfvars
│ │ └── ...
│ └── 📁 production/
│ ├── 📄 main.tfvars
│ └── ...
└── 📁 scripts/
├── 📄 deploy.sh
├── 📄 destroy.sh
└── ...
Service Infrastructure
To maintain a modular and self-contained approach to infrastructure management, each application or service has its own "infrastructure" folder within the respective code repository. These folders encapsulate the infrastructure resources required by the application, such as load balancers, databases, or storage buckets. Note that these resources are outside of the scope of "foundational" to the overall architecture. By having separate infrastructure definitions for each application, we can manage and deploy them independently, facilitating rapid iteration and deployment of new features.
📁 Application Infrastructure
├── 📄 README.md
├── 📄 main.tf
├── 📄 variables.tf
├── 📁 modules/
│ ├── 📁 load_balancer/
│ │ ├── 📄 main.tf
│ │ ├── 📄 variables.tf
│ │ └── ...
│ ├── 📁 database/
│ │ ├── 📄 main.tf
│ │ ├── 📄 variables.tf
│ │ └── ...
│ └── ...
├── 📁 scripts/
│ ├── 📄 deploy.sh
│ ├── 📄 destroy.sh
│ └── ...
├── 📁 app/
│ ├── 📄 Dockerfile
│ ├── 📄 app.py
│ └── ...
└── 📁 tests/
├── 📄 test_suites.py
└── ...
Leveraging GitOps
Using GitOps, any changes made to the service repository (specifically the infrastructure folder), or the foundational infrastructure repository, will trigger a pipeline that applies the changes to the associated infrastructure in AWS. This ensures that infrastructure updates are appropriately scoped, seamless, auditable, and consistent across environments.
Summary
This vision provides a cohesive and efficient infrastructure management system that harnesses the power of Terraform and GitOps. By leveraging the foundational repository alongside individual infrastructure folders for services, we can achieve a streamlined workflow with improved collaboration, version control, and deployment agility. It empowers engineers to focus on building reliable and scalable applications, while maintaining a clear separation of infrastructure concerns and appropriate cloud access permissions.
Additional Considerations
Though my preferences lie with Terraform, the same workflow could be achieved with other IaC tooling such as AWS CDK or Ansible. Additionally, automated GitOps could be removed from the workflow by relying on the DevOps team to manually provision new changes post-merge.
