Work / Migration
Migration, March 2024
3,000 microservices moved to Kubernetes with no downtime
A financial technology company ran more than 3,000 microservices on more than 300 physical servers. We moved every service to a Kubernetes platform with GitOps delivery. The server count fell by 40 % and no service was offline during the move.
The estate before the migration
The client ran more than 3,000 microservices, written in Java and Go, on more than 300 physical servers. Every service was deployed by hand. The estate had four problems.
Hardware refresh
The physical servers were ageing and had to be replaced every 3 to 4 years at a high cost.
Low utilization
Average CPU utilization was below 20 %, and memory was over-provisioned across the clusters.
Manual deployment
Each service took 2 to 4 hours to deploy by hand, and a high share of deployments failed.
No automatic scaling
Capacity could not grow during peak traffic, so services degraded at peak.
The migration method
We moved the services in batches with the strangler fig pattern. In this pattern, new services on the Kubernetes platform take over traffic from the old services one group at a time, until the old servers carry no traffic and are retired. No service was offline at any step. The work ran in five phases.
Plan
Setup
Pilot
Batches
Cutover
The platform
We designed a multi-cluster Kubernetes platform with one management plane, GitOps deployment and one observability stack for every service. The platform has three parts. Rancher gives one control plane for every cluster, and Kong Gateway applies rate limits and authentication in front of the services.
Production 40 nodes
Staging 15 nodes
Development 10 nodes
Technology stack
| Area | Tools |
|---|---|
| Orchestration | KubernetesRancherHelm |
| CI and CD | ArgoCDJenkinsGitLab CI |
| Monitoring | PrometheusGrafanaJaeger |
| Security | TrivySonarQubeVault |
The deployment pipeline
We built a GitOps pipeline that runs without manual steps. A deployment now takes 5 minutes instead of 2 to 4 hours. The pipeline has six stages. Jenkins or GitLab CI builds the image. Trivy and SonarQube scan the code and the image. Harbor stores the image, ArgoCD applies the change from Git and Kubernetes performs a rolling update.
| Measure | Before | After |
|---|---|---|
| Deployment time per service | 2 to 4 h | 5 min |
| Deployments completed without failure, % | 60 | 99.5 |
| Rollback | Manual | Automated |
Security controls
The client is a financial technology company, so we applied controls at every layer of the platform.
No known CVE in production
The pipeline blocks an image with a known vulnerability before it is deployed.
Secrets in Vault
HashiCorp Vault issues secrets dynamically and rotates them.
Network isolation
Kubernetes network policies limit traffic between services to the paths that are declared.
Assessment passed
The platform passed the client's internal red team and blue team security assessments.
Observability
Every one of the 3,000 services reports metrics, traces and logs to one stack.
Prometheus collects metrics. Grafana shows dashboards and raises alerts.
Jaeger traces each request across the services it calls.
The ELK stack stores the logs of every service in one place, integrated with Splunk.
Application monitoring with Dynatrace
For the critical workloads we deployed Dynatrace OneAgent. The agent instruments every service without a code change or an SDK. Davis, the Dynatrace analysis engine, identifies performance limits and their root causes. OneAgent is installed as a DaemonSet on every node and detects every service, container, pod and dependency. Performance of each service is recorded continuously. Davis raises an alert within seconds of an anomaly, with a root cause and a recommended change.
Resource figures
Kubernetes places several services on each node, so the same load runs on fewer servers at higher utilization.
| Resource | Before, physical | After, Kubernetes | Change |
|---|---|---|---|
| CPU utilization, % | 15 to 20 | 60 to 70 | 3.5 times |
| Servers | 300+ | 180+ | 40 % fewer |
| Deployment time per service | 2 to 4 h | 5 min | 95 % shorter |
| Operations effort | Very high | Moderate | Standardized |
What the client received
- Three Kubernetes clusters, production, staging and development, with automatic scaling and one management plane in Rancher.
- A GitOps pipeline from commit to rolling update, with a security scan at every build.
- Security controls at the code, container, runtime and infrastructure layers.
- One observability stack for metrics, traces and logs, and Dynatrace on the critical workloads.
- Runbooks for every migration pattern and training for the client's team.
- The physical servers retired after cutover, with the cost saving validated.
A migration plan for your services
We map the dependencies of your services, set up the platform, move a pilot group and then the rest in batches. Each batch has a tested rollback. You receive the platform, the pipeline, the runbooks and the documentation.
Talk to an engineer