Crossplane by example on GCP

Crossplane is an open source Kubernetes add-on that enables platform teams to assemble infrastructure from multiple vendors, and expose higher level self-service APIs for application teams to consume, without having to write any code. It allows you to compose cloud infrastructure and services based on XRD (cross resource definitions) that extends the existing Kubernete CRD (Custom Resource Definition). To achieve this awesome goal, you have to use various repositories that reside in the GitHub crossplane, crossplane-contrib and upbound organisations. As adaptor of that new technology, you can rely the official documentation where a lot of details are gathered.

To ease our understanding and document our experiments, we have created a crossplane-example repository that will take you step-by-step to use Crossplane to deploy your infrastructure on top of Google Cloud and also develop a user interface and Helm chart that access a database created by Crossplane.

Crossplane community is welcoming, just like the Crossplane logo is fun!

We are thankful for the community support received on Slack as for the community videos found on the Youtube channel.

We hope that the examples will be useful to new-comers. Follow the steps documented in the README, the details for those steps to take are in this folder.

The next sections bring more background on Crossplane and how Datalayer, and hopefully others like you, can benefit from those awesome and unique features.

Cloud and Gitops native

Back in 2018, we have developed the very first interation of the Datalayer platform on Kubernetes with home-made forged tools like kuber-go, now in the attic, to deploy and manage Kubernetes on various clouds.

We have continued looking at tools like:

• Terraform which is in the IaC (Infrastructure as Code) vein, with proprietary language.
• Pulumi is also IaC with multi-language support but needing an account on the pulumi systems.
• Cluster API which is limited to the Kubernetes cluster creation.

Instead of those standard approach, we have been thrilled by the pure declarative way, a-la-kubernetes, that Crossplane provides to operate the cloud infrastructure and services you need. So instead of building our own cloud-chain toolkit, we have decided to embrace the GitOps principles with YAML declarations that allow to create all kind of infrastructure and service resources. The reconciliation loops implemented by Kubernetes Crossplane operators ensure that any drift is fixed.

If your are allergic to YAML, which we can understand, you may pass your turn. If you really can not live without Terraform, you will be interested in some work in progress to better support the Terraform ecosystem.

A rich ecosystem

Cloud providers are available to create managed infrastructure, like Kubernetes clusters, databases, IAM roles, service accounts... on the major clouds like Google cloud, Amazon AWS, Microsoft Azure...

You don't have to limit to the infrastructure side, you can also deploy applications with Helm, create SQL resources like role on your database. The list continues on and on, just have a look at the repositories in crossplane-contrib GitHub organization.

On top of the managed infrastructure, you can create custom compositions that ships platform composed of managed infrastructures and services. A platform could be for example the sum of a Kubernetes cluser, a Database, a Bucket and Helm charts deployed on the Kubernetes cluster, all working together, meaning that the application would connect to the Database and the Bucket. There are already a few of those reference platforms available.

To navigate the growing CRD/XRD definitions, you can use a documentation website for CRD/XRD. You give the repository hosting the CRD/XRD and the website will introspect and generate for you a nice documentation page, see for example the generated documentation for the Google cloud provider.

Future-proof

If you are still reluctant to invest in that burgeoning technology, here are a few arguments.

• Crossplane is already a CNCF sandbox project, being now proposed to migrate to incubation status.

• The latest Kubecon Europe 2021 has shown a lot of participants and traction.

• The current version is 1.2 and demonstrates enough maturity and stability.

• Some customers claim to run it in production environment.

• Finally, the open-source repositories are backed by a commercial company, Upbound, which also operates a cloud to host your custom configurations.

🛫 A Plane for Datalayer

At Datalayer, we are pivoting our cloud solutions to leverage more Crossplane goodies with 2 tracks:

1. Developing clouder, a software component to better integrate Crossplane with Flux, a CI/CD solution that makes your GitOps flow a breeze. Clouder will deliver easier bootstraping and a WEB user interface for operators to manage and monitor the resources.

2. Building an opiniated configuration platform jupyter-crossplane-gcp for Jupyter notebooks, reusing the existing JupyterHub Helm chart.

Stay tuned for news on those initiatives!