Nov. 7, 2023, 10:55 p.m.
In this post I would like to do a quick breakdown on Google Cloud Platform (GCP) and cloud computing in general.
Let’s get started by examining cloud computing. Cloud computing is the process of using third party resources to perform IT functions. These usually include but are not limited to: computing, storage, networking, and dev ops.
A cloud provider, like Google in this case, gives access to it’s resources for a fee.
Google has data centers (locations where the physical resources are stored) located all over the globe. Each data center is located in a region, which in turn is made up of zones. This becomes important when designing systems because locating resources in the same geographic area that requests are coming from will increase performance. Furthermore, spreading resources out across different regions / zones will improve redundancy and disaster recovery.
GCP resources are attached to projects. A project can be thought of as an organizing entity for the resources contain therein; a project contains the shared settings, permissions, and other metadata that will be applied to the resources it contains. Resources within a single project work together quite easily. If you have resources in one project which need to communicate with resources in another you can do this using one of two Networking tools: Shared VPC, or VPC Network peering.
You can interact with GCP in one of three ways: through the Google Cloud Con- sole (a web based GUI), through the Command Line Interface (access through the terminal), and Client API libraries.
Let’s jump into a high level overview of GCP’s main services–this is no way an exhaustive list.
App Engine is a fully managed server less platform for building web applications. Basically it’s a place where you can focus on development and let GCP take care of all the DevOps activities eg. provisioning servers and scaling your app to meet demand.
App Engine comes in two flavours: Standard and Flexible. The standard envi- ronment does not give you much control over how the underlying infrastructure (containers) are configured, but you get the benefit of not having to worry about configuring that infrastructure. The Flexible environment on the other hand gives you slightly more control; your application is run in Docker con- tainers on Compute Engine Virtual Machines (VMs). Some examples of things you can do in the Flexible environment that you cannot do in the Standard: write your app in any programming language, adjust the Compute Engine ma- chine type, access external services through GCP client libraries (making your
app more portable), and writing your own health checks. You can think of the Flexible environment as an intermediary step between the hands off Standard Environment and the very hands on Google Kubernetes Engine (GKE) which will be discussed below.
Kubernetes is a software for managing containerized applications. It handles deploying applications, rolling out changes, scaling, and monitoring. GKE is a cloud based Kubernetes service.
Some key concepts to understand about Kubernetes:
Clusters are groups of nodes (computers) that handle the computation. Pods (described below) run on nodes. They are managed by a control plane which usually runs across multiple computers. The components of the control plane are as follows: kube-apiserver: exposes the kubernetes api. This is how the control plane receives information from the nodes.
etcd: stores all cluster data.
kube-scheduler: assigns pods to nodes.
kube-controller-manager: runs controller processes eg responding when nodes go down, watches for jobs then creates pods to run those jobs, creates service accounts for new namespaces etc.
cloud-controller-manager: allows you to link your cluster with your cloud providers api.
kubelet: an agent that runs on each node in the cluster and ensures that the containers are running in a pod.
Kube-proxy: maintains network rules on nodes.
container runtime: manages the execution and life-cycle of containers.
Workload Components: Pod: smallest deployable unit of computing. It is a group of one or more (but usually one) containers that shares computing re- sources.
Deployments: declarative updates for Pods and Replicasets.
ReplicaSet: a group of identical pods. Usually used to grantee the availability of a specified number of pods.
StatefulSet: used to manage stateful applications. Basically the same as a Deployment, but the pods being managed are assigned ids and requests are tied to a specific pod so that stateful information can be retrieved from a Persistent Volume.
DaemonSet: A DaemonSet ensures all or some nodes run a copy of a pod. Common uses are running logs collection on all nodes, running monitoring, running a daemon to handle cluster storage.
Jobs: A job creates pods and will continue to retry execution of the pods until a specified number of them successfully terminate. As they complete the Job tracks it’s progress.
Compute Engine is a service that allows you to create virtual machines on GCP infrastructure. What these machines do is entirely up to you. You can also create instance groups which are groups of identical virtual machines that are deployed according to a template.
Is a service for writing small snipits of code that respond to events or messages in your cloud environment.
Is a service for running containers that are invocable via events or messages. I like to think of Cloud Run as a more in-depth version of Cloud Functions.
Cloud Storage is bucket based storage available across GCP. Buckets are con- tainers you can store objects in.
Cloud Storage has the following storage classes: Standard: for “hot” data that is frequently accessed Nearline: for data accessed once every 30 days or less Cold- line: for data accessed once every 90 days or less Archive: for data accessed once a year or less
Persistent Disk are durable network storage devices that your VMs can access like the physical hardware disks of a computer. They are block storage devices.
Also block storage devices, but these are physically attached to the server that hosts the VM. This offers superior performance, especially for operations that require high input output operations per second or low latency.
Cloud Filestore is GCP’s network file system. It gives VMs, Kubernetes clusters, and on prem machines access to a file system hosted in the cloud.
This is tool for building ETL pipelines without managing any of the infrastruc- ture. Basically just using a graphical interface to plan out the pipeline and then run it.
Big Query is Google Cloud’s Data Warehousing solution. A Data Warehouse is a collection of structured data organized for a particular purpose. Some common methods of loading data into BigQuery are listed below:
Batch Loading Load jobs: are a BigQuery utility used to load data from Cloud Storage or from a local file.
SQL: Use the LOAD DATA statement
BigQuery Data Transfer Service: automated batch loading on a scheduled and managed basis
BigQuery Storage Write API: Using the API you can batch process an arbitrarily large number of records in a single atomic operation. The difference between the API and load jobs is the API does not require you to stage the data to intermediate storage such as Cloud Storage.
Other managed services also sometimes have options to export to BigQuery.
NOTE: as a general rule most file based transfers should use load jobs or SQL LOAD DATA statements.
Streaming Data BigQuery Storage Write API: this can be used for high-throughput streaming ingestion.
Dataflow: can be used with the Apache Beam Software Development Kit to set up a streaming pipeline. You could do this directly or route it through Pub/Sub first. Dataflow is a managed service for processing data.
DataStream: can be used to update data and schemas automatically using BigQuery’s change data capture function and the Storage Write API.
Pub/Sub: This is a messaging service you can use to coordinate streaming analytics and data integration pipelines. You can use BigQuery Subscriptions to write messages directly to an existing BigQuery table.
A managed Apache Spark and Hadoop service that is used for batch processing, querying, streaming, and machine learning. Helps you create hadoop clusters quickly, manage them, and turn them off when you don’t need them.
Managed Apache Airflow service that helps you create, schedule, and monitor workflows.
A NoSQL database which uses a document model (kind of like a python dictio- nary) to store data. It is often used in app development.
GCP’s managed NoSQL big data service.
Cloud SQL is a managed service for managing relational data bases using MySQL, PostgreSQL, or SQL Server.
A more powerfull SQL solution that offers transactional consistency at a global scale, and synchronous replication of data for high availability.
Google’s caching solution. Provides a managed service for handling Redis and Memcache.
GCP has several types of load balancers which can be organized according to the protocol, Wether they are global or regional, and whether they are external or internal. For reference a load balancer is a too used to spread requests for a service across a number of servers / clusters.
HTTP(s) external global
SSL Proxy external global
TCP Proxy external global
Network TCP/UDP external regional
Internal TCP/UDP regional Internal
HTTP(s) regional
A domain name service run by Google.
GCP’s Content Delivery Network uses Google’s global points of presence to cache HTTP(s) load balanced content for users of your apps.
This forms a direct peering connection between your business’s network and GCP. An important note: Direct peering exists outside of GCP, so unless you need to Google Workspace Apps the recommended method is through Dedicated Interconnect or Partner Interconnect.
Peering is the voluntary connection of two networks.
provides a direct physical connection between your on prem network and GCP.
Provides a direct physical connection between your on prem network and GCP through a supported service provider.
This creates a VPN to connect your on prem systems with GCP. Traffic travels over the internet, but all traffic is encrypted.
There you have it an overview of cloud computing and Google Cloud Platform.