Python, Java

Explanation: Cloud Endpoints for the App Engine standard generation 1 environment historically used Endpoints Frameworks, which only supports the Java 8 and Python 2.7 runtime environments.(Time of Writing)

https://cloud.google.com/endpoints/docs/choose-endpoints-option

Major incidents should be declared when there is a noticeable outage that is affecting customer or stakeholder access. Revenue loss should be expected as part of this incident

Use your own authentication mechanism and manage your own credentials, Google Apps Directory Sync integrates with LDAP

Explanation: GCDS can be used and can bring your own creds to gcp https://cloud.google.com/iam/

Availability for Both

Explanation:Both Database and the Web application need to be available. Production apps should have full availability. https://landing.google.com/sre/sre-book/chapters/service-level-objectives/

Bigtable is ideal for storing very large amounts of data in a key-value store and supports high read and write throughput at low latency for fast access to large amounts of data. https://cloud.google.com/bigtable

Rotating Secrets is a must and Cache Secrets locally many times a day.

Explanation: These are common secret management best practices

Rotating secrets (Use multiple versions)

Cache secrets locally (Several times per hour)

Using a separate solution or platform

Your first instinct when approaching an incident like this may be to try and find the root cause. Still, the best thing to do in that scenario is to triage the issue and to make the system work as well as it can under the given circumstances before beginning to identify the root cause.

References: https://sre.google/sre-book/effective-troubleshooting/ - Triage

"Use Stackdriver Monitoring to check for a spike in CPU or memory usage for the affected region." - The priority is to ensure customers are still being served traffic. The issue can be debugged after the system is recovered for the users.

"Add an extra node pool that consists of high memory and high CPU machine type instances to the cluster." - Whilst this answer is along the right lines such that the customer-facing impact should be minimised, the report specifies that 'none of the user's in a specific region can connect to the application', adding an additional performance node pool will not reduce the impact to the customers as they cannot reach the cluster.

"Use Stackdriver Logging to filter on the clusters in the affected region, and inspect error messages in the logs." - The priority is to ensure customers are still being served traffic. The issue can be debugged after the system is recovered for the users.

PII stands for Personally Identifiable Information, any information that can be used to identify an individual. In this question, the PII leaks into certain log entry fields, and all PII entries begin with the text userinfo. The goal is to capture these log entries in a secure location for later review and prevent them from leaking to Cloud Logging.

The correct answer is: Use a Fluentd filter plugin with the Cloud Monitoring Agent to remove log entries containing userinfo, and then copy the entries to a Cloud Storage bucket. Fluentd is a log management tool that can be used to filter logs before they are sent to Cloud Logging. By using a Fluentd filter plugin with the Cloud Logging Agent, the log entries containing userinfo can be removed and copied to a Cloud Storage bucket. This will ensure that the PII is securely stored in a separate location and will not leak to Cloud Logging.

"Create a basic log filter matching userinfo, and then configure a log export in the Cloud Logging console with Cloud Storage as a sink." - This option would create a basic log filter matching userinfo, but it would not prevent the PII from leaking to Cloud Logging.

"Create an advanced log filter matching userinfo, configure a log export in the Cloud Logging console with Cloud Storage as a sink, and then configure a log exclusion with userinfo as a filter." - This option would create an advanced log filter matching userinfo, but it would not prevent the PII from leaking to Cloud Logging. Additionally, the log exclusion with userinfo as a filter would not be necessary since the log entries containing userinfo can be removed with the Fluentd filter to avoid them reaching Cloud Monitoring at all.

"Use a Fluentd filter plugin with the Cloud Logging Agent to remove log entries containing userinfo, create an advanced log filter matching userinfo, and then configure a log export in the Cloud Logging console with Cloud Storage as a sink." - This option would be redundant because the log entries containing userinfo are already being removed with the Fluentd filter. Additionally, the log export with Cloud Storage as a sink would not be necessary since the log entries are being stored in a separate Cloud Storage bucket.

To limit the containers deployed to your cluster, you would use 'Binary Authorization'. Binary Authorisation allows users to enforce that containers deployed to a Kubernetes cluster are cryptographically signed by a trusted authority and verified by a Binary Authorization attestor.

References:

'Binary Authorization Overview' - https://cloud.google.com/binary-authorization/docs/cloud-build

"Create a custom builder for Cloud Build that will only push images to gcr.io/altostrat-images." - This will not limit the images to only be used from Altostrat

"Add logic to the deployment pipeline to check that all manifests contain only images from gcr.io/altostrat-images." - This solution may work. Still, it does not fulfil the requirement that the solution must minimise development time, 'Binary Authrorization' will fulfil the requirements with less development effort.

"Add a tag to each image in gcr.io/altostrat-images and check that this tag is present when the image is deployed." - This solution may work but does not fulfil the requirement that the solution must minimise development time.

The easiest and most cost-effective way to increase the image build time is by caching layers of the Docker image, the intermediate artefacts.

References: 'Best Practices for Speeding up Builds' - https://cloud.google.com/build/docs/optimize-builds/speeding-up-builds#using_a_cached_docker_image

"Run multiple Jenkins agents to parallelise the build." - We are using Cloud Build and not Jenkins.

"Use multiple smaller build steps to minimise execution time." - Whilst this would reduce the build time, this may take a lot of development effort to implement a compressing of the build steps. Furthermore, using multiple smaller steps will not reduce the overall build time if the actions taken in each step remain the same.

"Use larger Cloud Build virtual machines (VMs) by using the machine-type option." - This would speed up the build time but does not minimise cost.