SLIs are an indicator of the service being provided to a specific component. SLIs are made up of two numbers, the number of good events divided by the total number of actual events. The report generation feature is a request-driven API, as opposed to a pipeline or storage component. As such, if we wanted to create an availability SLI for the request-driven service, the proportion of requests that results in a successful response would be correct.

Reference:

https://sre.google/workbook/implementing-slos/#slis-for-different-types-of-services - Useful Table for creating potential SLIs for different types of components

"As the I/O wait times aggregated across all report generation backends" - The I/O wait times across all report generation backend will not give us any insight into the availability of the service.

"As the application's report generation queue size compared to a known-good threshold" - SLI's are made up of comparison's between numbers of events and not a comparison to a 'known-good' threshold.

"As the reporting backend, PD throughput capacity compared to a known-good threshold" - As the reporting backend PD throughput capacity compared to a known-good threshold.

Assigning the correct permissions to the service account used by Cloud Build (the Cloud Build service account) will allow the pipeline to deploy new images to GKE whilst minimising development effort.

References:

'Configuring user-specified service accounts' - https://cloud.google.com/build/docs/securing-builds/configure-user-specified-service-accounts

"Specify the Container Developer role for Cloud Build in the cloudbuild.yaml file." - In the cloudbuild.yaml file, we only specify the serviceAccount and not the ROLE used by the pipeline. For Cloud Build to be able to use the permissions that the role grants, the role must be attached to a service account.

"Create a new service account with the Container Developer role and use it to run Cloud Build." - This would work; however, this does not meet the requirements which need us to minimise development time. It does not include the development time required to grant additional permissions that the service account needs

A Service Level Objective (SLO) is a target value set for a Service Level Indicator (SLI) to ensure that the user experience does not degrade too much. An SLI is a metric used to measure the performance of a service. In this case, the best SLI to use would be a quality SLI, which is the ratio of non-degraded responses to total responses. This SLI is the best option because it measures the quality of the user experience by looking at the ratio of non-degraded responses to total responses.

"An availability SLI" - the ratio of healthy microservices to the total number of microservices - is not the best option because it does not measure the quality of the user experience. It only measures the availability of the microservices, which does not necessarily correlate to the quality of the user experience.

"A freshness SLI" - the proportion of widgets that have been updated within the last 10 minutes - is not the best option because it does not measure the quality of the user experience. It only measures the freshness of the content, which does not necessarily correlate to the quality of the user experience.

"A latency SLI" - the ratio of microservice calls that complete in under 100 ms to the total number of microservice calls - is not the best option because it does not measure the quality of the user experience. It only measures the latency of the microservice calls, which does not necessarily correlate to the quality of the user experience.

The correct answer is "Create a Development and a Production GKE cluster in separate projects. In each cluster, create a Kubernetes namespace per team, and then configure Kubernetes Role-based access control (RBAC) so that each team can only access its own namespace"

Configuring Kubernetes Role-based access control (RBAC) is the most cost-effective and secure solution, as it allows for enterprise multi-tenancy using GKE, which reduces costs and provides control access by RBAC. This ensures that different teams cannot access other teams' environments.

References: Kubernetes RBAC Authorization - https://kubernetes.io/docs/reference/access-authn-authz/rbac/

"Create one GCP Project per team. In each project, create a cluster for Development and one for Production. Grant the teams IAM access to their respective clusters." - Creating a GCP Project per team would not be cost-effective to administer and run.

"Create one GCP Project per team. In each project, create a cluster with a Kubernetes namespace for Development and one for Production. Grant the teams IAM access to their respective clusters." - We need to create separate development and production environments and this would mean both 'development' and 'production' would run in the same Kubernetes cluster.

"Create a Development and a Production GKE cluster in separate projects. In each cluster, create a Kubernetes namespace per team, and then configure Identity Aware Proxy so that each team can only access its own namespace." - Configuring Identity Aware Proxy would not provide the necessary level of access control.

To maximise bandwidth for transferring the images to the cluster, we need the registry closer to the system where they need to be deployed. This means we must choose a registry in the eu-west-1 region. The question also tells us that the registry must be scalable, a private image repository hosted on a cloud instance would not be scalable so we will choose the eu.gcr.io registry as it is the closest to our production systems.

Reference:

Google bandwidth limits ranking - "https://cloud.google.com/compute/docs/network-bandwidth#vm-out-dest-i"

"Push the images to Google Container Registry (GCR) using the gcr.io hostname." - The gcr.io hostname hosts images in the US.

"Push the images to Google Container Registry (GCR) using the us.gcr.io hostname." - The us.gcr.io hostname hosts images in the US

"Push the images to a private image registry running on a Compute Engine instance in the eu-west-1 region." - The question asks that the registry be scalable. Hosting the private image registry on a Compute Engine instance will not be able scalable as the Google managed service.

A postmortem is a written record of an incident, its impact, the actions taken to mitigate or resolve it, the root cause(s), and the follow-up actions to prevent the incident from recurring'. These are components that make up a postmortem.

Reference:

Second Paragraph - https://sre.google/sre-book/postmortem-culture/

"A list of employees responsible for causing the incident" - A key theme in the Cloud DevOps Engineer exams when it comes to postmortems is to keep them blameless. The focus should be on identifying the contributing causes of the incident without indicting a team or individual for bad behaviour.

"Your opinion of the incident's severity compared to past incidents" - Given that you want to prevent similar incidents in the future, adding your opinion about the previous incident's severity does not contribute to that.

"Copies of the design documents for all the services impacted by the incident" - The design documents are relevant to preventing similar incidents in the future.

In order to measure the new Service Level Indicator (SLI) to increase coverage in case of a potential load balancer misconfiguration, CDN failure, or other global networking catastrophes, the two best options are "Instrumentation coded directly in the client" and "A synthetic client that periodically sends simulated user requests."

"Instrumentation coded directly in the client" - This option is correct because client instrumentation allows for quickly gaining insight into customer's happiness, which is necessary to detect CDN/LB failures. If you use an APM suite or frontend framework that provides client instrumentation, you can quickly gain insight into your customer's experience.

"A synthetic client that periodically sends simulated user requests" - This option is also correct because a synthetic client can help detect potential load balancer misconfigurations, CDN failures, or other global networking catastrophes. Additionally, a synthetic client can send simulated user requests to test the service regularly to ensure it is properly functioning and provides an accurate representation of the customers' experience.

"Your application servers' logs" - This option is incorrect because running health checks at application/server level will not cover the potential load balancer misconfigurations, CDN failures, or other global networking catastrophes. Additionally, logs cannot detect or inform us of a customer's experience, which is necessary to detect CDN/LB failures as those components are closest to the customer.

"Metrics exported from the application servers" - This option is incorrect because metrics exported from the application servers cannot detect the application from the customer's perspective, which is necessary to detect CDN/LB failures. Additionally, metrics exported from the application servers will not be able to detect potential load balancer misconfigurations, CDN failures, or other global networking catastrophes.

"GKE health checks for your application servers" - This option is incorrect because GKE health checks for your application servers will not be able to detect potential load balancer misconfigurations, CDN failures, or other global networking catastrophes. Additionally, GKE health checks for your application servers only check that the pod the application runs is in 'running' state.

Google colloquially calls a peak traffic event a 'Black Friday'. Preparing for a Black Friday is made up of 3 stages, the planning stage, the preparation stage and the execution stage that occur in that order. The planning stage is for creating projections and planning the technical architecture to account for the Black Fridays. The preparation stage is the testing stage, and the execution stage is experiencing and serving traffic on Black Friday.

Before implementing any auto-scaling measures, we must test the application to see where the failure occurs to make the system more resilient in the future.

References:

https://cloud.google.com/architecture/black-friday-production-readiness#set_up_load_and_performance_testing

"Enable AutoScaling on the production clusters, in case there is growth." - You must test the infrastructure before enabling auto-scaling. Enabling the auto-scaling without making educated, data-driven decisions could easily lead to excessive costs as it may not be the correct strategy to implement. Before doing this, testing must occur, it may be the case that your infrastructure is able to handle the increased load and enabling auto-scaling is not entirely necessary.

"Pre-provision double the compute power used last season, expecting growth." - Making this decision before careful planning and testing is useless. Your infrastructure may need 20x the size or be fine at the current size. Doubling without deliberation and capacity planning is not the most effective or efficient action.

"Create a runbook on inflating the disaster recovery (DR) environment if there is growth." - Creating the runbook in case of an outage should be much later in the list of activities than doing load testing, this should be a last resort action. 'Hope is not a strategy'

The best approach to efficiently provide updates to everyone affected by the outage is to provide periodic updates to all stakeholders promptly and commit to a next update time in all communications. This approach ensures that all stakeholders, both internal and external, are kept up to date on the progress of the incident and that they are aware of when the next update will be provided. This approach also allows the Incident Response Team to focus on providing responses directly to customers, which is the most important stakeholder group in this situation.

"Focus on responding to internal stakeholders at least every 30 minutes" - This approach does not provide enough information to all stakeholders and does not provide a clear timeline for when the next update will be provided.

"Delegate the responding to internal stakeholder emails to another member of the Incident Response Team" - This approach does not provide enough information to all stakeholders and does not provide a clear timeline for when the next update will be provided.

"Provide all internal stakeholder emails to the Incident Commander, and allow them to manage internal communications" - This approach does not provide enough information to all stakeholders and does not provide a clear timeline for when the next update will be provided.

Note that this question is referring to owners and collaborators of action items. Assigning actions to individual owners is not deeming them responsible for the outage but giving them accountability (leading to actions). Ideally, an owner of an action item is a single point of contact who is responsible for the postmortem, follow-up and completion.

Reference: https://sre.google/workbook/postmortem-culture/ - 'Missing Ownership'

"Assign multiple owners for each item to guarantee that the team addresses items quickly." - As per Google's recommendations, having a single owner results in accountability, which leads to action. Having multiple owners dilutes the accountability, more often leading to inaction.

"Assign collaborators but no individual owners to the items to keep the postmortem blameless." - Whilst it is correct that post-mortems should be kept blameless, assigning owners to action items to rectify what is wrong is not blaming any individual engineers for the outage.

"Assign the team lead as the owner for all action items because they are in charge of the SRE team." - The SRE team lead will not resolve all problems personally, so this makes the actual ownership of the action items unclear.