No description
Find a file
sebright 848d4bbe90 adservice: enable structured logging and OpenCensus log correlation (#59)
This commit enables OpenCensus log correlation using
opencensus-contrib-log-correlation-log4j2
(https://github.com/census-instrumentation/opencensus-java/tree/master/contrib/log_correlation/log4j2).
The library inserts the trace ID, span ID, and sampling decision into every
Log4j log event.  This commit includes other changes to make the tracing data
available in the logs, in a format that can be interpreted by the Stackdriver
Logging agent:

- Convert all adservice log statements from java.util.logging to Log4j.

- Specify a JSON format for Log4j output.  This is related to issue #47.

- Add the trace ID, span ID, and sampling decision to the JSON format.  Trace
  ID and span ID use special keys that the Stackdriver Logging agent uses to
  populate the tracing data fields in the LogEntry uploaded to Stackdriver
  (https://cloud.google.com/logging/docs/agent/configuration#special_fields_in_structured_payloads).
  However, Stackdriver won't be able to link the traces and log entries until
  fluent-plugin-google-cloud can automatically format the trace ID with the
  format expected by Stackdriver
  (https://github.com/GoogleCloudPlatform/fluent-plugin-google-cloud/issues/239,
  https://github.com/GoogleCloudPlatform/fluent-plugin-google-cloud/pull/260).

This commit also upgrades OpenCensus to 0.16.1 in order to use
opencensus-contrib-log-correlation-log4j2.
2018-10-01 22:45:09 -07:00
docs Add development principles (#56) 2018-09-30 16:49:08 -07:00
img README: add adservice to services list + diagram 2018-08-22 17:11:53 -07:00
istio-manifests Add more license headers 2018-07-25 21:25:27 -07:00
kubernetes-manifests grpc: implement health check in go services (#27) 2018-09-20 10:31:31 -07:00
pb pb: add "categories" field to Product (#60) 2018-10-01 21:33:25 -07:00
src adservice: enable structured logging and OpenCensus log correlation (#59) 2018-10-01 22:45:09 -07:00
tests/cartservice Add more license headers 2018-07-25 21:25:27 -07:00
.gitignore Initial commit for Ads Service. (#21) 2018-08-22 17:02:46 -07:00
.travis.yml travis: add docker build for images (#38) 2018-09-06 15:35:36 -07:00
CONTRIBUTING.md Add development principles (#56) 2018-09-30 16:49:08 -07:00
LICENSE add LICENSE, CONTRIBUTING.md 2018-07-25 21:17:04 -07:00
README.md README: fix syntax 2018-08-30 11:01:09 -07:00
skaffold.yaml skaffold: add gcb timeout (default is too low) 2018-09-26 12:22:51 -07:00

Hipster Shop: Cloud-Native Microservices Demo Application

This project contains a 10-tier microservices application. The application is a web-based e-commerce app called “Hipster Shop” where users can browse items, add them to the cart, and purchase them.

Google uses this application to demonstrate Kubernetes, GKE, Istio, Stackdriver, gRPC and similar cloud-native technologies nowadays.

Screenshots

Home Page Checkout Screen
Screenshot of store homepage Screenshot of checkout screen

Service Architecture

Hipster Shop is composed of many microservices written in different languages that talk to each other over gRPC.

Architecture of
microservices

Find Protocol Buffers Descriptions at the ./pb directory.

Service Language Description
frontend Go Exposes an HTTP server to serve the website. Does not require signup/login and generates session IDs for all users automatically.
cartservice C# Stores the items in the user's shipping cart in Redis and retrieves it.
productcatalogservice Go Provides the list of products from a JSON file and ability to search products and get individual products.
currencyservice Node.js Converts one money amount to another currency. Uses real values fetched from European Central Bank. It's the highest QPS service.
paymentservice Node.js Charges the given credit card info (hypothetically😇) with the given amount and returns a transaction ID.
shippingservice Go Gives shipping cost estimates based on the shopping cart. Ships items to the given address (hypothetically😇)
emailservice Python Sends users an order confirmation email (hypothetically😇).
checkoutservice Go Retrieves user cart, prepares order and orchestrates the payment, shipping and the email notification.
recommendationservice Python Recommends other products based on what's given in the cart.
adservice Java Provides text ads based on given context words.
loadgenerator Python/Locust Continuously sends requests imitating realistic user shopping flows to the frontend.

Features

  • Kubernetes/GKE: The app is designed to run on Kubernetes (both locally on "Docker for Desktop", as well as on the cloud with GKE).
  • gRPC: Microservices use a high volume of gRPC calls to communicate to each other.
  • Istio: Application works on Istio service mesh.
  • OpenCensus Tracing: Most services are instrumented using OpenCensus trace interceptors for gRPC/HTTP.
  • Stackdriver APM: Many services are instrumented with Profiling, Tracing and Debugging. In addition to these, using Istio enables features like Request/Response Metrics and Context Graph out of the box. When it is running out of Google Cloud, this code path remains inactive.
  • Skaffold: Application is deployed to Kubernetes with a single command using Skaffold.
  • Synthetic Load Generation: The application demo comes with a background job that creates realistic usage patterns on the website using Locust load generator.

Installation

Note: that the first build can take up to 20-30 minutes. Consequent builds will be faster.

Option 1: Running locally with “Docker for Desktop”

💡 Recommended if you're planning to develop the application.

  1. Install tools to run a Kubernetes cluster locally:

    • kubectl (can be installed via gcloud components install kubectl)
    • Docker for Desktop (Mac/Windows): It provides Kubernetes support as noted here.
    • skaffold
  2. Launch “Docker for Desktop”. Go to Preferences and choose “Enable Kubernetes”.

  3. Run kubectl get nodes to verify you're connected to “Kubernetes on Docker”.

  4. Run skaffold run (first time will be slow, it can take ~20-30 minutes). This will build and deploy the application. If you need to rebuild the images automatically as you refactor he code, run skaffold dev command.

  5. Run kubectl get pods to verify the Pods are ready and running. The application frontend should be available at http://localhost:80 on your machine.

Option 2: Running on Google Kubernetes Engine (GKE)

💡 Recommended for demos and making it available publicly.

  1. Install tools specified in the previous section (Docker, kubectl, skaffold)

  2. Create a Google Kubernetes Engine cluster and make sure kubectl is pointing to the cluster.

     gcloud services enable container.googleapis.com
    
     gcloud container clusters create demo --enable-autoupgrade \
         --enable-autoscaling --min-nodes=3 --max-nodes=10 --num-nodes=5
    
     kubectl get nodes
    
  3. Enable Google Container Registry (GCR) on your GCP project and configure the docker CLI to authenticate to GCR:

    gcloud services enable containerregistry.googleapis.com
    
    gcloud auth configure-docker -q
    
  4. Set your project ID on image names:

    • Edit skaffold.yaml, update the imageName: fields that look like gcr.io/[PROJECT_ID] with your own GCP project ID.

    • Similarly, edit all Kubernetes Deployment manifests in the ./kubernetes-manifests directory. Find the image: fields with gcr.io/[...] and change them to your own GCP project ID.

  5. Run skaffold run from the root of this repository. This command:

    • builds the container images
    • pushes them to GCR
    • applies the ./kubernetes-manifests deploying the application to Kubernetes.
  6. Find the IP address of your application, then visit the application on your browser to confirm installation.

    kubectl get service frontend-external
    

(Optional) Deploying on a Istio-installed cluster

Note: you followed GKE deployment steps above, run skaffold delete first to delete what's deployed.

  1. Create a GKE cluster.

  2. Install Istio without mutual TLS option. (Istio mTLS is not yet supported on this demo.)

  3. Install the automatic sidecar injection (annotate the default namespace with the label):

    kubectl label namespace default istio-injection=enabled
    
  4. Apply the manifests in ./istio-manifests directory.

    kubectl apply -f ./istio-manifests
    

    This is required only once.

  5. Deploy the application with skaffold run.

  6. Run kubectl get pods to see pods are in a healthy and ready state.

  7. Find the IP address of your istio gateway Ingress or Service, and visit the application.

    INGRESS_HOST="$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')"
    
    echo "$INGRESS_HOST"
    
    curl -v "http://$INGRESS_HOST"
    

Note to fellow Googlers: Please fill out the form at go/microservices-demo if you are using this application.

This is not an official Google project.