The Reference Architecture presented in the following pages combines the knowledge and experience of many teams to formulate a best-practices design and simplify the process for creating a stable, highly-available environment in which to run an application. One of the main aims of a Reference Architecture is to provide a Reference platform that can be deployed and used as a reference for a final design.
Core Services
Verne (Spring Boot) has a dependency on three core services;
- version storage (MongoDB).
- search (ElasticSearch).
- messaging (RabbitMQ).
Verne – Spring Boot
Verne is a Spring Boot application. Spring Boot is a powerful Java-based application framework that provides the full power of the Spring platform to build enterprise-grade web applications. Verne Spring Boot is delivered as a containerised application – as a Docker image available from the FosterMoore Docker repository.
Verne directly consumes version storage, messaging and search services as well as providing metrics on Verne services.
Version Storage – MongoDB
Verne uses MongoDB for entity persistent storage. MongoDB offers a mature and scalable solution that can scale from the typical master/secondary model to a distributed multi-region sharded model. MongoDB can be backed with commercial support from MongoDB if required.
Messaging – RabbitMQ
Verne leverages RabbitMQ for its internal messaging services, providing the necessary decoupling of components. Its high-availability and robustness of messaging aids application scalability, reliability and performance. It offers a small footprint, high throughput and fits well in a microservices architecture. RabbitMQ is a protocol-based messaging solution designed for cloud computing and modern, widely distributed web applications. It is the de-facto standard for cloud messaging and the leading implementer of Advanced Message Queuing Protocol (AMQP), an open-standards alternative to costly, proprietary commercial messaging technologies.
Search – ElasticSearch
ElasticSearch is the search engine used to power Verne search capabilities. ElasticSearch is a search engine based on Lucene and provides a distributed, multitenant-capable full-text search engine with an HTTP web interface and schema-free JSON documents. ElasticSearch is developed by Elastic and written in Java, and released as open-source under the terms of the Apache License and also has optional support model available.
Supporting Services
Visualisation – Kibana
Kibana provides visualisation capabilities on top of the content indexed on an ElasticSearch cluster. Users can create different types of visualisations – bar, line and scatter plots, pie charts and maps – on top of large volumes of data. The Verne reference platform uses Kibana to present a view into business services usage as well as access for detailed service logging.
Service Mesh – Traefik
Traefik (pronounced traffic) is the leading open-source Cloud Native Edge Router that supports HTTP reverse proxying and load balancing making deploying microservices easy. Traefik integrates with many infrastructure components (Docker, Swarm mode, Kubernetes, Marathon, Consul, Etcd, Rancher, Amazon ECS, …) and configures itself automatically and dynamically.
Monitoring
Metric Storage – Prometheus
Prometheus is an open-source system monitoring and alerting toolkit originally built at SoundCloud. Since its inception in 2012, many companies and organisations have adopted Prometheus, and the project has a very active developer and user community along with being the de facto monitoring standard for containerised services. It is now a standalone open source project and maintained independently of any company. To emphasise this, and to clarify the project’s governance structure, Prometheus joined the Cloud Native Computing Foundation in 2016 as the second hosted the project, after Kubernetes. All components in the Verene – Catalyst Cloud Reference Architecture have metrics that are collected by Prometheus and made available to Grafana. Alerts are also configured against the collected time-series data with the Prometheus Alert Manager forwarding these Alerts to services such as PagerDuty.
Visualisation Grafana
Grafana is an open-source visualisation tool that can be used on top of a variety of different data stores but is most commonly used together with Prometheus, InfluxDB, and also Elasticsearch and Logz.io. Grafana is designed for analyzing and visualizing metrics such as system CPU, memory, disk and, I/O utilisation.
Logging
EFK – ElasticSearch, Fluentd, Kibana Stack
Logging services are intended to gathering logging events(using Fluentd) from the cluster itself as well as logging events from services in the cluster such as Verne. Log events are forwarded to ElasticSearch for storage and administrators can use Kibana for visualisation. This is a commonly used logging pattern call EFK (is the default logging pattern for OpenShift).
The EFK stack is a modified version of the ELK stack and contains:
- Elasticsearch: An object store where all log events are stored.
- Fluentd: Gathers logs from nodes and services and then forwards them to Elasticsearch.
- Kibana: A web UI for Elasticsearch used for searching and visualisation.
