User:MoyaKB/sandbox

= ODPI Egeria = ODPi Egeria is an open source project dedicated to making metadata open and automatically exchanged between tools and data platforms, no matter which vendor they come from.

ODPi Egeria is designed to support the following principles:


 * The maintenance of metadata must be automated to scale to the sheer volumes and variety of data involved in modern business.  Similarly the use of metadata should be used to drive the governance of data and create a business friendly logical interface to the data landscape.
 * The availability of metadata management must become ubiquitous in cloud platforms and large data platforms, such as Apache Hadoop so that the processing engines on these platforms can rely on its availability and build capability around it.
 * Metadata access must become open and remotely accessible so that tools from different vendors can work with metadata located on different platforms. This implies unique identifiers for metadata elements, some level of standardization in the types and formats for metadata and standard interfaces for manipulating metadata.
 * Wherever possible, discovery and maintenance of metadata has to an integral part of all tools that access, change and move information.

Egeria provides an Apache 2.0 licensed platform to support vendors that sign up to the open metadata and governance manifesto.

The open metadata and governance platform provides type system, frameworks, APIs, event payloads and interchange protocols to enable tools, engines and platforms to exchange metadata in order to get the best value from data whilst ensuring it is properly governed.

History
The Linux Foundation announced ODPi Egeria for Open Sharing, Exchange and Governance of Metadata on 27th August 2018.

IBM and ING, vendors and end users collaborated on the first Egeria release, which was initially incubated as part of the Apache Atlas project (an open source metadata repository designed for the Apache Hadoop ecosystem). IBM and ING jump-started Egeria with a significant code donation

Project Objectives
The objectives of the new open metadata and governance function for Egeria: Automation, Business Value and Connectivity. Automation Open metadata and governance provides an API for components that capture metadata from data platforms as data sources are created and changed. This metadata is stored in the metadata repository and results in notifications to alert governance and discovery services about the new/changed data source. It provides frameworks and servers to host bespoke components that automate the capture of detailed metadata and the actions necessary to govern data and its related assets. This includes:

A discovery server that manages discovery services for analyzing and capturing metadata about new data sources.

A stewardship server that manages stewardship services for resolving issues detected during the governance and use of data.

Business Value Open metadata and governance provides specialized access services and user interfaces for key data roles such as CDO, Data Scientist, Developer, DevOps Operator, Asset Owner, and for Applications. This enables metadata to directly support the work of people in the organization. The access services can also be used by tools from different vendors to deliver business value with open metadata.

Connectivity Finally, connectivity enables a peer-to-peer Metadata Highway offering open metadata exchange, linking and federation between heterogeneous metadata repositories.

Components
The open metadata and governance project is divided into the following pieces:


 * Common types for open metadata - these types are built from the Egeria type system and define the types stored in the graph database as well as payloads for notifications and APIs.
 * Open Metadata Repository Services (OMRS) - Open metadata repository APIs and notifications to enable metadata repositories to exchange metadata in a peer-to-peer metadata repository cohort. This capability is located in each metadata repository and collectively they enable what is referred to as the “metadata highway”.
 * Open Metadata Access Services (OMAS) - Consumer-centric APIs and notifications for specific classes of tools and applications. The OMAS services call the OMRS to access metadata from any open metadata repository.
 * New frameworks:
 * Open Connector Framework (OCF) - provides factories for connectors with access APIs for data resources and metadata together. The OMRS is also built as a set of metadata repository connectors and the OMAS services use the OCF to connect to the appropriate OMRS connector.
 * Open Discovery Framework (ODF) - provides management for automated processes and analytics to analyze the content of data resources and update the metadata about them.
 * Governance Action Framework (GAF) - provides audit logging and governance enforcement services for implementing enforcement points in data engines, security managers such as Apache Ranger, and APIs. It also adds stewardship services for analyzing audit logs and resolving issues identified in exceptions raised by the enforcement services.
 * Open Metadata Graph Repository - A set of stores linked together with a graph database. These stores provide linkage between business, technical and operational metadata along with logs for auditing, operational lineage, metering and exception management.
 * Open Lineage Services - Services for collecting and querying lineage information across multiple heterogeneous metadata repositories.

At this current time, there is a huge investment into Egeria to add the open metadata and governance features plus also work on adoption of this technology across the data industry.