Collibra Edge

Challenge

Setting up Collibra Edge and Designing an Onboarding Process

Setting up Collibra Edge and creating an efficient onboarding process for new systems posed a significant challenge for one of our clients in the retail sector. The client's existing environment relied on Jobserver, requiring manual upgrades and connections per schema to ingest the structural metadata. However, with the need to classify data for regulatory purposes like GDPR, migrating to Collibra Edge became imperative. Edge was the replacement for Jobserver, relying on modern technologies such as Kubernetes to function and ensure a more secure setup for ingesting metadata. Doing so, we could set a solid foundation for effective data governance, unlocking the true potential of data management, paving the way for better decision-making and regulatory compliance.

Approach

Navigating the Unknown and Overcoming Migration Hurdles

We embarked on the journey of setting up Collibra Edge. The task involved configuring an on-premise Edge server, connecting it to both on-premise and SaaS data sources. With determination, we tackled networking complexities and successfully established a healthy Edge Site.

During the migration process of already existing metadata, we faced challenges due to the lack of a streamlined workflow for migrating Jobserver ingested schemas to Collibra Edge. However, through meticulous analysis of existing connections and collaboration with stakeholders, we devised an alternative approach.

There were 4 steps in this alternative approach:

1. Extracting all the existing information from a Jobserver connection.
2. Analysing this information for any links not related to automated catalog ingestions.
3. Where applicable, we needed to update the full name of those existing assets to comply with Edge asset naming convention.
4. Linking the schema with its linked tables and columns to the new Edge database asset.

By following the above steps Edge would accept the existing assets as if they were ingested through the normal flow.

Of course, we also had a lot of new system that needed to be onboarded. About 60+ systems were onboarded by prioritizing the systems based on importance and difficulty. Doing so we could efficiently divide the onboarding process into manageable waves. To facilitate communication with system owners, we provided them with a comprehensive step-by-step guide containing all the necessary information. This approach significantly reduced troubleshooting time and ensured smooth onboarding of multiple systems.

Impact

Streamlining Onboarding and Unlocking Data Governance Potential

Our efforts led to streamlined onboarding of source systems onto the Collibra catalog, resulting in having the metadata to field level in our catalog. Meaning, we could now start with classifying data source fields to indicate their sensitivity level. For example, Customer E-mail would be classified as Confidential, Company Address as Public. This classification was crucial for other teams that required access to the data and for ensuring compliance with regulations. Additionally, we could link business information to physical data, enabling a holistic view of data governance and strengthening the overall framework.

In conclusion, our successful implementation of Collibra Edge and the streamlined onboarding process had a profound impact. Together with the client we overcame challenges by navigating the unknown, finding alternative migration solutions, and ensuring efficient communication. As a result, we established a solid foundation for effective data governance, enabling the client to classify and connect data seamlessly.