Nearly two years in the past, we laid out our concept of recent metadata for the fashionable knowledge stack, the Knowledge Catalog 3.0.
Since then, it’s been a dream seeing our imaginative and prescient come to life. Energetic metadata has change into a class and central speaking level within the knowledge world. Gartner, Forrester, and G2 alike made main strikes to scrap outdated concepts of knowledge cataloging and introduce new ones round lively metadata and fashionable cataloging.
This transformation was pushed by actual product developments, ones that turned this summary dream into one thing that knowledge groups might truly use.
In 2022, our engineering workforce shipped over 200 new options, together with a model new model of our product. These enhancements proved the facility of lively metadata and even received us ranked as a Chief within the Forrester Wave, Q2 2022!
Earlier than we tackle the following 12 months of latest options, let’s take a couple of minutes to look again on an incredible 12 months. Listed here are 15 of our favourite options from 2022.
TL;DR: 15 updates in Atlan you need to find out about
From siloed instruments to embedded context
We now have all the time been pushed by one core concept — variety is the one actuality. Knowledge groups are one of the numerous groups at the moment, from the instruments they use and the individuals they work with to their very own use circumstances and personas. This variety is each a energy and a weak spot.
Knowledge groups work throughout a various set of instruments — BI instruments like Looker, collaboration instruments like GitHub, communication instruments like Slack, ticketing programs like Jira, and extra. Too usually, these flip into siloed knowledge programs. When you’re in a dashboard, you need to know for those who can belief it then and there, somewhat than switching and looking out throughout different instruments.
In 2022, we constructed variety into the core of our product. By activating metadata, Atlan sends essential context again into the workflows and instruments the place the people of knowledge work day-after-day.
Improve collaboration and seize data with our Slack integration
With our Slack integration, launched in January 2022, we set the stage for in depth collaboration on the earth of metadata. Atlan’s deep integration with Slack goes past simply sending messages — it’s about sharing and capturing vital data throughout your group.
Atlan’s Slack integration consists of:
- Wealthy preview hyperlinks: Share belongings along with your workforce on Slack and see detailed previews.
- Workflow alerts: Get alerts in Slack at any time when an Atlan workflow fails or succeeds.
- Ask a query from Atlan: Ask a query by clicking on the Slack icon for any asset. That Slack dialog may also be documented routinely in Atlan for future reference.
- Doc threads instantly from Slack to Atlan: Simply share Slack conversations to particular belongings in Atlan.
- Search phrases and queries instantly from Slack: Seek for phrases and queries in Atlan with out leaving Slack.
Keep in move with our Atlan Chrome Extension
Energetic metadata makes it attainable for metadata to move effortlessly and rapidly throughout your entire knowledge stack, embedding enriched context and knowledge in each device within the knowledge stack.
Think about you’re a dashboard and need to know who owns it and what a metric means. How would you go about it? Effectively, you’ll in all probability have to determine who the proprietor was. Then you definitely would more than likely take a display screen seize of the dashboard, electronic mail or Slack it to that proprietor, and ask your questions. Not anymore!
Now you may skip all these steps by merely clicking on the Atlan icon in your browser. It is going to open the Atlan Chrome extension proper in your dashboard, supplying you with all of the context you want proper whenever you want it. Atlan makes this attainable within the instruments the place your knowledge groups and knowledge shoppers spend most of their time — Looker, Tableau On-line, Snowsight, Mode, Energy BI, BigQuery, and Salesforce, with extra instruments on the best way.
Remedy points sooner with our Jira integration
Monitoring points and fixing issues simply received so much simpler with our Jira integration. For a lot of knowledge groups, attending to the “root trigger” of damaged dashboards or inconsistent numbers generally is a lengthy, drawn-out course of. Not anymore!
With Atlan’s Jira integration, customers can now rapidly open a brand new Jira ticket or handle an present ticket instantly from any Atlan asset. Simply click on the Atlan sidebar.
Reduce threat throughout knowledge creation with our GitHub integration
When an information engineer makes adjustments to a dbt mannequin, they may unknowingly be affecting tens or tons of of individuals, tables, or dashboards. How can we decrease this threat whereas additionally offering transparency for the info engineer and knowledge shopper?
With our integration between Atlan and GitHub, knowledge engineers can now entry all of the context they should decrease the danger for knowledge shoppers.
Right here’s an instance of what this seems to be like: say that you just’re an information engineer, and also you’ve created a pull request. When the GitHub motion runs, it’s going to routinely create an inventory of all downstream belongings that shall be affected by this request — earlier than you make the change. From there, you may attain out to customers upfront, or analysis and take a look at the belongings to see how they is likely to be affected.
From dispersed, handbook instruments to automated 360° context
Simply as knowledge groups at the moment use numerous instruments, their knowledge ecosystem is made up of numerous knowledge belongings. These are not simply tables made up of rows and columns. Knowledge belongings are exams, metrics, pipelines, code, queries, dashboards, and extra — all scattered throughout totally different knowledge instruments and programs.
At the moment, whenever you onboard an engineer, you simply share a hyperlink to a GitHub repository. This has the code, documentation, historical past, collaborators, and every little thing else it is advisable know concerning the code, multi function place.
Nevertheless, it takes a number of time and handbook effort to arrange and produce documentation into conventional knowledge catalogs at the moment — and simply as quickly as you’re accomplished, they’re old-fashioned. At Atlan, we wished to make knowledge belongings simply as simple to share and perceive as software program on GitHub. Together with activating metadata again into day by day workflows, our workspace provides customers a 360-degree view of each asset by routinely pulling in and stitching collectively metadata from each device in your knowledge stack.
Atlan Playbooks: the Zapier for metadata
One of many widespread questions we get from knowledge groups is, “How can we automate our metadata?” Whereas different groups like advertising or gross sales can simply do action-based automation at scale with instruments like Zapier or Salesforce, knowledge groups usually don’t have the bandwidth to code customized automations for every numerous use case. Why can’t there be a Zapier for metadata?
That’s the reason we’ve got developed the primary low-code/no-code metadata automation for knowledge groups. With Atlan’s Playbooks, customers can now create rule-based automations at scale. These can be utilized to drive countless use circumstances throughout your group. Listed here are just a few examples:
- Deprecate belongings: Mark any belongings that haven’t been queried within the final 30 days as deprecated.
- Add/change possession: For Salesforce belongings which are lacking an proprietor, add RevOps because the proprietor.
- Report failed belongings: Put up an inventory on Slack of each desk and dashboard with a “failed” Airflow standing.
- Defend delicate knowledge: Connect GDPR customized metadata on all belongings tagged as “PII”.
- Flag upstream alerts: Notify downstream house owners when an upstream asset is tagged with a warning or announcement.
Cut back handbook documentation with Trident solutions
Conventional metadata administration generally is a tiresome, handbook course of that requires a number of human intervention from knowledge leaders and stewards.
That’s why we launched Trident, which supplies automated solutions for brand spanking new descriptions, house owners, phrases, and classifications. Trident makes metadata enrichment simpler in order that knowledge groups can spend extra time on significant initiatives and fewer time writing descriptions and classifying knowledge. Since its launch, one-third of all description updates on Atlan have been made with Trident!
P.S. We just lately gave a preview of the potential of Trident by combining it with GPT-3 to create Trident AI — more and more clever automated descriptions, enterprise phrases, READMEs, and extra.
Handle your knowledge and lower prices with asset recognition and utilization metrics
One of many greatest challenges going through knowledge groups at the moment is managing their knowledge and decreasing knowledge infrastructure prices. That’s why we launched utilization and recognition metrics.
Knowledge groups can now analyze their Snowflake knowledge belongings with out leaving Atlan. With these new metrics, organizations can determine their most used belongings, discover individuals with essentially the most context on belongings, determine unused belongings, and extra. These metrics embody:
- Asset question logs: The variety of occasions an information asset has been queried (i.e. asset recognition), when it was final up to date, and so on.
- Consumer question logs: A knowledge asset’s prime customers and most up-to-date customers, who final queried it, and so on.
- Value and efficiency optimization logs: Which queries are slowest, most costly, and so on.
Get full visibility and column-level lineage with our Fivetran Metadata API integration
Atlan’s deep partnership and integration with Fivetran creates the primary end-to-end view of the fashionable knowledge stack. By leveraging Fivetran’s new Metadata API, Atlan now builds column-level lineage out of your Fivetran sources all the best way to BI instruments like Looker and Tableau — supplying you with full visibility throughout your fashionable knowledge stack with highly effective root trigger and influence evaluation.
We’re excited to companion with Atlan to deliver higher metadata administration to our clients. Knowledge governance is a big wrestle for knowledge groups at the moment.
Collectively, Fivetran and Atlan are fixing this downside by serving to knowledge groups achieve much-needed visibility on the place knowledge got here from, who accessed it, and what adjustments have occurred within the pipeline.
Fraser Harris, VP of Product at Fivetran
Make metrics and fashions first-class residents with our dbt Semantic Layer integration
One other huge partnership from this 12 months was Atlan + dbt. With our new Semantic Layer integration, dbt metrics and fashions at the moment are first-class residents in Atlan.
Atlan’s deep integration with the Semantic Layer brings dbt’s wealthy metrics and fashions into the remainder of the info stack, permitting knowledge groups to create repeatable, metadata properties — like desk house owners and verified tags — and construct documentation requirements.
dbt’s metrics are additionally now part of Atlan’s column-level lineage, spanning from supply programs and knowledge storage to transformation and BI. With elevated context, visibility, and self-service, this integration helps numerous knowledge groups work collectively higher.
Get a birds-eye view of your knowledge property with our Reporting Middle
At Atlan, we’ve seen a sample of questions come up as our clients collect extra metadata. Our customers need to perceive how their knowledge panorama and its metadata are evolving.
That’s why we launched the Reporting Middle, which supplies a chicken’s-eye view of all of the metadata updates in Atlan via visualizations that can assist you observe adoption, curation, and metadata administration. Briefly, Reporting Middle provides you knowledge in your metadata. It doesn’t get extra meta than this.
With the Reporting Middle customers can now:
- Monitor their belongings
- Observe glossary phrases and classes
- Implement governance by monitoring personas, functions, classifications and requests
- Perceive queries and question runs from insights in Atlan.
From generic instruments to personalised experiences
One other problem with conventional knowledge catalogs and metadata options is that they deal with each person the identical, delivering a generic expertise throughout the group.
The issue with that method is, context means various things to totally different individuals. For instance, an information engineer seems to be at a desk and so they’re inquisitive about transformations, knowledge high quality, and pipeline stability. In the meantime, an analyst might care about column names and descriptions, frequency distributions, and fashionable queries ran in opposition to that desk. Lastly, a enterprise person may simply see a metric and marvel “What does this metric imply?” and “Can I belief it?”
We wished to vary that narrative and construct a platform that delivers a personalised expertise for every person throughout your group.
Create a Netflix-like knowledge expertise with Personas and Functions
If Netflix can create a personalised viewing expertise for his or her customers, we thought, why can’t Atlan do the identical factor for knowledge groups? That’s precisely what we did with the introduction of personas and functions.
Personas and functions give admins the flexibility to regulate entry to belongings, insurance policies, and preferences tailor-made to the function of every particular person. Personas present asset-level controls for every workforce, whereas functions group belongings collectively for cross-team initiatives.
For instance, the advertising persona might give all advertising individuals entry to sure Snowflake tables and Tableau belongings. With a function, some members of the advertising workforce can also get entry to different advertising marketing campaign knowledge and dashboards which are related to campaigns they’re operating this 12 months.
We’ve seen this having every kind of fascinating implementations inside organizations. For example, many knowledge groups utilizing Atlan have began enthusiastic about implementing paradigms like the info mesh, which at its core has this idea of a website. Utilizing personas and functions, you may create extraordinarily personalised experiences by area, which then permits your knowledge mesh to begin being profitable.
Curate person experiences with customized metadata
Customized metadata is without doubt one of the most beneficial options in Atlan. Together with creating their very own metadata (like an information freshness rating or knowledge high quality metrics), admins can choose which customized metadata needs to be seen to every persona whereas limiting visibility to those who wouldn’t discover this data related.
For instance, it’s seemingly {that a} finance persona gained’t want entry to dbt transformations and Airflow DAGs, however they might must dig into lineage to troubleshoot an odd Salesforce report or question a Snowflake desk to find out ARR.
With customized metadata, Atlan supplies the flexibility to curate every particular person’s distinctive expertise, serving to them work extra effectively with the proper data and making certain they don’t get overwhelmed by your entire knowledge property.
Energy personalised lively metadata with an all-new Atlan person interface
At Atlan, we’ve all the time been dedicated to serving to knowledge groups work collectively higher. We began out as an information workforce ourselves and failed 3 times in making an attempt to implement an information catalog. So we determined to construct Atlan to be the form of platform and companion that we wished we had again then.
After over a 12 months of labor, we launched a model new model of Atlan final 12 months. From a whole redesign of the Atlan person interface to highly effective lively metadata and personalization options, this replace made it simpler than ever for knowledge groups to construct belief and democratize knowledge with metadata.
From closed, siloed instruments to open programs
When you may have a various set of customers, utilizing a various set of instruments and belongings, all throughout industries, knowledge stacks, and extra, there shall be a variety in use circumstances. Each firm’s use case is totally different, and one dimension doesn’t match all. That’s why we care about constructing open and extensible platforms that permit knowledge groups to resolve their very own issues in their very own methods.
Drive countless integrations and use circumstances with a absolutely open API structure
Offering a totally open API structure permits each motion to be programmatically pushed in Atlan. We now open all our metadata change occasions via AWS Eventbridge and a Kafka queue, Lamda webhooks, and extra.
On the core of the Atlan platform, every little thing is open by default — and this permits knowledge groups throughout any group to construct for and remedy all kinds of use circumstances. Listed here are a pair that we’ve got seen organizations begin to implement.
Programmatic knowledge archival and deletion
One among our clients, a media firm, purchases a number of exterior knowledge. The info has an expiration date, so that they arrange handbook processes to often handle and delete their expired knowledge. This typically led to knowledge not being deleted, leading to authorized and compliance dangers. This course of additionally pressured the info workforce to often spend vital time on handbook work, leading to misplaced productiveness and workforce output.
Now, utilizing the Atlan API, a notification is triggered 30 days earlier than the contractual finish date for the info. This creates a Jira ticket, letting the suitable events know that the info can then be deleted or archived. Via this new programmatic metadata workflow, the corporate was not solely capable of get rid of authorized threat but additionally save vital time and productiveness.
Energetic knowledge belief administration
One other buyer has built-in Atlan with an information high quality device and energy lively knowledge belief administration. They use our open APIs to usher in anomalies which are detected by the info high quality device, after which downstream BI belongings and doubtlessly impacted enterprise metrics are routinely flagged. For instance, if a desk feeding a KPI has an information high quality problem, Atlan routinely creates an Announcement for the KPI within the Atlan enterprise glossary.
Construct customized event-based automations with our AWS EventBridge integration
Many groups throughout organizations, from knowledge observability to incident administration, are constructing automations that may set off actions primarily based on occasions. However there’s by no means been something out there for metadata administration to create a majority of these event-driven use circumstances for metadata… till now.
Knowledge groups can now use Atlan to create production-grade, event-driven automations for the world of metadata. This leverages our integration between Atlan and AWS EventBridge, an AWS service that creates an occasion and lets customers eat and construct use circumstances on it, to create Atlan metadata occasions into an EventBridge account.
Listed here are some examples of how this may work in Atlan:
- Possession alerts: Get notifications in Slack when there’s a change within the possession of an asset.
- Propagation and classification: If somebody marks a area as PII in an upstream knowledge supply, routinely create a masking coverage to vary all associated fields downstream.
- And numerous extra use circumstances, resembling notifications round schema adjustments on belongings in Atlan for knowledge engineering, login/logout occasions for safety groups, or triggering a Fivetran enrichment occasion to kick off an Atlan workflow.
—
💡 To be taught extra about the entire releases for 2022, try shipped.atlan.com. It’s also possible to subscribe to our Product Updates publication for all the newest information.
💡 To take a look at Atlan for your self, go to our product tour.
💡 Prepared to begin utilizing these new options? Attain out to our Gross sales workforce or your Buyer Success Supervisor to learn the way.
