The AI Times is a weekly newsletter covering the biggest AI, machine learning, big data, and automation news from around the globe.

A|I: The AI Times – Google & Nvidia are betting on this Canadian startup to boost ML model performance

Generative AI (GenAI) has sparked tremendous interest in the business world. According to Gartner, more than 80 percent of enterprises will have used

Domino Fall 2023 Release Expands Platform to Fast-Track and Future-Proof All AI

Machine learning operations startup Domino Data Lab Inc. today announced a major update to its artificial intelligence development platform that it says will make it easier for companies to build generative AI models and other applications.

Domino Data bids to fast-track generative AI model development with project templates

SAN FRANCISCO, CALIFORNIA, UNITED STATES, October 31, 2023 /EINPresswire / -- In a strategic move that promises to redefine the landscape of B2B data solutions, FlashIntel is proud to announce its official partnership with Databricks , marking its esteemed ...

Flashintel Joins Databricks Marketplace: A Partnership Poised To Revolutionize B2B Data Solutions

PORTLAND, PORTLAND, OR, UNITED STATE, October 31, 2023 /EINPresswire / -- Allied Market Research published a new report, titled, " The USD 17 Billion Distributed Cloud Market Reach by 2031 | Top Players such as - Databricks, Axway and VMware." The report ...

USD 17 Billion Distributed Cloud Market Reach By 2031 | Top Players Such As - Databricks, Axway And Vmware

Crisp Acquires Atlas Technology Group from Advantage Solutions to Empower CPG Brands with Better Data

Most software leaders will manage generative AI efforts by 2025, Gartner predicts. This trend has implications for their roles that go well beyond application development.

Overseeing generative AI: New software leadership roles emerge

For Q3 2023, ServiceNow reported $2.28 billion in revenue, a 27% increase from a year-ago quarter, beating the Street consensus of $2.274 billion. The company’s earnings per share (EPS) increased to $2.92, 49% up from $1.96 a year ago. ServiceNow’s subscription revenue increased 24.5% year-over-year to $2.216 in the latest Q3 earnings report.

These 3 AI Stocks Offer Great Value But Are Under the Radar for Now

Data lakehouse platforms rely on rapid, high-quality data ingestion for their value, and Arcion's acquisition would solve these issues

Why Databricks Acquired Arcion

Redgate's portfolio of solutions helps organizations reliably solve the complex challenges of database management across the DevOps lifecycle on any database, any platform, anywhere. We make life easier for IT leaders, development and operations teams, increasing efficiency, reducing errors and protecting business-critical data.

Redgate to livestream key sessions from the upcoming PASS Data Community Summit free to the worldwide data community

Get the most recent info and news about Databricks on HackerNoon, where 10k+ technologists publish stories for 4M+ monthly readers.

Databricks is the lakehouse company. More than 7,000 organizations worldwide — including Comcast, Condé Nast, H&M, and over 40% of the Fortune 500 — rely on the Databricks Lakehouse Platform to unify their data, analytics and AI. Databricks is headquartered in San Francisco, with offices around the globe. Founded by the original creators of Apache Spark™, Delta Lake and MLflow, Databricks is on a mission to help data teams solve the world’s toughest problems. --- Attention: Databricks applicants Due to reports of phishing, we’re requesting that all Databricks applicants apply through our official Careers page at databricks.com/company/careers (good news — you are here). All official communication from Databricks will come from email addresses ending with @databricks.com or @goodtime.io (our meeting tool).

Databricks

Let's Build an MLOps Pipeline With Databricks and Spark - Part 2

At HackerNoon, we value stories about building groundbreaking tech. These questions aren’t just about AI mechanics—they’re about the innovation, challenges, and creativity of bringing cutting-edge tools to life, a perfect fit for our community of tech leaders, builders, and future-minded readers. Introduction My name is Aniruth. I work on the storage team at Databricks, where we work on enabling customers to save large amounts of data in an open, scalable format with our Data Intelligence Platform. Specifically, I work on our interoperability efforts with Delta Lake and Apache Iceberg. HackerNoon: What’s the main problem your AI product is designed to solve, and what is effective about your approach to solving it? Aniruth: Databricks unifies data and AI to give customers actionable intelligence—what we call data intelligence. This includes ingesting large amounts of data, ETL, large-scale storage, business intelligence queries, and AI workloads. The techniques used in machine learning in the past decade have been around since the 1980s; the rise of big data made it feasible to run algorithms at scale. Techniques like few shot prompting or RAG rely on high-quality data. Models that have better data win against those with better architectures. Databricks has put significant investment into leading efforts in the data space, pioneering the lakehouse architecture with open data formats and open governance, where customers are able to get the best insights with the best performance from data lakes. What criteria did you use to select the specific AI models for this product, and how does your company make decisions like this? We use AI models in a number of ways in the product - such as Llama 3 for the AI Assistant. We believe in an open data and AI ecosystem and encourage our customers to use any model of their choice. We help make sure customers have end-to-end governance across the AI lifecycle regardless of the models they use, so they can focus on making their models purpose-built for their use cases. How do you ensure your product delivers accurate and unbiased results to users? We have spent significant effort and investment in prioritizing accuracy and unbiased answers for AI usage within our products, and continue to conduct frequent testing. What is your day in the life like as a product manager? The data and AI space is rapidly evolving, so it’s very important to keep up to date. My day can include talking to customers, market analysis, putting together a product requirements document, or preparing marketing materials. My favorite part is getting to make diagrams illustrating how things will work, as it’s pretty fun to transform an idea into a visual. What’s the next big breakthrough in AI that everyone should be watching for? There’s a lot of big breakthroughs coming soon. One that I’m particularly interested in is the hyperpersonalization of content. For the past decade, ads have been fine-tuned to the specific watcher. Some elements of content have been tuned, such as what thumbnail Netflix shows a user, but the actual content (the video itself) has not been. It’ll be interesting to see how directors/producers balance telling the story they want to and the user’s interests. What’s been the biggest challenge you’ve faced in bringing this AI product to market, from concept to launch? I work on large-scale data storage, which can be very confusing to understand. We have various AI optimizations on data, but there’s often questions about when these optimizations are run, how they work, what they don’t cover, etc. With these kinds of questions, it is important to ensure we have clear, consistent messaging about what we are building and why. I’ve found that explaining the cause of limitations resonates very well with customers. How do you envision AI evolving to better understand and respond to human emotions, and what challenges or opportunities do you see in that area? Multimodal models are going to get significantly better in the coming years, which will change our primary mode of interaction with AI. Figuring out human emotion is significantly easier from visual or audio information compared to text. I think there's an opportunity to create more natural interactions in a wider array of scenarios. What performance metrics or KPIs do you track to gauge the success of this AI product? We typically want to see good feedback and usage. I talk to customers pretty often to get a sense of how and why they think about our products, which is key to explain why we see certain trends in metrics. How do you approach designing a product experience that’s not only functional but also engaging and memorable for users? Large-scale data products are notoriously challenging to use. Simple examples are easy to set up, but production workloads typically involve confusing configuration and code. It’s been a high priority for me to build out functionality that customers need, while making the product very simple to use. The future is where any business gets insights from their data easily. In the current world, data-driven business insights are typically restricted to the biggest companies - but even they would prefer a simpler experience. In the long-term, how do you think AI will help the individual reach one’s full potential? For the individual, I’m very excited about AI integrations into hardware. Up to now, we’ve largely seen AI in software applications like websites. There’s many larger applications of building devices that utilize AI, and we’re already starting to see some of those implications in cars and phones. Where do you see the product evolving in the next few years, and what features are you most excited to add? Databricks is on a path towards becoming increasingly simple and more powerful at the same time. There are a lot of efforts we are working on across the board, from making big-scale data and compute easy to work with to improving performance on queries and workflows. Personally, I think we have some exciting features coming soon throughout the product that make workflows way easier with AI. Examples include AI-generated comments on data, AI code suggestions in the notebook editors, and AI interfaces to chat with data (for example, Databricks AI/BI Genie). What’s your perspective on AI’s impact on jobs, and how do you address those concerns in your product strategy? There are concerns over whether AI would reduce the number of jobs. Our products are designed to increase valuable insights, which often come in conjunction with users. For example, with AI/BI Genie, users can create interfaces on their own data. This is a magical experience, where users can ask questions and get answers specific to them. In fact, users can even check the SQL being used to confirm it’s what they’re looking for. This is collaborative with analysts, reducing the time it takes them to go from idea to insight. What user feedback surprised you and led to changes in your roadmap, strategy or product experience? One major surprise for me was the complexity that some of the larger companies have. This introduces requirements into the product that I wouldn’t have considered on my own. A common example is thinking of migration strategies when introducing a new product. Typically, large companies will have either put together existing technologies (commonly open-source software) or have built custom software to solve the problem that our new product offering tackles. It usually takes a bit of time to understand why and how these are put together to ensure we have a solution that covers all possibilities. Would you like to take a stab at answering some of these questions? The link for the template is HERE. Interested in reading the content from all of our writing prompts? Click HERE. At HackerNoon, we value stories about building groundbreaking tech. These questions aren’t just about AI mechanics—they’re about the innovation, challenges, and creativity of bringing cutting-edge tools to life, a perfect fit for our community of tech leaders, builders, and future-minded readers. At HackerNoon, we value stories about building groundbreaking tech. These questions aren’t just about AI mechanics—they’re about the innovation, challenges, and creativity of bringing cutting-edge tools to life, a perfect fit for our community of tech leaders, builders, and future-minded readers. Introduction My name is Aniruth . I work on the storage team at Databricks , where we work on enabling customers to save large amounts of data in an open, scalable format with our Data Intelligence Platform. Specifically, I work on our interoperability efforts with Delta Lake and Apache Iceberg. Aniruth Databricks HackerNoon: What’s the main problem your AI product is designed to solve, and what is effective about your approach to solving it? HackerNoon: Aniruth : Databricks unifies data and AI to give customers actionable intelligence—what we call data intelligence. This includes ingesting large amounts of data, ETL, large-scale storage, business intelligence queries, and AI workloads. The techniques used in machine learning in the past decade have been around since the 1980s; the rise of big data made it feasible to run algorithms at scale. Aniruth Aniruth Techniques like few shot prompting or RAG rely on high-quality data. Models that have better data win against those with better architectures. Databricks has put significant investment into leading efforts in the data space, pioneering the lakehouse architecture with open data formats and open governance, where customers are able to get the best insights with the best performance from data lakes. What criteria did you use to select the specific AI models for this product, and how does your company make decisions like this? We use AI models in a number of ways in the product - such as Llama 3 for the AI Assistant. We believe in an open data and AI ecosystem and encourage our customers to use any model of their choice. We help make sure customers have end-to-end governance across the AI lifecycle regardless of the models they use, so they can focus on making their models purpose-built for their use cases. How do you ensure your product delivers accurate and unbiased results to users? We have spent significant effort and investment in prioritizing accuracy and unbiased answers for AI usage within our products, and continue to conduct frequent testing. What is your day in the life like as a product manager? The data and AI space is rapidly evolving, so it’s very important to keep up to date. My day can include talking to customers, market analysis, putting together a product requirements document, or preparing marketing materials. My favorite part is getting to make diagrams illustrating how things will work, as it’s pretty fun to transform an idea into a visual. What’s the next big breakthrough in AI that everyone should be watching for? There’s a lot of big breakthroughs coming soon. One that I’m particularly interested in is the hyperpersonalization of content. For the past decade, ads have been fine-tuned to the specific watcher. Some elements of content have been tuned, such as what thumbnail Netflix shows a user, but the actual content (the video itself) has not been. It’ll be interesting to see how directors/producers balance telling the story they want to and the user’s interests. What’s been the biggest challenge you’ve faced in bringing this AI product to market, from concept to launch? I work on large-scale data storage, which can be very confusing to understand. We have various AI optimizations on data, but there’s often questions about when these optimizations are run, how they work, what they don’t cover, etc. With these kinds of questions, it is important to ensure we have clear, consistent messaging about what we are building and why. I’ve found that explaining the cause of limitations resonates very well with customers. How do you envision AI evolving to better understand and respond to human emotions, and what challenges or opportunities do you see in that area? Multimodal models are going to get significantly better in the coming years, which will change our primary mode of interaction with AI. Figuring out human emotion is significantly easier from visual or audio information compared to text. I think there's an opportunity to create more natural interactions in a wider array of scenarios. What performance metrics or KPIs do you track to gauge the success of this AI product? We typically want to see good feedback and usage. I talk to customers pretty often to get a sense of how and why they think about our products, which is key to explain why we see certain trends in metrics. How do you approach designing a product experience that’s not only functional but also engaging and memorable for users? Large-scale data products are notoriously challenging to use. Simple examples are easy to set up, but production workloads typically involve confusing configuration and code. It’s been a high priority for me to build out functionality that customers need, while making the product very simple to use. The future is where any business gets insights from their data easily. In the current world, data-driven business insights are typically restricted to the biggest companies - but even they would prefer a simpler experience. In the long-term, how do you think AI will help the individual reach one’s full potential? For the individual, I’m very excited about AI integrations into hardware. Up to now, we’ve largely seen AI in software applications like websites. There’s many larger applications of building devices that utilize AI, and we’re already starting to see some of those implications in cars and phones. Where do you see the product evolving in the next few years, and what features are you most excited to add? Databricks is on a path towards becoming increasingly simple and more powerful at the same time. There are a lot of efforts we are working on across the board, from making big-scale data and compute easy to work with to improving performance on queries and workflows. Personally, I think we have some exciting features coming soon throughout the product that make workflows way easier with AI. Examples include AI-generated comments on data, AI code suggestions in the notebook editors, and AI interfaces to chat with data (for example, Databricks AI/BI Genie). What’s your perspective on AI’s impact on jobs, and how do you address those concerns in your product strategy? There are concerns over whether AI would reduce the number of jobs. Our products are designed to increase valuable insights, which often come in conjunction with users. For example, with AI/BI Genie, users can create interfaces on their own data. This is a magical experience, where users can ask questions and get answers specific to them. In fact, users can even check the SQL being used to confirm it’s what they’re looking for. This is collaborative with analysts, reducing the time it takes them to go from idea to insight. What user feedback surprised you and led to changes in your roadmap, strategy or product experience? One major surprise for me was the complexity that some of the larger companies have. This introduces requirements into the product that I wouldn’t have considered on my own. A common example is thinking of migration strategies when introducing a new product. Typically, large companies will have either put together existing technologies (commonly open-source software) or have built custom software to solve the problem that our new product offering tackles. It usually takes a bit of time to understand why and how these are put together to ensure we have a solution that covers all possibilities. Would you like to take a stab at answering some of these questions? The link for the template is HERE. Interested in reading the content from all of our writing prompts? Click HERE. Would you like to take a stab at answering some of these questions? The link for the template is HERE. Interested in reading the content from all of our writing prompts? Click HERE. The link for the template is HERE. The link for the template is HERE. HERE. HERE.

"The Future is Where Any Business Gets Insights From Their Data Easily" says Aniruth from Databricks

Databricks-dən Aniruth deyir: "Gələcək hər hansı bir biznesin məlumatlarından asanlıqla məlumat əldə etdiyi yerdir"

"El futuro es donde cualquier empresa puede obtener información de sus datos fácilmente", afirma Aniruth de Databricks

“Masa Depan adalah Tempat Setiap Bisnis Mendapatkan Wawasan Dari Data Mereka Dengan Mudah” kata Aniruth dari Databricks

"Il futuro è dove qualsiasi azienda ottiene facilmente informazioni dai propri dati", afferma Aniruth di Databricks

「未来は、どんな企業でもデータから簡単に洞察を得られる時代です」とDatabricksのAniruth氏は語る

„Budúcnosť je tam, kde každá firma ľahko získa informácie zo svojich údajov,“ hovorí Aniruth z Databricks

"Mustaqbalku waa halka uu ganacsi kastaa si sahal ah uga helo xogta xogtooda" ayuu yidhi Aniruth oo ka socda Databricks

"Framtiden är där alla företag enkelt får insikter från sina data" säger Aniruth från Databricks

Анирут аз Databricks мегӯяд: "Оянда он ҷоест, ки ҳама гуна тиҷорат аз маълумоти худ ба осонӣ фаҳмиш мегирад"

"Kelajak - bu har qanday biznes o'z ma'lumotlaridan osongina ma'lumot oladigan joy", deydi Databricks'dan Aniruth

"Ikamva kulapho naliphi na ishishini lifumana ulwazi kwidatha yalo ngokulula" utshilo uAniruth ovela kwiDatabricks.

"Ikusasa Yilapho Noma Yiliphi Ibhizinisi Lithola Ukuqonda Emininingwaneni Yalo Kalula" kusho u-Aniruth ovela ku-Databricks

MLOps With Databricks and Spark - Part 1

Building CI Pipeline with Databricks Asset Bundle and GitLab

ডেটাব্রিক্স অ্যাসেট বান্ডেল এবং গিটল্যাব সহ সিআই পাইপলাইন তৈরি করা

Erstellen einer CI-Pipeline mit Databricks Asset Bundle und GitLab

Creación de canalización de CI con Databricks Asset Bundle y GitLab

Création d'un pipeline CI avec Databricks Asset Bundle et GitLab

डेटाब्रिक्स एसेट बंडल और गिटलैब के साथ CI पाइपलाइन का निर्माण

Databricks Asset Bundle と GitLab を使用した CI パイプラインの構築

Databricks Asset Bundle 및 GitLab을 사용하여 CI 파이프라인 구축

Construindo Pipeline de CI com Databricks Asset Bundle e GitLab

Создание конвейера CI с помощью Databricks Asset Bundle и GitLab

Databricks Asset Bundle ve GitLab ile CI Pipeline Oluşturma

Xây dựng quy trình CI với gói tài sản Databricks và GitLab

使用 Databricks Asset Bundle 和 GitLab 构建 CI 管道

If you are using Databricks serving endpoint, and you wish to export metrics to Datadog, you can face with some challenges in Datadog documentation. This article will help you to solve them, at least it will help you to make a PoC. Here are list of links which you can use: https://docs.databricks.com/en/machine-learning/model-serving/metrics-export-serving-endpoint.html https://docs.datadoghq.com/agent/basic_agent_usage/ubuntu/?tab=agentv6v7 Databricks serving endpoint is a service which allows you to deploy your model and serve it. It is a part of Databricks MLflow. You can find more information about it here: https://docs.databricks.com/applications/mlflow/model-serving.html Once you created an endpoint, out of the box you can see some metrics in Databricks UI. Also list of metrics can be received via REST API: curl --request GET "https://${DATABRICKS_HOST}/api/2.0/serving-endpoints/{endpoint_name}/metrics" --header "Authorization: Bearer ${DATABRICKS_TOKEN}" But if you want to export them to Datadog, you need to install and configure Datadog agent. Pre-requisites Created Databricks serving endpoint Bearer PAT token behalf of Databricks user/service principal, which has `CAN_QUERY` permission on the endpoint Datadog API key Enabled OpenMetrics integration in your Datadog account, link: https://app.datadoghq.com/integrations Install Datadog agent Datadog offers several ways to install agent, they are listed here: , however looks like some of them are not working, or at least I was not able to make them work. That is why I decided to use the most simple way — install agent on Docker. https://app.datadoghq.com/account/settings/agent/latest?platform=overview Before agent installation, you need to create a file `datadog.yaml`, which should be located ` `, with the following content: /etc/datadog-agent/datadog.yaml api_key: "__DD_API_KEY__"
site: "datadoghq.com"
hostname: "__YOUR_HOST_NAME__" Then you need to create `metric-config.yaml` file, which should be located ` `, with the following content: /etc/datadog-agent/conf.d/openmetrics.d/conf.yaml.default instances:
 - openmetrics_endpoint: https://__DATABRICKS_HOST__/api/2.0/serving-endpoints/__ENDPOINT_NAME__/metrics

   metrics:
     - cpu_usage_percentage:
         name: cpu_usage_percentage
         type: gauge
     - mem_usage_percentage:
        name: mem_usage_percentage
        type: gauge
     - provisioned_concurrent_requests_total:
        name: provisioned_concurrent_requests_total
        type: gauge
     - request_4xx_count_total:
        name: request_4xx_count_total
        type: gauge
     - request_5xx_count_total:
        name: request_5xx_count_total
        type: gauge
     - request_count_total:
        name: request_count_total
        type: gauge
     - request_latency_ms:
        name: request_latency_ms
        type: histogram

   tag_by_endpoint: false

   send_distribution_buckets: true

   headers:
     Authorization: Bearer __DATABRICKS_TOKEN__
     Content-Type: application/openmetrics-text The full possible options for ` ` can be found in template file which is located in a container under ` ` conf.yaml.default /opt/datadog-agent/agent/conf.d/openmetrics.d/ I’ve created the next , because I have some difficulties with that one which is provided in Datadog documentation: Dockerfile FROM ubuntu:20.04

# Set environment variables
ENV DD_API_KEY=[SPECIFY_DD_API_KEY]
ENV DD_SITE=datadoghq.com
ENV DD_HOSTNAME=datadog-agent-ubuntu
ENV DATABRICKS_HOST=[SPECIFY_DATABRICKS_HOST]
ENV DATABRICKS_TOKEN=[SPECIFY_DATABRICKS_TOKEN]
ENV ENDPOINT_NAME=[SPECIFY_ENDPOINT_NAME]

# Install curl
RUN apt-get update && apt-get install -y curl

# Copy the datadog.yaml file into the Docker image
COPY datadog.yaml /etc/datadog-agent/datadog.yaml
RUN sed -i -e "s@__DD_API_KEY__@$DD_API_KEY@g" /etc/datadog-agent/datadog.yaml

# Install the Datadog agent
RUN bash -c "$(curl -L https://s3.amazonaws.com/dd-agent/scripts/install_script_agent7.sh)"

# Copy the metrics-config.yaml file into the Docker image
COPY metrics-config.yaml /etc/datadog-agent/conf.d/openmetrics.d/conf.yaml.default

RUN sed -i -e "s@__DATABRICKS_HOST__@$DATABRICKS_HOST@g; s@__ENDPOINT_NAME__@$ENDPOINT_NAME@g; s@__DATABRICKS_TOKEN__@$DATABRICKS_TOKEN@g; " /etc/datadog-agent/conf.d/openmetrics.d/conf.yaml.default

RUN service datadog-agent restart

# Run a process that doesn't exit
CMD ["tail", "-f", "/dev/null"] Usage Run serving endpoint Define environment variables in `Dockerfile’ Build docker image Run docker container # Step 1: Build the Docker image
# Navigate to the directory containing the Dockerfile
cd /path/to/directory/with/dockerfile

# Build the Docker image
# Replace 'my-image' with the name you want to give to your Docker image
docker build -t datadoc-agent-image .

# Step 2: Run a container from the built image
# Replace 'my-container' with the name you want to give to your Docker container
docker run --name datadog-agent -d datadoc-agent-image

# Step 3: Connect to a container and restart agent
docker exec -it datadog-agent  bash
service datadog-agent restart Some usefully commands for DataDog agent: datadog-agent status
datadog-agent health

service datadog-agent start
service datadog-agent restart Also appears . here

How to Export Metrics from Databricks Serving Endpoint to Datadog

Introduction Databricks is emerging as one of the main players in the MLOps and DevOps world. In the last month, I experienced this as part of my project at Areto and decided to write a series of hands-on blog posts on how to implement a MLOps Pipeline with Databricks. This is an ongoing learning process for me as well. I write each blog post after I experiment with different features and tools. So, I like to see this blog series as collective learning. Any comments and suggestions are welcome. In this blog post, we'll explore how to build a pipeline using the Databricks DBX tool and GitLab. This guide is designed for data engineers and scientists who are looking to streamline their data processing workflows. We'll cover key concepts, practical applications, and step-by-step instructions to help you integrate these tools effectively into your data projects. Continuous Integration (CI) CI/CD (Continuous Integration/Continuous Delivery): Continuous Integration and Continuous Delivery/Deployment (CI/CD) have become fundamental in modern software development practices. At its core, CI/CD is a method that emphasizes short, frequent updates to software through automated pipelines. While traditionally associated with software development, its principles are increasingly being adopted in data engineering and data science. The essence of CI/CD lies in its ability to automate the stages of software development, particularly in building, testing, and deploying code.  Continuous Integration begins with regular code commits to a shared repository branch, ensuring collaborative development without version conflicts. Each commit undergoes an automated build and test process, validating changes to enhance the quality and reliability of the final product. Databricks: Databricks has emerged as a premier cloud-based platform, uniting the realms of data engineering, machine learning, and analytics. Databricks excels in handling large-scale data processing and complex analytical tasks, but to leverage its full potential, teams need to navigate its multifaceted environment effectively. Integrating CI/CD processes within Databricks environments can streamline workflows, ensuring consistent, reproducible, and scalable data operations. This integration is crucial for teams aiming to maintain agility and efficiency in their data projects, enabling them to deliver reliable, high-quality data solutions consistently. Databricks Workflows: Databricks workflows are particularly significant in the context of CI/CD, as they offer a robust platform for automating and orchestrating data pipelines crucial to continuous integration and deployment. These workflows allow for the seamless scheduling and execution of tasks, which is essential for maintaining the frequent, automated update cycles characteristic of CI/CD. In a CI/CD pipeline, Databricks workflows can be used to automatically process and test large datasets, ensuring that data transformations and analyses are consistently accurate and up-to-date. The diversity of tasks that can be executed within Databricks workflows, such as Notebooks for exploratory data analysis, Python scripts for structured data processing, and Python wheels for custom package deployment, aligns well with the varied needs of CI/CD pipelines. By integrating these tasks into CI/CD workflows, data teams can ensure that every aspect of their data processing and analysis is continuously tested and integrated into the larger data strategy. This integration is key for developing resilient, scalable, and efficient data operations, enabling teams to deliver high-quality, reliable data products rapidly. Databricks CLI Extension (DBX): The Databricks CLI Extension, or DBX, is a pivotal tool in integrating Databricks with CI/CD pipelines, enhancing the automation and management of data workflows. The ability to programmatically control and manipulate data processes is crucial to implementing a CI/CD workflow, and DBX fills this role effectively. It provides a command-line interface for interacting with various Databricks components, such as workspaces, workflows, and clusters, facilitating seamless integration into automated pipelines. Hands-on In our hands-on example, we will create a (very) minimal project that uses dbx to deploy and run a Databricks workflow to manipulate, analyze, and test some data. This practical exercise showcases how Databricks can be used for real-world data analysis tasks. Our CI pipeline will play a crucial role in this process, as it will automate the deployment of our code as a Databricks workflow and validate its output. This validation step is vital to ensure the accuracy and reliability of our analysis. By the end of this exercise, you'll have a clear understanding of how to set up and run a Databricks workflow within a CI pipeline and how such a setup can be beneficial in analyzing and deriving insights from large datasets. This example will not only demonstrate the technical application of Databricks and CI principles but also offer a glimpse into the practical benefits of automated data analysis in a business context. Development pattern using Databricks GUI There are various development patterns to implement our CI pipeline. The common development workflows with dbx are to develop, test, and debug your code on your local environments and then use dbx to batch-run the local code on a target cluster. By integrating a remote repository with Databricks, we use a CI/CD platform such as , , or to automate running our remote repo’s code on our clusters. GitHub Actions Azure DevOps GitLab In this tutorial, for the sake of simplicity, we use Databricks GUI to develop and test our code. We follow these steps: Create a remote repository and clone it into our Databricks workspace. We use here. Gitlab Develop the program logic and test it inside the Databricks GUI. This includes Python scripts to build a Python Wheel package, scripts to test the data quality using Pytest, and a notebook to run the Pytest. Push the code to GitLab. The git push will trigger a Gitlab Runner to build, deploy and launch our workflows on Databricks using dbx Git integration As the first step, we . Next, we create a remote repository and . To allow our Gitlab runner to communicate with Databricks API through dbx, we should add two environment variables, and to our CI/CD pipeline configurations. configure Git credentials & connect a remote repo to Databricks clone it to our Databricks repo DATABRICKS_HOST DATABRICKS_TOKEN to generate a Databricks token, in your Databricks, go to User Settings →  Developer → Access tokens → manage → Generate new token. The Databricks host is the URL when you login into your Databricks workspace. It would be something like . The last part of the URL is your workspace id, and you should ignore that. https://dbc-dc87ke16-h5h2.cloud.databricks.com/ Finally we add the token and host to our Gitlab CI/CD setting. For that open your repo in Gitlab, go to Settings→ CI/CD → Variables → Add variables The project skeleton the project is structured into several folders and key files, each serving a specific purpose: .dbx folder: : defining the configuration of your DBX project. It contains environment settings, dependencies, and other project-specific parameters. project.json conf folder: : outlining the deployment configurations and environment settings. It defines workflows and their respective parameters and cluster configurations. deployment.yml : our wheel package. This folder includes: my_package folder A subfolder containing the main ETL task script . The ETL task load our dataset and create two new tables. tasks sample_etl_job.py The file. it includes common utilities that provide access to components such as common.py SparkSession Contains two Jupyter notebooks: notebooks folder: : plot the distribution of different features in our dataset explorative_analysis : used to execute for unit testing run_unit_test pytest This folder is dedicated to testing: tests folder: : includes pytest fixtures conftest.py : contains unit tests to validate the data structure in our tables. test_data.py In addition to these folders, there are two important files in the root directory: : the configuration file for GitLab's Continuous Integration (CI) service, defining the instructions and commands the CI pipeline should execute. .gitlab-ci.yml : for building my Python wheel package. It defines the package's metadata, dependencies, and build instructions. setup.py dbx-tutorial/
├─ .dbx/
│  ├─ project.json
├─ conf/
│  ├─ deployment.yml
├─ my_package/
│  ├─ tasks/
│  │  ├─ __init__.py
│  │  ├─ sample_etl_job.py
│  ├─ __init__.py
│  ├─ common.py
├─ tests/
│  ├─ conftest.py
│  ├─ test_data.py
├─ notebooks/
│  ├─ explorative_analysis
│  ├─ run_unit_test
├─ .gitlab-ci.yml
├─ setup.py Deployment configuration file is crucial for defining your DBX project's configuration. This file could be generated automatically when you run . Check the dbx website for more details about the . project.json dbx --init Profile file reference The “ ” option is used for local development. If you run the command inside the CI tool, you need to specify the and environment variables, and they will the profile variable. Since we don’t want to use the local environment and develop and run our code on Databricks UI, we don’t need to specify this option here. Instead, the dbx will pick them up from the CI/CD setting that we set earlier. profile dbx databricks host databricks token overwrite Your files that dbx will automatically upload to the system will be stored in They will be stored in .  You can read more about this in ml experiment. artifact_location dbx documentation {
    "environments": { 
        "default": {
            "profile": "dbx-tutorial",
            "storage_type": "mlflow",
            "properties": {
                "workspace_directory": "/Shared/dbx/dbx-tutorial", 
                "artifact_location": "dbfs:/Shared/dbx/projects/dbx-tutorial"
            }
        }
    },
    "inplace_jinja_support": false,
    "failsafe_cluster_reuse_with_assets": false,
    "context_based_upload_for_execute": false
} outlines the deployment configurations and environment settings. Here, we use and format to define the workflows. Each workflow defines an object inside the Databricks workspace: or deployment.yml 2.1 API wheel_task job delta live tables Pipeline. custom:
  existing_cluster_id: &existing_cluster_id
    existing_cluster_id: "1064-xxxxxx-xxxxxxxxx"

environments:
  default: #this should be same envoriment name as you defined in the project.json
    workflows:
      - name: "etl_job"
        tasks:
          - task_key: "main"
            <<: *existing_cluster_id
            python_wheel_task:
              package_name: "my_package"
              entry_point: "etl_job" # take a look at the setup.py entry_points section for details on how to define an entrypoint
          - task_key: "eda"
            <<: *existing_cluster_id
            notebook_task:
              notebook_path: "/Repos/<your DB username>/dbx-tutorial/notebooks/explorative_analysis"
              source: WORKSPACE
            depends_on:
              - task_key: "main"    
      - name: "test_job"
        tasks:
          - task_key: "main"
            <<: *existing_cluster_id
            notebook_task:
              notebook_path: "/Repos/<your DB username>/dbx-tutorial/notebooks/run_unit_test"
              source: WORKSPACE
            libraries:
            - pypi:
                package: pytest Here, we define two workflows: : consists of two tasks/jobs. "etl_job" The first task ( ) is of type . It used to run ETL task using the entry point that we define in the file. main python_wheel_task setup.py The second task ( ) is of type . It is used to run the notebook after the main tasks run successfully. Notice the use of property. etl notebook_task explorative_analysis depends_on : consists of only one job of type . It is used to run the notebook that is "test_job" notebook_task responsible for running the pytest. : in the above example, we run the workflow on an existing cluster. You can also create and run a new cluster for every deployment and launch of your work. For that, you should change the code as follows: Note custom:
  basic-cluster-props: &basic-cluster-props
    spark_version: "11.3.x-cpu-ml-scala2.12"

  basic-static-cluster: &basic-static-cluster
    new_cluster:
      <<: *basic-cluster-props
      num_workers: 1
      node_type_id: "Standard_E8_v3"

environments:
  default: 
    workflows:
      - name: "et_job"
        tasks:
          - task_key: "main"
            <<: *basic-static-cluster
						.... CI pipeline configuration Your GitLab CI pipeline is structured to automate the testing and deployment processes of your Databricks project. It consists of two main stages: and . In the stage, the runs the unit tests and deploys a separate workflow for testing. The stage, activated upon successful completion of the test stage, handles the deployment of your main ETL workflow. In general, the pipeline follows these steps: test deployment test unit-test-job deploy Build the project Push the build artifacts to the Databricks workspace Install the wheel package on your cluster Create the jobs on Databricks Workflows Run the jobs image: python:3.9

stages:          # List of stages for jobs, and their order of execution
  - test
  - deploy

unit-test-job:   # This job runs in the test stage.
  stage: test    # It only starts when the job in the build stage completes successfully.
  script:
    - echo "Running unit tests... This will take about 60 seconds."
    - echo "Code coverage is 90%"
    - pip install -e ".[local]"
    - dbx deploy --deployment-file conf/deployment.yml test_job --assets-only
    - dbx launch test_job --from-assets --trace

deploy-job:      # This job runs in the deploy stage.
  stage: deploy  # It only runs when *both* jobs in the test stage complete successfully.
  script:
    - echo "Deploying application..."
    - echo "Install dependencies"
    - pip install -e ".[local]"
    - echo "Deploying Job"
    - dbx deploy --deployment-file conf/deployment.yml etl_job
    - dbx launch etl_job --trace
    - echo "Application successfully deployed."
    - echo "remove all workflows." For workflow deployment and lunch, we use two commands: : to deploy the workflow definitions to the given environment. dbx deploy : to launch the given workload by its name in a given environment. dbx lunch In the we use the flag to avoid creating a job definition on our Databricks workflows during deployment. For the lunch command, we should use the flag. If this flag is provided, the launch command will search for the latest deployment that was assets-only. Lunching this, we can not see anything in the Workflows UI, but it will be a non-titled job  running by submit API. You can see this on the Job Runs page **** of your Databricks workspace. Check out the dbx documentation to read about and unit-test-job --assets-only --from-assets assest-based workflows other commands If you check the “Experiments” page of your Databricks workspace, you will find the build artifacts of each run of your CI pipeline. Databricks Asset Bundles currently, Databricks recommends using Asset Bundles for CI/CD and offers a migration guide from dbx to bundles.  If you understand the concepts in this post, it would be easy to switch to bundles. In the next post, I am going to explain how we can convert this project to an asset bundle project. Resources for this tutorial. Make sure you update the file with the correct and Repository conf/deployment.yml cluster_id DB username dbx documentation Databricks Best Practices: Development loop and CI/CD on Databricks with dbx Databricks Jobs API CI/CD techniques with Git and Databricks Repos What is CI/CD on Azure Databricks?

Building a CI Pipeline with Databricks dbx Tool and GitLab

In this post, we will look at creating an Azure data factory with a pipeline that loads Office 365 event data incrementally based on change data capture (CDC) information in the source of Change Data Feed(CDF) of a Delta lake table to an AWS S3 bucket. What we’ll cover: Create an ADF Pipeline that loads Calendar events from Offfice365 to a Blob container. Run a Databricks Notebook with the activity in the ADF pipeline, transform extracted Calendar event and merge to a Delta lake table. Load Change Data Feed on the Delta lake table to an AWS S3 bucket. The following diagram shows the architecture of the above steps: Prerequisites To build the pipeline, there are a few prerequisites that need to be met: Setup an Azure account. Register an AD application with Microsoft Graph permission. . Configure Microsoft Graph data connect consent request approver group t. Create an Azure Storage Accoun Create an AWS S3 bucket. Before going any further, we will need to create an Azure Storage account and grant the Azure AD application we registered the proper access to the container of the storage account. Check out the step-by-step Azure for Authorizing access to blobs using Azure Active Directory. guide 1. Create an Azure Data Factory Pipeline In this step, we will create an ADF Pipeline that loads Office365 Calendar Views data to an Azure Blob container. I will assume you already have an Azure Subscription and Resource Group setup. To create an Azure data factory, run the following command: az datafactory create --resource- --factory-name ADFDemo group RESOURCE_GROUP 1.1 Create linked services and datasets To connect to Office 365, we need to use the following command to create a linked service for Office 365 connection: az datafactory linked - service create - - resource - group RESOURCE_GROUP - - factory - name ADFDemo - - linked - service - name Offic365LinkedService - - properties @offic365LinkedService . json Replace APP_CLIENT_ID , TENANT_ID and APP_CLIENT_SECRET with corresponding values from your Active Directory tenant and App. Next, create a Calendar dataset named calendarDataset to reference the Calendar view data from Office 365. az datafactory dataset create - - resource - group RESOURCE_GROUP - - dataset - name calendarDataset - - factory - name ADFDemo - - properties @calendarDataset . json Now, let’s create another Linked Services for Blob Storage Account to store calendar data extracted from Office 365. az datafactory linked - service create - - resource - group RESOURCE_GROUP - - factory - name ADFDemo - - linked - service - name AzureStorageLinkedService - - properties @azureStorageLinkedService . json ECRET Replace YOUR_CONTAINER , APP_CLIENT_S , TENANT_ID and APP_CLIENT_SECRET with corresponding values from your Active Directory tenant, App, and storage account. Next, create a binary dataset named . outputDataset az datafactory dataset create - - resource - group RESOURCE_GROUP - - dataset - name outputDataset - - factory - name ADFDemo - - properties @outputDataset . json Note that @dataset.outputPath in the folderPath section is a global variable that we will set in the next step. 1.2 Create an ADF pipeline In this section, we will use the Azure CLI command to create a simple ADF pipeline with three activities: to set a global variable . SetVariableActivity outputPath named to load data from Office 365 to Azure blob storage. CopyActivity ExtractCalendarView to throw an error when the pipeline fails. FailActivity Enter the following commands to create ADF Pipeline with activities: az datafactory pipeline create --resource-group RESOURCE_GROUP --factory-name ADFDemo --name DemoPipeline  --pipeline @demoPipeline.json Once the pipeline is created, Navigate to the Azure portal and launch the Azure Data Factory studio. We can view the pipeline activities we have created from the pipeline designer. 2. Create Notebook in Azure Databricks In this section, we will create a notebook in Azure Databricks. You can refer to this to create a notebook in the Azure Databricks workspace. tutorial Before starting the notebook, we have to set up a connection from Azure Databricks to Azure Storage Account using a SAS key. Check out this for creating a Databricks secret scope. tutorial We are going to perform the following steps in the notebook: Mount a Blob storage container. Transform calendar view data. Create an SCD Type 1 delta table with Change Data Feed enabled. Merge input data frame to the delta table. Configure AWS S3 connection, and load change feed data to s3 bucket. The notebook would look similar to this: Now we have our notebook created, Let’s move back to step 1.1 to create linked services for Azure Databricks connection. az datafactory linked - service create - - resource - group RESOURCE_GROUP - - factory - name ADFDemo - - linked - service - name databricksLinkedService - - properties @databricksLinkedService . json Next, delete the pipeline that we created in step 1.2, recreate it to execute the Databricks notebook against the Databricks jobs cluster. Delete the existing pipeline we have created: az datafactory pipeline delete - - factory - name - - name ADFDemo - - resource - group RESOURCE_GROUP then recreate the pipeline using the following command: az datafactory pipeline create - - resource - group RESOURCE_GROUP - - factory - name ADFDemo - - name DemoPipeline - - pipeline @demoPipeline . json Now we should have our pipeline that looks like the following diagram. Let’s publish entities (linked services and pipeline) to the Azure Data Factory service using ADF studio. and trigger the pipeline. We can log on to the Azure Databricks workspace, go to Clusters and see the Job status as . pending execution, running, or terminated After the pipeline run is completed, log on to the AWS S3 console, we should see the CSV file in the following format is loaded to the destination s3 bucket. Conclusion In this article, I introduced the approach to load data incrementally from Office365 to an AWS S3 bucket using ADF and Azure Databricks. I hope you found this article useful. You can find the ARM template and notebook in my . GitHub repository

Building an ETL Pipeline to Load Data Incrementally from Office365 to S3 using ADF and Databricks

Introduction The Data space is almost as volatile as the AI space this year, with many players consolidating. In the summer of 2022, I wrote about The Data Table Format Wars, and the ecosystem has only gotten more interesting since then. I won’t get deep into the weeds on Data Lakes vs. Data Warehouse, etc., but suffice it to say, we are seeing everything coalesce around decoupling data storage from computing. Table formats are a way to standardize lightly opinionated storage. As I opined a couple of years ago, Apache Iceberg seems to be where they are settling. These formats tend to use Apache Parquet on object storage and provide a method for tracking new data and data state changes. In 2023, Databricks announced the open-sourcing of their Unity Catalog, and Onehouse announced OneTable (now X-Table). These projects are meant to give you a single interface to the three popular table formats: Delta, Hudi, and Iceberg. Then, June 2024 happened… What Happened? Snowflake was having its annual Snowflake Summit conference, and on June 3, 2024, they announced Apache Polaris (incubating), a vendor-neutral, open catalog for Apache Iceberg. The next day, and a week before the Databricks conference, Databricks announced their acquisition of Tabular, a company founded by the creators of Iceberg. This still seems weird to everyone from the outside because Iceberg is a spec; it isn’t a product, and Tabular isn’t even the main contributor to Iceberg. In my opinion, Databricks will somehow roll their Deltalake into Iceberg. Behind the scenes, Snowflake had been trying to acquire Tabular. Something happened at the 11th hour, and Databricks came in and laid down a ton more money, and the Snowflake bid was discarded. Snowflake was already two years into their commitment to supporting Iceberg as external storage, which made a lot of sense for them regardless. Snowflake has a reputation concerning the release speed after an announcement, so there was some skepticism, and some even wondered if any code had been written. It turns out Snowflake had been working on this for a while, and it was made available in very short order, with vendors like Dremio, Starburst, CelerData, and Upsolver showing solutions using it in less than two months: a delightful surprise, and a positive sign of things to come. All of that is a long way to set up Apache Polaris, and hopefully, it will give you a sense of the importance that a good catalog plays in all of this. How will any of your compute tools work if you don’t have a unified system to describe all those parquet files in your data lake? Polaris Overview Apache Polaris is an open-source catalog implementation for Apache Iceberg tables. Its main features include: REST API implementation: Polaris implements Iceberg's REST API, enabling interoperability across multiple query engines.
Multi-engine compatibility: It allows various query engines, such as Apache Spark, Apache Flink, Trino, and others, to read and write Iceberg tables through a single catalog.
Centralized management: Polaris provides centralized and secure access to Iceberg tables across REST-compatible query engines.
Namespace and metadata management: It supports creating namespaces for logical grouping and manages metadata for Iceberg tables.
Storage configurations: Polaris offers robust storage configurations for cloud providers like S3, Azure, and Google Cloud Storage.
Security features: It implements role-based access control (RBAC) and uses credential vending to secure query execution.
Open-source nature: Polaris is fully open-source, allowing for community contributions and vendor neutrality.
Interoperability: It enables multiple engines to be used on a single copy of data, reducing storage duplication and ETL costs.
Consistent access control: Polaris allows you to manage role-based access control and storage layer security consistently across engines from one place.
Flexibility in hosting: It can be hosted on various infrastructures, providing flexibility and reducing vendor lock-in. And a lovely little graphic from the docs to put a picture on everything. Borrowing from the Polaris docs for a picture once again, let’s dive into what is happening here in this entity hierarchy graphic: We can have multiple catalogs in a single Polaris install; in this diagram, it is only Catalog1 that has any namespaces or tables defined. A single catalog can be either internal or external. The internal tables allow for read/write functionality, meaning you have full control. The external tables are currently read-only and are intended to mirror data that is in some external catalog, such as a Glue catalog, which Polaris will see as the source of truth once connected. The catalog organizes your iceberg tables; you’ll want to configure it with your storage configuration for the applicable cloud system that you are on. This forms the first architectural layer in the Iceberg table spec. A namespace can have an arbitrary level of nesting and is used to logically group Iceberg tables within a catalog. An Iceberg table will always belong to a namespace. With this configuration in place, you can then use the Polaris catalog to describe your Iceberg tables in such a way as to make them easily accessible and manageable using a large variety of tools. That’s a quick overview of what is happening and what it looks like. The implementation inside of Snowflake provides a really handy UI to manage the implementation; other vendors have their own, and you can just do it through Spark if you like. Summary I continue to believe that Apache Iceberg is going to “win” the table format wars. Apache Polaris focusing on Iceberg is not a bad thing. I want to dig into how RBAC has been implemented in Polaris another time and see if it is as robust as what the Tabular product implemented, which I was a big fan of. I believe Polaris is a big win for the Iceberg community and will help accelerate adoption even further. I’ve seen talk about the Dremio Nessie project getting rolled in, and I’m curious to see what that will look like. I haven’t looked at Nessie in a couple of years, and I thought their git-like semantics mainly had been covered by the Iceberg tagging and branching, which I wrote about last year—yet another story for another day. I’m a big fan of what is happening here. If you are in this space, this is the right train to jump on. Check out my other What the Heck is… articles at the links below: What The Heck Is DuckDB?
What the Heck Is Malloy?
What the Heck is PRQL?
What the Heck is GlareDB?
What the Heck is SeaTunnel?
What the Heck is LanceDB?
What the heck is SDF?
What the Heck is Paimon?
What the Heck is Proton?
What the Heck is PuppyGraph?
What the Heck is GPTScript?
What the Heck is WarpStream? Introduction Introduction The Data space is almost as volatile as the AI space this year, with many players consolidating. In the summer of 2022, I wrote about The Data Table Format Wars , and the ecosystem has only gotten more interesting since then. I won’t get deep into the weeds on Data Lakes vs. Data Warehouse, etc., but suffice it to say, we are seeing everything coalesce around decoupling data storage from computing. The Data Table Format Wars Table formats are a way to standardize lightly opinionated storage. As I opined a couple of years ago, Apache Iceberg seems to be where they are settling. These formats tend to use Apache Parquet on object storage and provide a method for tracking new data and data state changes. In 2023, Databricks announced the open-sourcing of their Unity Catalog, and Onehouse announced OneTable (now X-Table). These projects are meant to give you a single interface to the three popular table formats: Delta, Hudi, and Iceberg. Then, June 2024 happened… announced announced What Happened? Snowflake was having its annual Snowflake Summit conference, and on June 3, 2024, they announced Apache Polaris (incubating), a vendor-neutral, open catalog for Apache Iceberg. The next day, and a week before the Databricks conference, Databricks announced their acquisition of Tabular, a company founded by the creators of Iceberg. This still seems weird to everyone from the outside because Iceberg is a spec; it isn’t a product, and Tabular isn’t even the main contributor to Iceberg. In my opinion, Databricks will somehow roll their Deltalake into Iceberg. announced announced Behind the scenes, Snowflake had been trying to acquire Tabular. Something happened at the 11th hour, and Databricks came in and laid down a ton more money, and the Snowflake bid was discarded. Snowflake was already two years into their commitment to supporting Iceberg as external storage, which made a lot of sense for them regardless. Snowflake has a reputation concerning the release speed after an announcement, so there was some skepticism, and some even wondered if any code had been written. It turns out Snowflake had been working on this for a while, and it was made available in very short order, with vendors like Dremio, Starburst, CelerData, and Upsolver showing solutions using it in less than two months: a delightful surprise, and a positive sign of things to come. All of that is a long way to set up Apache Polaris , and hopefully, it will give you a sense of the importance that a good catalog plays in all of this. How will any of your compute tools work if you don’t have a unified system to describe all those parquet files in your data lake? Apache Polaris Polaris Overview Apache Polaris is an open-source catalog implementation for Apache Iceberg tables. Its main features include: REST API implementation: Polaris implements Iceberg's REST API, enabling interoperability across multiple query engines. Multi-engine compatibility: It allows various query engines, such as Apache Spark, Apache Flink, Trino, and others, to read and write Iceberg tables through a single catalog. Centralized management: Polaris provides centralized and secure access to Iceberg tables across REST-compatible query engines. Namespace and metadata management: It supports creating namespaces for logical grouping and manages metadata for Iceberg tables. Storage configurations: Polaris offers robust storage configurations for cloud providers like S3, Azure, and Google Cloud Storage. Security features: It implements role-based access control (RBAC) and uses credential vending to secure query execution. Open-source nature: Polaris is fully open-source, allowing for community contributions and vendor neutrality. Interoperability: It enables multiple engines to be used on a single copy of data, reducing storage duplication and ETL costs. Consistent access control: Polaris allows you to manage role-based access control and storage layer security consistently across engines from one place. Flexibility in hosting: It can be hosted on various infrastructures, providing flexibility and reducing vendor lock-in. REST API implementation: Polaris implements Iceberg's REST API, enabling interoperability across multiple query engines. Multi-engine compatibility: It allows various query engines, such as Apache Spark, Apache Flink, Trino, and others, to read and write Iceberg tables through a single catalog. Centralized management: Polaris provides centralized and secure access to Iceberg tables across REST-compatible query engines. Namespace and metadata management: It supports creating namespaces for logical grouping and manages metadata for Iceberg tables. Storage configurations: Polaris offers robust storage configurations for cloud providers like S3, Azure, and Google Cloud Storage. Security features: It implements role-based access control (RBAC) and uses credential vending to secure query execution. Open-source nature: Polaris is fully open-source, allowing for community contributions and vendor neutrality. Interoperability: It enables multiple engines to be used on a single copy of data, reducing storage duplication and ETL costs. Consistent access control: Polaris allows you to manage role-based access control and storage layer security consistently across engines from one place. Flexibility in hosting: It can be hosted on various infrastructures, providing flexibility and reducing vendor lock-in. And a lovely little graphic from the docs to put a picture on everything. Borrowing from the Polaris docs for a picture once again, let’s dive into what is happening here in this entity hierarchy graphic: We can have multiple catalogs in a single Polaris install; in this diagram, it is only Catalog1 that has any namespaces or tables defined. A single catalog can be either internal or external. The internal tables allow for read/write functionality, meaning you have full control. The external tables are currently read-only and are intended to mirror data that is in some external catalog, such as a Glue catalog, which Polaris will see as the source of truth once connected. The catalog organizes your iceberg tables; you’ll want to configure it with your storage configuration for the applicable cloud system that you are on. This forms the first architectural layer in the Iceberg table spec. A namespace can have an arbitrary level of nesting and is used to logically group Iceberg tables within a catalog. An Iceberg table will always belong to a namespace. With this configuration in place, you can then use the Polaris catalog to describe your Iceberg tables in such a way as to make them easily accessible and manageable using a large variety of tools. That’s a quick overview of what is happening and what it looks like. The implementation inside of Snowflake provides a really handy UI to manage the implementation; other vendors have their own, and you can just do it through Spark if you like. Summary I continue to believe that Apache Iceberg is going to “win” the table format wars. Apache Polaris focusing on Iceberg is not a bad thing. I want to dig into how RBAC has been implemented in Polaris another time and see if it is as robust as what the Tabular product implemented, which I was a big fan of. I believe Polaris is a big win for the Iceberg community and will help accelerate adoption even further. I’ve seen talk about the Dremio Nessie project getting rolled in, and I’m curious to see what that will look like. I haven’t looked at Nessie in a couple of years, and I thought their git-like semantics mainly had been covered by the Iceberg tagging and branching, which I wrote about last year—yet another story for another day. wrote about I’m a big fan of what is happening here. If you are in this space, this is the right train to jump on. Check out my other What the Heck is… articles at the links below: What The Heck Is DuckDB? What the Heck Is Malloy? What the Heck is PRQL? What the Heck is GlareDB? What the Heck is SeaTunnel? What the Heck is LanceDB? What the heck is SDF? What the Heck is Paimon? What the Heck is Proton? What the Heck is PuppyGraph? What the Heck is GPTScript? What the Heck is WarpStream? What The Heck Is DuckDB? What The Heck Is DuckDB? What the Heck Is Malloy? What the Heck Is Malloy? What the Heck is PRQL? What the Heck is PRQL? What the Heck is GlareDB? What the Heck is GlareDB? What the Heck is SeaTunnel? What the Heck is SeaTunnel? What the Heck is LanceDB? What the Heck is LanceDB? What the heck is SDF? What the heck is SDF? What the Heck is Paimon? What the Heck is Paimon? What the Heck is Proton? What the Heck is Proton? What the Heck is PuppyGraph? What the Heck is PuppyGraph? What the Heck is GPTScript? What the Heck is GPTScript? What the Heck is WarpStream? What the Heck is WarpStream?

DATABRICKS

databricks.com

5K-10K employees

Since 2013

COMPANY RANKING

DATABRICKS

EVERGREEN INDEX #5019

Representing this company? Click here to claim and customize this page!

Databricks WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

DATABRICKS

databricks.com

5K-10K employees

Since 2013

COMPANY RANKING

DATABRICKS

EVERGREEN INDEX #5019

RELATED COMPANIESRANK

STORIES ABOUT DATABRICKS

Representing this company? Click here to claim and customize this page!

Databricks WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!