Since <strong>Docker</strong> is a US company, we must comply with US export control regulations. In an effort to comply with these, we now block all IP addresses that are located in Cuba, Iran, North Korea, Republic of Crimea, Sudan, and Syria. If you are not in one of these cities, countries, or regions and ...

Docker hub перестал работать в России — Разработка на vc.ru

Securely build, share and run modern applications anywhere

Docker

Your Docker Containers Are Probably Full of Security Holes—Here’s How to Fix Them Now

Firms increasingly make use of artificial intelligence (AI) infrastructures to host and manage autonomous workloads. Consequently, there has been significant demand for scalable as well as resilient infrastructures that will be able to meet heterogeneous application or cloud requirements. Organizations use Kubernetes and Docker to meet such needs because firms realize that both are highly effective use cases that deliver scalable AI infrastructures. Deploying AI infrastructure typically provides adequate computation power to execute and process large datasets. These demands can translate into the need for scalable methods that enable AI models to run on large workloads without hurting performance. Why Companies Need to Scale Up Their AI Infrastructure AI systems, nonetheless, are also resource-intensive, normally demanding both high computing capacity and the ability to process high levels of data. As more advanced AI applications and a larger scale become required, scalability becomes more critical. Scalability ensures that AI systems can handle increasing workloads without any loss of performance. Expanding Data Volumes The growing amount of data is a concern for AI systems in many facets. Most AI models, especially those based on deep learning, heavily depend on large amounts of data during training and inference. However, without adequate scalable infrastructure, processing and interpreting such enormous quantities of data is a roadblock. Optimized Performance Scalable AI hardware supports reliable and stable performance despite drastically overwhelming computational loads. With Kubernetes, horizontal scaling of AI jobs is a breeze, and the dynamic resizing of replica numbers can be done as a function of necessity. In contrast, Docker containers support lean, isolated environments for running AI models where resource conflict is not a performance bottleneck. Effective Resource Management Efficient use of resources is the key to cost-effective and sustainable AI deployment. Kubernetes' resource requests and limits allow for fine-grained CPU and memory resource management by avoiding underprovisioning and overprovisioning. Docker's resource management fills the gap by isolating container resources. Scaling AI Infrastructure With Kubernetes and Docker Containerization is one of the milestones in the evolution of scalable artificial intelligence infrastructure. Containerization of the AI application and its dependencies in a Docker container ensures consistency throughout the development, testing, and deployment environments. First, you must define a Dockerfile in order to install the environment. The Dockerfile is a series of instructions about how to build a Docker image. It declares a base image, the dependencies required, and the initial setup commands that apply to your app. The following is a basic Dockerfile for a Python machine-learning model: # Use an official Python runtime as a parent image
FROM python:3.9-slim
 
# Set the working directory in the container
WORKDIR /usr/src/app
 
# Copy the current directory contents into the container
COPY . .
 
# Install any needed packages specified in requirements.txt
RUN pip install --no-cache-dir -r requirements.txt
 
# Expose the port the app runs on
EXPOSE 5000
 
# Define environment variable
ENV NAME World
 
# Run the app
CMD ["python", "./app.py"] If the Dockerfile is ready, then you can build the Docker image and run the container. Run the following commands: # Build the Docker image
docker build -t ml-model:latest .
 
# Run the container
docker run -p 5000:5000 ml-model:latest Deploying the Dockerized AI Model to Kubernetes Kubernetes provides a wide range of orchestration features that enable efficient application management in the containerized infrastructure. Deployment of the Docker image on Kubernetes ensures that a specified number of application replicas is always running. The following is an example of deployment.yaml file that you can use to deploy your Dockerized machine learning model: apiVersion: apps/v1
kind: Deployment
metadata:
  name: ml-model-deployment
spec:
  replicas: 3  
  selector:
	matchLabels:
  	app: ml-model
  template:
	metadata:
  	labels:
    	app: ml-model
	spec:
  	containers:
  	- name: ml-model-container
    	image: ml-model:latest
    	ports:
    	- containerPort: 5000 The above code snippet shows how to deploy the AI model, but you also need to make the model externally accessible. You will need to expose it by defining a Kubernetes Service. The service.yaml below illustrates an example: apiVersion: v1
kind: Service
metadata:
  name: ml-model-service
spec:
  selector:
	app: ml-model
  ports:
	- protocol: TCP
  	port: 80
  	targetPort: 5000
  type: LoadBalancer Use the kubectl command-line tool to apply the deployment and service configurations: # Deploy the application
kubectl apply -f deployment.yaml
 
# Expose the service
kubectl apply -f service.yaml Scaling With Kubernetes Kubernetes provides excellent scaling capabilities to AI environments, maximizing resource utilization and performance. Horizontal scaling is done by adding additional containers, and vertical scaling involves adding additional resources like CPU or memory to a container. Horizontal Scaling Horizontal scaling is used to scale up the number of replicas (Pods) of an AI system to handle a higher workload. The process requires enabling dynamic scaling depending on the number of replicas. The command used to enable such a process is `kubectl scale`. The particular command is used to set up the deployment to function up to a maximum of five replicas: `kubectl scale --replicas=5 deployment/ml-model-deployment` The command scales up the ml-model-deployment to use five replicas of the machine-learning model container. The system dynamically provisions more Pods to meet the required number afterward. Automatic Scaling using the Horizontal Pod Autoscaler (HPA) Kubernetes facilitates auto-scaling using the Horizontal Pod Autoscaler (HPA). The HPA dynamically adjusts the number of replicas based on resource use, i.e., CPU or memory, in relation to set limits. The YAML configuration shown below is a relevant example of an HPA that dynamically scales for ml-model-deployment in response to CPU use: apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  name: ml-model-hpa
spec:
  scaleTargetRef:
	apiVersion: apps/v1
	kind: Deployment
	name: ml-model-deployment
  minReplicas: 2
  maxReplicas: 10
  targetCPUUtilizationPercentage: 50 In this setup, scaleTargetRef is used to define the Deployment to be scaled, i.e., ml-model-deployment. The minimum replica count is set using MinReplicas, while the maximum replica count is controlled using maxReplicas. In addition, the CPU utilization percentage is set using targetCPUUtilizationPercentage, i.e., to 50%. CPU utilization of more than 50% across all Pods results in scaling up the replica count to a maximum of 10 automatically. As soon as CPU utilization drops below the set percentage, Kubernetes automatically reduces the replica count in order to release resources. Vertical Scaling Horizontal scaling is mainly to cope with more traffic, whereas vertical scaling provides more resources (such as CPU or memory) to existing containers. The process is to scale up or down resource requests and limits in the Kubernetes Deployment. In order to scale up the CPU and memory limits of the ml-model-deployment, one would need to open the deployment.yaml file: apiVersion: apps/v1
kind: Deployment
metadata:
  name: ml-model-deployment
spec:
  replicas: 3
  selector:
	matchLabels:
  	app: ml-model
  template:
	metadata:
  	labels:
    	app: ml-model
	spec:
  	containers:
  	- name: ml-model-container
    	image: ml-model:latest
    	ports:
    	- containerPort: 5000
    	resources:
      	requests:
        	cpu: "1"
        	memory: "2Gi"
      	limits:
        	cpu: "2"
        	memory: "4Gi" In this updated configuration: requests specify the minimum resources required for the container.limits define the maximum resources the container can use. Firms increasingly make use of artificial intelligence (AI) infrastructures to host and manage autonomous workloads. Consequently, there has been significant demand for scalable as well as resilient infrastructures that will be able to meet heterogeneous application or cloud requirements. Organizations use Kubernetes and Docker to meet such needs because firms realize that both are highly effective use cases that deliver scalable AI infrastructures. there has been significant demand for scalable Kubernetes Docker Deploying AI infrastructure typically provides adequate computation power to execute and process large datasets. These demands can translate into the need for scalable methods that enable AI models to run on large workloads without hurting performance. Why Companies Need to Scale Up Their AI Infrastructure Why Companies Need to Scale Up Their AI Infrastructure AI systems , nonetheless, are also resource-intensive, normally demanding both high computing capacity and the ability to process high levels of data. As more advanced AI applications and a larger scale become required, scalability becomes more critical. Scalability ensures that AI systems can handle increasing workloads without any loss of performance. AI systems Expanding Data Volumes Expanding Data Volumes The growing amount of data is a concern for AI systems in many facets. Most AI models, especially those based on deep learning, heavily depend on large amounts of data during training and inference. However, without adequate scalable infrastructure, processing and interpreting such enormous quantities of data is a roadblock . enormous quantities of data is a roadblock Optimized Performance Optimized Performance Scalable AI hardware supports reliable and stable performance despite drastically overwhelming computational loads. With Kubernetes, horizontal scaling of AI jobs is a breeze, and the dynamic resizing of replica numbers can be done as a function of necessity. In contrast, Docker containers support lean, isolated environments for running AI models where resource conflict is not a performance bottleneck. Effective Resource Management Effective Resource Management Efficient use of resources is the key to cost-effective and sustainable AI deployment. Kubernetes' resource requests and limits allow for fine-grained CPU and memory resource management by avoiding underprovisioning and overprovisioning. Docker's resource management fills the gap by isolating container resources. Scaling AI Infrastructure With Kubernetes and Docker Scaling AI Infrastructure With Kubernetes and Docker Containerization is one of the milestones in the evolution of scalable artificial intelligence infrastructure. Containerization of the AI application and its dependencies in a Docker container ensures consistency throughout the development, testing, and deployment environments. First, you must define a Dockerfile in order to install the environment. The Dockerfile is a series of instructions about how to build a Docker image. It declares a base image, the dependencies required, and the initial setup commands that apply to your app. The following is a basic Dockerfile for a Python machine-learning model: # Use an official Python runtime as a parent image
FROM python:3.9-slim
 
# Set the working directory in the container
WORKDIR /usr/src/app
 
# Copy the current directory contents into the container
COPY . .
 
# Install any needed packages specified in requirements.txt
RUN pip install --no-cache-dir -r requirements.txt
 
# Expose the port the app runs on
EXPOSE 5000
 
# Define environment variable
ENV NAME World
 
# Run the app
CMD ["python", "./app.py"] If the Dockerfile is ready, then you can build the Docker image and run the container. Run the following commands: # Build the Docker image
docker build -t ml-model:latest .
 
# Run the container
docker run -p 5000:5000 ml-model:latest Deploying the Dockerized AI Model to Kubernetes Deploying the Dockerized AI Model to Kubernetes Kubernetes provides a wide range of orchestration features that enable efficient application management in the containerized infrastructure. Deployment of the Docker image on Kubernetes ensures that a specified number of application replicas is always running. The following is an example of deployment.yaml file that you can use to deploy your Dockerized machine learning model : Kubernetes deploy your Dockerized machine learning model apiVersion: apps/v1
kind: Deployment
metadata:
  name: ml-model-deployment
spec:
  replicas: 3  
  selector:
	matchLabels:
  	app: ml-model
  template:
	metadata:
  	labels:
    	app: ml-model
	spec:
  	containers:
  	- name: ml-model-container
    	image: ml-model:latest
    	ports:
    	- containerPort: 5000 The above code snippet shows how to deploy the AI model, but you also need to make the model externally accessible. You will need to expose it by defining a Kubernetes Service. The service.yaml below illustrates an example: apiVersion: v1
kind: Service
metadata:
  name: ml-model-service
spec:
  selector:
	app: ml-model
  ports:
	- protocol: TCP
  	port: 80
  	targetPort: 5000
  type: LoadBalancer Use the kubectl command-line tool to apply the deployment and service configurations: # Deploy the application
kubectl apply -f deployment.yaml
 
# Expose the service
kubectl apply -f service.yaml Scaling With Kubernetes Scaling With Kubernetes Kubernetes provides excellent scaling capabilities to AI environments, maximizing resource utilization and performance. Horizontal scaling is done by adding additional containers, and vertical scaling involves adding additional resources like CPU or memory to a container. Horizontal Scaling Horizontal Scaling Horizontal scaling is used to scale up the number of replicas (Pods) of an AI system to handle a higher workload. The process requires enabling dynamic scaling depending on the number of replicas. The command used to enable such a process is `kubectl scale`. The particular command is used to set up the deployment to function up to a maximum of five replicas: `kubectl scale --replicas=5 deployment/ml-model-deployment` The command scales up the ml-model-deployment to use five replicas of the machine-learning model container. The system dynamically provisions more Pods to meet the required number afterward. Automatic Scaling using the Horizontal Pod Autoscaler (HPA) Automatic Scaling using the Horizontal Pod Autoscaler (HPA) Kubernetes facilitates auto-scaling using the Horizontal Pod Autoscaler (HPA). The HPA dynamically adjusts the number of replicas based on resource use, i.e., CPU or memory, in relation to set limits. The YAML configuration shown below is a relevant example of an HPA that dynamically scales for ml-model-deployment in response to CPU use: apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  name: ml-model-hpa
spec:
  scaleTargetRef:
	apiVersion: apps/v1
	kind: Deployment
	name: ml-model-deployment
  minReplicas: 2
  maxReplicas: 10
  targetCPUUtilizationPercentage: 50 In this setup, scaleTargetRef is used to define the Deployment to be scaled, i.e., ml-model-deployment. The minimum replica count is set using MinReplicas, while the maximum replica count is controlled using maxReplicas. In addition, the CPU utilization percentage is set using targetCPUUtilizationPercentage, i.e., to 50%. CPU utilization of more than 50% across all Pods results in scaling up the replica count to a maximum of 10 automatically. As soon as CPU utilization drops below the set percentage, Kubernetes automatically reduces the replica count in order to release resources. Vertical Scaling Vertical Scaling Horizontal scaling is mainly to cope with more traffic, whereas vertical scaling provides more resources (such as CPU or memory) to existing containers. The process is to scale up or down resource requests and limits in the Kubernetes Deployment. In order to scale up the CPU and memory limits of the ml-model-deployment, one would need to open the deployment.yaml file: apiVersion: apps/v1
kind: Deployment
metadata:
  name: ml-model-deployment
spec:
  replicas: 3
  selector:
	matchLabels:
  	app: ml-model
  template:
	metadata:
  	labels:
    	app: ml-model
	spec:
  	containers:
  	- name: ml-model-container
    	image: ml-model:latest
    	ports:
    	- containerPort: 5000
    	resources:
      	requests:
        	cpu: "1"
        	memory: "2Gi"
      	limits:
        	cpu: "2"
        	memory: "4Gi" In this updated configuration: requests specify the minimum resources required for the container. limits define the maximum resources the container can use. requests specify the minimum resources required for the container. limits define the maximum resources the container can use.

How to Scale AI Infrastructure With Kubernetes and Docker

Com escalar la infraestructura d'IA amb Kubernetes i Docker

Πώς να κλιμακώσετε την υποδομή AI με το Kubernetes και το Docker

Cómo escalar la infraestructura de IA con Kubernetes y Docker

چگونه زیرساخت هوش مصنوعی را با Kubernetes و Docker مقیاس بندی کنیم

Kako skalirati AI infrastrukturu s Kubernetesom i Dockerom

Kubernetes と Docker で AI インフラストラクチャを拡張する方法

Kaip padidinti AI infrastruktūrą naudojant „Kubernetes“ ir „Docker“.

Kubernetes සහ Docker සමඟ AI යටිතල පහසුකම් පරිමාණය කරන්නේ කෙසේද?

Hur man skalar AI-infrastruktur med Kubernetes och Docker

Paano I-scale ang AI Infrastructure Gamit ang Kubernetes at Docker

کبرنیٹس اور ڈوکر کے ساتھ اے آئی انفراسٹرکچر کی پیمائش کیسے کریں۔

Cách mở rộng cơ sở hạ tầng AI với Kubernetes và Docker

Optimizing Docker Images is More Than Just a One and Done Thing

Optimitzar les imatges de Docker és més que una cosa feta

Optimizar las imágenes de Docker es más que una tarea que se realiza una sola vez

بهینه سازی تصاویر Docker چیزی فراتر از یک کار انجام شده است

Docker イメージの最適化は一度で終わるものではありません

Ny fanatsarana ny sary Docker dia mihoatra noho ny zavatra iray sy vita

Докерын зургуудыг оновчтой болгох нь зөвхөн нэг бөгөөд хийсэн зүйл биш юм

डकर छविहरू अप्टिमाइज गर्नु केवल एक र सम्पन्न चीज भन्दा बढी हो

Otimizar imagens do Docker é mais do que algo feito uma vez e pronto

Docker Imahinakuna allinchayqa Aswan Hukllamanta chaymanta Rurasqamanta

Hagaajinta Sawirrada Docker waa in ka badan hal iyo wax la sameeyay

Docker suratlaryny optimizirlemek diňe bir we ýerine ýetirilen zat däl

Ang Pag-optimize sa Mga Imahe ng Docker ay Higit pa sa Isa at Tapos na Bagay

This is the third part of the series Container Networking. I will explain a little bit about the docker container networking in this blog post. I followed the steps mentioned in this to install docker on Ubuntu. Post docker install, you can see the docker0 device in the list ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 02:fd:4d:34:55:76 brd ff:ff:ff:ff:ff:ff
11: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default 
    link/ether 02:42:cf:b4:c1:a8 brd ff:ff:ff:ff:ff:ff Let’s create a busybox container. docker run --name bb -dt busybox docker ps -a
CONTAINER ID   IMAGE     COMMAND   CREATED              STATUS              PORTS    NAMES
02429964e449   busybox   "sh"      About a minute ago   Up About a minute             bb In the device list, you see the veth7b01920@if16 interface is created with the master as docker0. ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 02:fd:4d:34:55:76 brd ff:ff:ff:ff:ff:ff
11: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether 02:42:cf:b4:c1:a8 brd ff:ff:ff:ff:ff:ff
17: veth7b01920@if16: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP mode DEFAULT group default 
    link/ether e2:15:50:88:11:03 brd ff:ff:ff:ff:ff:ff link-netnsid 2 Let’s check the network namespace. ip netns list And you won’t see any network namespaces…How come??? This is because: ip netns list command looks up network namespaces file in the 
/var/run/netns directory.

However, the Docker daemon doesn’t create a reference of
the network namespace file in the /var/run/netns directory
after the creation. Therefore, ip netns ls cannot resolve the
network namespace file

Ref: https://www.baeldung.com/linux/docker-network-namespace-invisible If you want ip netns list to show the namespace name docker has created, then follow the below steps export container_name=bb
container_pid=$(sudo docker inspect -f '' $container_name)
echo $container_pid

sudo touch /var/run/netns/$container_name
sudo mount -o bind /proc/$container_pid/ns/net /var/run/netns/$container_name Now, you can see the namespace name. ip netns list
bb (id: 0) sudo ip netns exec bb ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
16: eth0@if17: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0 The following diagram will help you visualize it better. With this, I’m signing off. I hope this blog series helps to de-clutter container networking. This is the third part of the series Container Networking . I will explain a little bit about the docker container networking in this blog post. Container Networking I followed the steps mentioned in this to install docker on Ubuntu. this Post docker install, you can see the docker0 device in the list docker0 ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 02:fd:4d:34:55:76 brd ff:ff:ff:ff:ff:ff
11: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default 
    link/ether 02:42:cf:b4:c1:a8 brd ff:ff:ff:ff:ff:ff ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 02:fd:4d:34:55:76 brd ff:ff:ff:ff:ff:ff
11: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default 
    link/ether 02:42:cf:b4:c1:a8 brd ff:ff:ff:ff:ff:ff Let’s create a busybox container. docker run --name bb -dt busybox docker run --name bb -dt busybox docker ps -a
CONTAINER ID   IMAGE     COMMAND   CREATED              STATUS              PORTS    NAMES
02429964e449   busybox   "sh"      About a minute ago   Up About a minute             bb docker ps -a
CONTAINER ID   IMAGE     COMMAND   CREATED              STATUS              PORTS    NAMES
02429964e449   busybox   "sh"      About a minute ago   Up About a minute             bb In the device list, you see the veth7b01920@if16 interface is created with the master as docker0. veth7b01920@if16 ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 02:fd:4d:34:55:76 brd ff:ff:ff:ff:ff:ff
11: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether 02:42:cf:b4:c1:a8 brd ff:ff:ff:ff:ff:ff
17: veth7b01920@if16: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP mode DEFAULT group default 
    link/ether e2:15:50:88:11:03 brd ff:ff:ff:ff:ff:ff link-netnsid 2 ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 02:fd:4d:34:55:76 brd ff:ff:ff:ff:ff:ff
11: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether 02:42:cf:b4:c1:a8 brd ff:ff:ff:ff:ff:ff
17: veth7b01920@if16: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP mode DEFAULT group default 
    link/ether e2:15:50:88:11:03 brd ff:ff:ff:ff:ff:ff link-netnsid 2 Let’s check the network namespace. ip netns list ip netns list And you won’t see any network namespaces…How come??? This is because: ip netns list command looks up network namespaces file in the 
/var/run/netns directory.

However, the Docker daemon doesn’t create a reference of
the network namespace file in the /var/run/netns directory
after the creation. Therefore, ip netns ls cannot resolve the
network namespace file

Ref: https://www.baeldung.com/linux/docker-network-namespace-invisible ip netns list command looks up network namespaces file in the 
/var/run/netns directory.

However, the Docker daemon doesn’t create a reference of
the network namespace file in the /var/run/netns directory
after the creation. Therefore, ip netns ls cannot resolve the
network namespace file

Ref: https://www.baeldung.com/linux/docker-network-namespace-invisible If you want ip netns list to show the namespace name docker has created, then follow the below steps ip netns list export container_name=bb
container_pid=$(sudo docker inspect -f '' $container_name)
echo $container_pid

sudo touch /var/run/netns/$container_name
sudo mount -o bind /proc/$container_pid/ns/net /var/run/netns/$container_name export container_name=bb
container_pid=$(sudo docker inspect -f '' $container_name)
echo $container_pid

sudo touch /var/run/netns/$container_name
sudo mount -o bind /proc/$container_pid/ns/net /var/run/netns/$container_name Now, you can see the namespace name. ip netns list
bb (id: 0) ip netns list
bb (id: 0) sudo ip netns exec bb ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
16: eth0@if17: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0 sudo ip netns exec bb ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
16: eth0@if17: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0 The following diagram will help you visualize it better. With this, I’m signing off. I hope this blog series helps to de-clutter container networking.

A Guide to Container Networking: Part III - Docker Container Networking

When building Python applications with Poetry in a Docker container, we sometimes encounter issues accessing private packages stored in the Google Artifact Registry (GAR). Locally, this challenge arises because docker build cannot directly handle the Google Cloud credentials in the same way as our CI/CD pipeline, where we leverage service accounts and the Kaniko action for secure builds. To resolve this for local development, we'll configure Google Cloud credentials within Docker using Docker Compose, enabling secure access to the GAR repository during the build process. Non-Containerized Pre-requirements Python ^3.12
Poetry ^1.7.1
Google Cloud SDK ^489.0.0
Keyring ^24.0.0
keyrings.google-artifactregistry-auth ^1.0.0
Read access to your GAR Python repository
Logged into GCP (gcloud auth application-default login) Setup Keyring Setup
Install keyring support for Google Artifact Registry:
pip install keyring
pip install keyrings.google-artifactregistry-auth Connect Poetry to your GAR repo
Access to the private repository in the Google Artifact Registry can be managed through Poetry. First, configure a custom source in Poetry for the GAR repository by running:
poetry source add --priority=explicit <PACKAGE_NAME> https://<REGION>-python.pkg.dev/<PROJECT>/<REGISTRY>/simple Be sure to append /simple to the repository URL for compatibility. Now you can install packages from your private repo: poetry add --source <PACKAGE_NAME> some-package@1.0.0 Notes In some cases, accessing the repository may require setting an explicit OAuth token for authentication in Poetry. Use the following command to configure this globally in Poetry: poetry config http-basic.<PACKAGE_NAME> oauth2accesstoken $(gcloud auth print-access-token) Containerized Pre-requirements Docker ^20.10
Docker Compose ^2.0
Google Cloud SDK ^489.0.0
Read access to your GAR Python repository
Logged into GCP (gcloud auth application-default login) Setup Secrets Configuration
First, define a secret in docker-compose.yaml using the local path to your credentials file:
secrets:
  gcloud_credentials:
    file: ~/.config/gcloud/application_default_credentials.json



We define gcloud_credentials.file as a consistent path for Unix-like environments in docker-compose.yaml:


This configuration securely passes the credentials file from your local machine to the build context without exposing sensitive data.




Dockerfile Adjustments
In the Dockerfile, we handle credentials with the following setup:
ARG GOOGLE_APPLICATION_CREDENTIALS
ENV GOOGLE_APPLICATION_CREDENTIALS=${GOOGLE_APPLICATION_CREDENTIALS}

RUN --mount=type=secret,id=gcloud_credentials \
    mkdir -p $(dirname ${GOOGLE_APPLICATION_CREDENTIALS}) && \
    cp /run/secrets/gcloud_credentials ${GOOGLE_APPLICATION_CREDENTIALS}



--mount=type=secret,id=gcloud_credentials: Securely mounts the credentials during the build process.


GOOGLE_APPLICATION_CREDENTIALS: Specifies the credential file's path within the container.




Service Configurations
Here, gcloud_credentials is the secret mounted at build time, as specified in the secrets configuration.
some-service:
  build:
    context: .
    dockerfile: ./Dockerfile.local
    args:
      GOOGLE_APPLICATION_CREDENTIALS: /tmp/application_default_credentials.json
    secrets:
      - gcloud_credentials Usage Just run: docker-compose up --build Conclusion This approach allows local Docker builds to access private GAR resources securely, ensuring that credentials are handled appropriately and remain protected. When building Python applications with Poetry in a Docker container, we sometimes encounter issues accessing private packages stored in the Google Artifact Registry (GAR). Locally, this challenge arises because docker build cannot directly handle the Google Cloud credentials in the same way as our CI/CD pipeline, where we leverage service accounts and the Kaniko action for secure builds. To resolve this for local development, we'll configure Google Cloud credentials within Docker using Docker Compose, enabling secure access to the GAR repository during the build process. docker build Docker Non-Containerized Pre-requirements Python ^3.12 Poetry ^1.7.1 Google Cloud SDK ^489.0.0 Keyring ^24.0.0 keyrings.google-artifactregistry-auth ^1.0.0 Read access to your GAR Python repository Logged into GCP (gcloud auth application-default login) Python ^3.12 Poetry ^1.7.1 Google Cloud SDK ^489.0.0 Keyring ^24.0.0 keyrings.google-artifactregistry-auth ^1.0.0 Read access to your GAR Python repository Logged into GCP ( gcloud auth application-default login ) gcloud auth application-default login Setup Keyring Setup
Install keyring support for Google Artifact Registry:
pip install keyring
pip install keyrings.google-artifactregistry-auth Keyring Setup Install keyring support for Google Artifact Registry: pip install keyring
pip install keyrings.google-artifactregistry-auth Keyring Setup Keyring Setup Install keyring support for Google Artifact Registry: keyring pip install keyring
pip install keyrings.google-artifactregistry-auth pip install keyring
pip install keyrings.google-artifactregistry-auth Connect Poetry to your GAR repo
Access to the private repository in the Google Artifact Registry can be managed through Poetry. First, configure a custom source in Poetry for the GAR repository by running:
poetry source add --priority=explicit <PACKAGE_NAME> https://<REGION>-python.pkg.dev/<PROJECT>/<REGISTRY>/simple Connect Poetry to your GAR repo Access to the private repository in the Google Artifact Registry can be managed through Poetry. First, configure a custom source in Poetry for the GAR repository by running: poetry source add --priority=explicit <PACKAGE_NAME> https://<REGION>-python.pkg.dev/<PROJECT>/<REGISTRY>/simple Connect Poetry to your GAR repo Connect Poetry to your GAR repo Access to the private repository in the Google Artifact Registry can be managed through Poetry. First, configure a custom source in Poetry for the GAR repository by running: poetry source add --priority=explicit <PACKAGE_NAME> https://<REGION>-python.pkg.dev/<PROJECT>/<REGISTRY>/simple poetry source add --priority=explicit <PACKAGE_NAME> https://<REGION>-python.pkg.dev/<PROJECT>/<REGISTRY>/simple Be sure to append /simple to the repository URL for compatibility. Be sure to append /simple to the repository URL for compatibility. /simple Now you can install packages from your private repo: poetry add --source <PACKAGE_NAME> some-package@1.0.0 poetry add --source <PACKAGE_NAME> some-package@1.0.0 Notes In some cases, accessing the repository may require setting an explicit OAuth token for authentication in Poetry. Use the following command to configure this globally in Poetry: poetry config http-basic.<PACKAGE_NAME> oauth2accesstoken $(gcloud auth print-access-token) poetry config http-basic.<PACKAGE_NAME> oauth2accesstoken $(gcloud auth print-access-token) Containerized Pre-requirements Docker ^20.10 Docker Compose ^2.0 Google Cloud SDK ^489.0.0 Read access to your GAR Python repository Logged into GCP (gcloud auth application-default login) Docker ^20.10 Docker Compose ^2.0 Google Cloud SDK ^489.0.0 Read access to your GAR Python repository Logged into GCP ( gcloud auth application-default login ) gcloud auth application-default login Setup Secrets Configuration
First, define a secret in docker-compose.yaml using the local path to your credentials file:
secrets:
  gcloud_credentials:
    file: ~/.config/gcloud/application_default_credentials.json



We define gcloud_credentials.file as a consistent path for Unix-like environments in docker-compose.yaml:


This configuration securely passes the credentials file from your local machine to the build context without exposing sensitive data. Dockerfile Adjustments
In the Dockerfile, we handle credentials with the following setup:
ARG GOOGLE_APPLICATION_CREDENTIALS
ENV GOOGLE_APPLICATION_CREDENTIALS=${GOOGLE_APPLICATION_CREDENTIALS}

RUN --mount=type=secret,id=gcloud_credentials \
    mkdir -p $(dirname ${GOOGLE_APPLICATION_CREDENTIALS}) && \
    cp /run/secrets/gcloud_credentials ${GOOGLE_APPLICATION_CREDENTIALS}



--mount=type=secret,id=gcloud_credentials: Securely mounts the credentials during the build process.


GOOGLE_APPLICATION_CREDENTIALS: Specifies the credential file's path within the container. Service Configurations
Here, gcloud_credentials is the secret mounted at build time, as specified in the secrets configuration.
some-service:
  build:
    context: .
    dockerfile: ./Dockerfile.local
    args:
      GOOGLE_APPLICATION_CREDENTIALS: /tmp/application_default_credentials.json
    secrets:
      - gcloud_credentials Secrets Configuration First, define a secret in docker-compose.yaml using the local path to your credentials file: secrets:
  gcloud_credentials:
    file: ~/.config/gcloud/application_default_credentials.json We define gcloud_credentials.file as a consistent path for Unix-like environments in docker-compose.yaml:


This configuration securely passes the credentials file from your local machine to the build context without exposing sensitive data. Secrets Configuration Secrets Configuration First, define a secret in docker-compose.yaml using the local path to your credentials file: docker-compose.yaml secrets:
  gcloud_credentials:
    file: ~/.config/gcloud/application_default_credentials.json secrets:
  gcloud_credentials:
    file: ~/.config/gcloud/application_default_credentials.json We define gcloud_credentials.file as a consistent path for Unix-like environments in docker-compose.yaml: This configuration securely passes the credentials file from your local machine to the build context without exposing sensitive data. We define gcloud_credentials.file as a consistent path for Unix-like environments in docker-compose.yaml: We define gcloud_credentials.file as a consistent path for Unix-like environments in docker-compose.yaml : gcloud_credentials.file docker-compose.yaml This configuration securely passes the credentials file from your local machine to the build context without exposing sensitive data. This configuration securely passes the credentials file from your local machine to the build context without exposing sensitive data. Dockerfile Adjustments In the Dockerfile, we handle credentials with the following setup: ARG GOOGLE_APPLICATION_CREDENTIALS
ENV GOOGLE_APPLICATION_CREDENTIALS=${GOOGLE_APPLICATION_CREDENTIALS}

RUN --mount=type=secret,id=gcloud_credentials \
    mkdir -p $(dirname ${GOOGLE_APPLICATION_CREDENTIALS}) && \
    cp /run/secrets/gcloud_credentials ${GOOGLE_APPLICATION_CREDENTIALS} --mount=type=secret,id=gcloud_credentials: Securely mounts the credentials during the build process.


GOOGLE_APPLICATION_CREDENTIALS: Specifies the credential file's path within the container. Dockerfile Adjustments Dockerfile Adjustments In the Dockerfile, we handle credentials with the following setup: ARG GOOGLE_APPLICATION_CREDENTIALS
ENV GOOGLE_APPLICATION_CREDENTIALS=${GOOGLE_APPLICATION_CREDENTIALS}

RUN --mount=type=secret,id=gcloud_credentials \
    mkdir -p $(dirname ${GOOGLE_APPLICATION_CREDENTIALS}) && \
    cp /run/secrets/gcloud_credentials ${GOOGLE_APPLICATION_CREDENTIALS} ARG GOOGLE_APPLICATION_CREDENTIALS
ENV GOOGLE_APPLICATION_CREDENTIALS=${GOOGLE_APPLICATION_CREDENTIALS}

RUN --mount=type=secret,id=gcloud_credentials \
    mkdir -p $(dirname ${GOOGLE_APPLICATION_CREDENTIALS}) && \
    cp /run/secrets/gcloud_credentials ${GOOGLE_APPLICATION_CREDENTIALS} --mount=type=secret,id=gcloud_credentials: Securely mounts the credentials during the build process. GOOGLE_APPLICATION_CREDENTIALS: Specifies the credential file's path within the container. --mount=type=secret,id=gcloud_credentials: Securely mounts the credentials during the build process. --mount=type=secret,id=gcloud_credentials : Securely mounts the credentials during the build process. --mount=type=secret,id=gcloud_credentials GOOGLE_APPLICATION_CREDENTIALS: Specifies the credential file's path within the container. GOOGLE_APPLICATION_CREDENTIALS : Specifies the credential file's path within the container. GOOGLE_APPLICATION_CREDENTIALS Service Configurations Here, gcloud_credentials is the secret mounted at build time, as specified in the secrets configuration. some-service:
  build:
    context: .
    dockerfile: ./Dockerfile.local
    args:
      GOOGLE_APPLICATION_CREDENTIALS: /tmp/application_default_credentials.json
    secrets:
      - gcloud_credentials Service Configurations Service Configurations Here, gcloud_credentials is the secret mounted at build time, as specified in the secrets configuration. gcloud_credentials some-service:
  build:
    context: .
    dockerfile: ./Dockerfile.local
    args:
      GOOGLE_APPLICATION_CREDENTIALS: /tmp/application_default_credentials.json
    secrets:
      - gcloud_credentials some-service:
  build:
    context: .
    dockerfile: ./Dockerfile.local
    args:
      GOOGLE_APPLICATION_CREDENTIALS: /tmp/application_default_credentials.json
    secrets:
      - gcloud_credentials Usage Just run: docker-compose up --build docker-compose up --build Conclusion This approach allows local Docker builds to access private GAR resources securely, ensuring that credentials are handled appropriately and remain protected. Docker

Accessing Private Google Artifact Registry with Poetry: Local and Docker Setup

How to Use Docker to Template Local Database

Are you here because of the error message above when building the Dockerfile? You're not alone. This is a common issue that many developers encounter. Understanding the Issue Dockerfile keywords can be written in uppercase or lowercase, but it’s best not to mix them for clarity and consistency. The error message "WARN: FromAsCasing: 'as' and 'FROM' keywords' casing do not match" specifically points to a mismatch in the casing of the 'FROM' and 'AS' keywords. This rule highlights instances where mixed case is used in a FROM instruction that includes the AS keyword to specify a stage name. For example, FROM ubuntu as a builder would trigger this warning because 'FROM' is in uppercase while 'as' is in lowercase. Why Casing Matters Consistency in casing improves readability and reduces the potential for errors. When working in teams or on open-source projects, maintaining a standard practice for casing can prevent misunderstandings and ensure that scripts and Dockerfiles are more predictable and easier to manage. In Docker, the 'FROM' keyword is used to specify the base image, and the 'AS' keyword is used to name the build stage in multi-stage builds. The inconsistency in casing can lead to warnings and, in some cases, errors that can disrupt your build process. Recent Changes in Docker Some recent versions of Docker enforce stricter guidelines for keyword casing. This means that Docker now expects the 'FROM' and 'AS' keywords to match in casing, typically both being in uppercase. This enforcement helps maintain a standard across Dockerfiles and aids in avoiding potential issues that arise from mixed casing. Quick Fix To resolve this issue, you need to ensure that both 'FROM' and 'AS' keywords are in the same case. The recommended approach is to use uppercase for both, as shown below: # Use the Ubuntu image and name the stage "builder"
FROM ubuntu:latest AS builder This will prevent the warning from appearing during your build process. Additional Tips Consistency Across the Dockerfile: Ensure that all Dockerfile instructions follow the same casing convention. This includes other keywords like RUN, CMD, COPY, and ENV.
Linting Tools: Use Dockerfile linting tools to automatically check for issues related to casing and other best practices. These tools can be integrated into your CI/CD pipeline to catch errors early.
Documentation and Team Standards: Document the casing standards in your project's guidelines and ensure that all team members are aware of and follow these standards.
Updating Existing Dockerfiles: If you have existing Dockerfiles with mixed casing, take the time to update them. This can prevent future warnings and ensure that your Dockerfiles are compliant with the latest Docker versions. Consistency is key, and following these guidelines will help maintain the quality and reliability of your Docker projects. Are you here because of the error message above when building the Dockerfile? You're not alone. This is a common issue that many developers encounter. Understanding the Issue Understanding the Issue Dockerfile keywords can be written in uppercase or lowercase, but it’s best not to mix them for clarity and consistency. The error message "WARN: FromAsCasing: 'as' and 'FROM' keywords' casing do not match" specifically points to a mismatch in the casing of the 'FROM' and 'AS' keywords. This rule highlights instances where mixed case is used in a FROM instruction that includes the AS keyword to specify a stage name. For example, FROM ubuntu as a builder would trigger this warning because 'FROM' is in uppercase while 'as' is in lowercase. FROM ubuntu as a builder Why Casing Matters Why Casing Matters Consistency in casing improves readability and reduces the potential for errors. When working in teams or on open-source projects, maintaining a standard practice for casing can prevent misunderstandings and ensure that scripts and Dockerfiles are more predictable and easier to manage. In Docker, the 'FROM' keyword is used to specify the base image, and the 'AS' keyword is used to name the build stage in multi-stage builds. The inconsistency in casing can lead to warnings and, in some cases, errors that can disrupt your build process. Recent Changes in Docker Some recent versions of Docker enforce stricter guidelines for keyword casing. This means that Docker now expects the 'FROM' and 'AS' keywords to match in casing, typically both being in uppercase. This enforcement helps maintain a standard across Dockerfiles and aids in avoiding potential issues that arise from mixed casing. Quick Fix To resolve this issue, you need to ensure that both 'FROM' and 'AS' keywords are in the same case. The recommended approach is to use uppercase for both, as shown below: # Use the Ubuntu image and name the stage "builder"
FROM ubuntu:latest AS builder # Use the Ubuntu image and name the stage "builder"
FROM ubuntu:latest AS builder This will prevent the warning from appearing during your build process. Additional Tips Consistency Across the Dockerfile: Ensure that all Dockerfile instructions follow the same casing convention. This includes other keywords like RUN, CMD, COPY, and ENV. Linting Tools: Use Dockerfile linting tools to automatically check for issues related to casing and other best practices. These tools can be integrated into your CI/CD pipeline to catch errors early. Documentation and Team Standards: Document the casing standards in your project's guidelines and ensure that all team members are aware of and follow these standards. Updating Existing Dockerfiles: If you have existing Dockerfiles with mixed casing, take the time to update them. This can prevent future warnings and ensure that your Dockerfiles are compliant with the latest Docker versions. Consistency Across the Dockerfile : Ensure that all Dockerfile instructions follow the same casing convention. This includes other keywords like RUN , CMD , COPY , and ENV . Consistency Across the Dockerfile RUN CMD COPY ENV Linting Tools : Use Dockerfile linting tools to automatically check for issues related to casing and other best practices. These tools can be integrated into your CI/CD pipeline to catch errors early. Linting Tools CI/CD Documentation and Team Standards : Document the casing standards in your project's guidelines and ensure that all team members are aware of and follow these standards. Documentation and Team Standards Updating Existing Dockerfiles : If you have existing Dockerfiles with mixed casing, take the time to update them. This can prevent future warnings and ensure that your Dockerfiles are compliant with the latest Docker versions. Updating Existing Dockerfiles Consistency is key, and following these guidelines will help maintain the quality and reliability of your Docker projects.

Fix Dockerfile Error: WARN: FromAsCasing: 'as' and 'FROM' Keywords' Casing Do Not match

There is a requirement to use Spark Operator in a K8s cluster to run a spark job. The official image contains many vulnerabilities, including those due to Hadoop libraries. Let's build our own Spark Operator image. To build our image, we'll need a spark image as a base image and a Golang image to build Spark Operator itself. Spark image Building a Spark image without Hadoop using a specific version of Spark RUN curl -L https://dlcdn.apache.org/spark/spark-3.5.1/spark-3.5.1-bin-without-hadoop.tgz -o spark-3.5.1-bin-without-hadoop.tgz \
    && tar -xvzf spark-3.5.1-bin-without-hadoop.tgz \
    && mv spark-3.5.1-bin-without-hadoop /opt/spark \
    && rm spark-3.5.1-bin-without-hadoop.tgz Spark-operator image We build the Spark Operator image, we will need several Hadoop libraries to run submit commands. For example, the FIPS version build is given, the differences in the build and run commands. For building on Go, the parameter GOEXPERIMENT=boringcrypto is used For running spark-submit, the java parameter for Bouncy Castle is used Djavax.net.ssl.trustStorePassword=password You can build an image without FIPS changes. To run spark-submit, we will add Hadoop libraries during the build process: hadoop-client-runtime
hadoop-client-api
slf4j-api entrypoint.sh is used from the official Kubeflow repository https://github.com/kubeflow/spark-operator/blob/master/entrypoint.sh Example Dockerfile for building Spark Operator ARG SPARK_IMAGE=spark-3.5.1-bin-without-hadoop
ARG GOLANG_IMAGE=golang-1.21
ARG SPARK_OPERATOR_VERSION=1.3.1
ARG HADOOP_VERSION_DEFAULT=3.4.0
ARG HADOOP_TMP_HOME="/opt/hadoop"
ARG TARGETARCH=amd64

# Prepare spark-operator build
FROM ${GOLANG_IMAGE} as builder
WORKDIR /app/spark-operator

ARG SPARK_OPERATOR_VERSION
RUN curl -Ls https://github.com/kubeflow/spark-operator/archive/refs/tags/spark-operator-chart-${SPARK_OPERATOR_VERSION}.tar.gz | tar -xz --strip-components 1 -C /app/spark-operator

RUN GOTOOLCHAIN=go1.22.3 go mod download

# Build
ARG TARGETARCH
RUN CGO_ENABLED=0 GOOS=linux GOARCH=${TARGETARCH} GO111MODULE=on GOTOOLCHAIN=go1.22.3 GOEXPERIMENT=boringcrypto go build -a -o /app/spark-operator/spark-operator main.go

#Install Hadoop jars
ARG HADOOP_VERSION_DEFAULT
ARG HADOOP_TMP_HOME
RUN mkdir -p ${HADOOP_TMP_HOME}
RUN curl -Ls https://archive.apache.org/dist/hadoop/common/hadoop-${HADOOP_VERSION_DEFAULT}/hadoop-${HADOOP_VERSION_DEFAULT}.tar.gz | tar -xz --strip-components 1 -C ${HADOOP_TMP_HOME}

# Prepare spark-operator image
FROM ${ECR_URL}:${SPARK_IMAGE}
WORKDIR /opt/spark-operator
USER root

ENV PATH $JAVA_HOME/bin:$PATH
ENV SPARK_HOME="/opt/spark"
ENV JAVA_HOME="/opt/jdk-11.0.21"
ENV SPARK_SUBMIT_OPTS="${SPARK_SUBMIT_OPTS} -Djavax.net.ssl.trustStorePassword=password"
ENV PATH=${PATH}:${SPARK_HOME}/bin:${JAVA_HOME}/bin:

RUN yum update -y && \
    yum install --setopt=tsflags=nodocs -y openssl && \
    yum clean all

ARG HADOOP_TMP_HOME
COPY --from=builder ${HADOOP_TMP_HOME}/share/hadoop/client/hadoop-client-runtime-*.jar ${HADOOP_TMP_HOME}/share/hadoop/client/hadoop-client-api-*.jar ${HADOOP_TMP_HOME}/share/hadoop/common/lib/slf4j-api-*.jar /opt/spark/jars/

COPY --from=builder /app/spark-operator/spark-operator /opt/spark-operator/
COPY --from=builder /app/spark-operator/hack/gencerts.sh /usr/bin/

COPY entrypoint.sh /opt/spark-operator/
RUN chmod a+x /opt/spark-operator/entrypoint.sh
ENTRYPOINT ["/opt/spark-operator/entrypoint.sh"] Conclusion After the build, we still have several vulnerabilities in the Hadoop library hadoop-client-runtime: org.apache.avro:avro (hadoop-client-runtime-3.4.0.jar) – CVE-2023-39410
org.apache.commons:commons-compress – CVE-2024-25710, CVE-2024-26308 Since without this library we'll not be able to run spark-submit, but the rest of the huge part of the vulnerabilities is removed along with the main Hadoop libraries. There is a requirement to use Spark Operator in a K8s cluster to run a spark job. The official image contains many vulnerabilities, including those due to Hadoop libraries. Let's build our own Spark Operator image. To build our image, we'll need a spark image as a base image and a Golang image to build Spark Operator itself. Spark image Building a Spark image without Hadoop using a specific version of Spark RUN curl -L https://dlcdn.apache.org/spark/spark-3.5.1/spark-3.5.1-bin-without-hadoop.tgz -o spark-3.5.1-bin-without-hadoop.tgz \
    && tar -xvzf spark-3.5.1-bin-without-hadoop.tgz \
    && mv spark-3.5.1-bin-without-hadoop /opt/spark \
    && rm spark-3.5.1-bin-without-hadoop.tgz RUN curl -L https://dlcdn.apache.org/spark/spark-3.5.1/spark-3.5.1-bin-without-hadoop.tgz -o spark-3.5.1-bin-without-hadoop.tgz \
    && tar -xvzf spark-3.5.1-bin-without-hadoop.tgz \
    && mv spark-3.5.1-bin-without-hadoop /opt/spark \
    && rm spark-3.5.1-bin-without-hadoop.tgz Spark-operator image We build the Spark Operator image, we will need several Hadoop libraries to run submit commands. For example, the FIPS version build is given, the differences in the build and run commands. For building on Go, the parameter GOEXPERIMENT=boringcrypto is used GOEXPERIMENT=boringcrypto For running spark-submit, the java parameter for Bouncy Castle is used Djavax.net.ssl.trustStorePassword=password Djavax.net.ssl.trustStorePassword=password You can build an image without FIPS changes. To run spark-submit, we will add Hadoop libraries during the build process: hadoop-client-runtime hadoop-client-api slf4j-api hadoop-client-runtime hadoop-client-runtime hadoop-client-api hadoop-client-api slf4j-api slf4j-api entrypoint.sh is used from the official Kubeflow repository https://github.com/kubeflow/spark-operator/blob/master/entrypoint.sh entrypoint.sh https://github.com/kubeflow/spark-operator/blob/master/entrypoint.sh Example Dockerfile for building Spark Operator ARG SPARK_IMAGE=spark-3.5.1-bin-without-hadoop
ARG GOLANG_IMAGE=golang-1.21
ARG SPARK_OPERATOR_VERSION=1.3.1
ARG HADOOP_VERSION_DEFAULT=3.4.0
ARG HADOOP_TMP_HOME="/opt/hadoop"
ARG TARGETARCH=amd64

# Prepare spark-operator build
FROM ${GOLANG_IMAGE} as builder
WORKDIR /app/spark-operator

ARG SPARK_OPERATOR_VERSION
RUN curl -Ls https://github.com/kubeflow/spark-operator/archive/refs/tags/spark-operator-chart-${SPARK_OPERATOR_VERSION}.tar.gz | tar -xz --strip-components 1 -C /app/spark-operator

RUN GOTOOLCHAIN=go1.22.3 go mod download

# Build
ARG TARGETARCH
RUN CGO_ENABLED=0 GOOS=linux GOARCH=${TARGETARCH} GO111MODULE=on GOTOOLCHAIN=go1.22.3 GOEXPERIMENT=boringcrypto go build -a -o /app/spark-operator/spark-operator main.go

#Install Hadoop jars
ARG HADOOP_VERSION_DEFAULT
ARG HADOOP_TMP_HOME
RUN mkdir -p ${HADOOP_TMP_HOME}
RUN curl -Ls https://archive.apache.org/dist/hadoop/common/hadoop-${HADOOP_VERSION_DEFAULT}/hadoop-${HADOOP_VERSION_DEFAULT}.tar.gz | tar -xz --strip-components 1 -C ${HADOOP_TMP_HOME}

# Prepare spark-operator image
FROM ${ECR_URL}:${SPARK_IMAGE}
WORKDIR /opt/spark-operator
USER root

ENV PATH $JAVA_HOME/bin:$PATH
ENV SPARK_HOME="/opt/spark"
ENV JAVA_HOME="/opt/jdk-11.0.21"
ENV SPARK_SUBMIT_OPTS="${SPARK_SUBMIT_OPTS} -Djavax.net.ssl.trustStorePassword=password"
ENV PATH=${PATH}:${SPARK_HOME}/bin:${JAVA_HOME}/bin:

RUN yum update -y && \
    yum install --setopt=tsflags=nodocs -y openssl && \
    yum clean all

ARG HADOOP_TMP_HOME
COPY --from=builder ${HADOOP_TMP_HOME}/share/hadoop/client/hadoop-client-runtime-*.jar ${HADOOP_TMP_HOME}/share/hadoop/client/hadoop-client-api-*.jar ${HADOOP_TMP_HOME}/share/hadoop/common/lib/slf4j-api-*.jar /opt/spark/jars/

COPY --from=builder /app/spark-operator/spark-operator /opt/spark-operator/
COPY --from=builder /app/spark-operator/hack/gencerts.sh /usr/bin/

COPY entrypoint.sh /opt/spark-operator/
RUN chmod a+x /opt/spark-operator/entrypoint.sh
ENTRYPOINT ["/opt/spark-operator/entrypoint.sh"] ARG SPARK_IMAGE=spark-3.5.1-bin-without-hadoop
ARG GOLANG_IMAGE=golang-1.21
ARG SPARK_OPERATOR_VERSION=1.3.1
ARG HADOOP_VERSION_DEFAULT=3.4.0
ARG HADOOP_TMP_HOME="/opt/hadoop"
ARG TARGETARCH=amd64

# Prepare spark-operator build
FROM ${GOLANG_IMAGE} as builder
WORKDIR /app/spark-operator

ARG SPARK_OPERATOR_VERSION
RUN curl -Ls https://github.com/kubeflow/spark-operator/archive/refs/tags/spark-operator-chart-${SPARK_OPERATOR_VERSION}.tar.gz | tar -xz --strip-components 1 -C /app/spark-operator

RUN GOTOOLCHAIN=go1.22.3 go mod download

# Build
ARG TARGETARCH
RUN CGO_ENABLED=0 GOOS=linux GOARCH=${TARGETARCH} GO111MODULE=on GOTOOLCHAIN=go1.22.3 GOEXPERIMENT=boringcrypto go build -a -o /app/spark-operator/spark-operator main.go

#Install Hadoop jars
ARG HADOOP_VERSION_DEFAULT
ARG HADOOP_TMP_HOME
RUN mkdir -p ${HADOOP_TMP_HOME}
RUN curl -Ls https://archive.apache.org/dist/hadoop/common/hadoop-${HADOOP_VERSION_DEFAULT}/hadoop-${HADOOP_VERSION_DEFAULT}.tar.gz | tar -xz --strip-components 1 -C ${HADOOP_TMP_HOME}

# Prepare spark-operator image
FROM ${ECR_URL}:${SPARK_IMAGE}
WORKDIR /opt/spark-operator
USER root

ENV PATH $JAVA_HOME/bin:$PATH
ENV SPARK_HOME="/opt/spark"
ENV JAVA_HOME="/opt/jdk-11.0.21"
ENV SPARK_SUBMIT_OPTS="${SPARK_SUBMIT_OPTS} -Djavax.net.ssl.trustStorePassword=password"
ENV PATH=${PATH}:${SPARK_HOME}/bin:${JAVA_HOME}/bin:

RUN yum update -y && \
    yum install --setopt=tsflags=nodocs -y openssl && \
    yum clean all

ARG HADOOP_TMP_HOME
COPY --from=builder ${HADOOP_TMP_HOME}/share/hadoop/client/hadoop-client-runtime-*.jar ${HADOOP_TMP_HOME}/share/hadoop/client/hadoop-client-api-*.jar ${HADOOP_TMP_HOME}/share/hadoop/common/lib/slf4j-api-*.jar /opt/spark/jars/

COPY --from=builder /app/spark-operator/spark-operator /opt/spark-operator/
COPY --from=builder /app/spark-operator/hack/gencerts.sh /usr/bin/

COPY entrypoint.sh /opt/spark-operator/
RUN chmod a+x /opt/spark-operator/entrypoint.sh
ENTRYPOINT ["/opt/spark-operator/entrypoint.sh"] Conclusion After the build, we still have several vulnerabilities in the Hadoop library hadoop-client-runtime: org.apache.avro:avro (hadoop-client-runtime-3.4.0.jar) – CVE-2023-39410 org.apache.commons:commons-compress – CVE-2024-25710, CVE-2024-26308 org.apache.avro:avro (hadoop-client-runtime-3.4.0.jar) – CVE-2023-39410 org.apache.commons:commons-compress – CVE-2024-25710, CVE-2024-26308 Since without this library we'll not be able to run spark-submit, but the rest of the huge part of the vulnerabilities is removed along with the main Hadoop libraries.

Building a Custom Docker Image for K8s Spark Operator to Fix Vulnerabilities

DOCKER

docker.com

-- employees

Since n.d.

COMPANY RANKING

DOCKER

EVERGREEN INDEX #5512

Representing this company? Click here to claim and customize this page!

Docker WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

DOCKER

docker.com

-- employees

Since n.d.

COMPANY RANKING

DOCKER

EVERGREEN INDEX #5512

RELATED COMPANIESRANK

STORIES ABOUT DOCKER

Representing this company? Click here to claim and customize this page!

Docker WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!