Securely build, share and run modern applications anywhere

Docker

docker

Since <strong>Docker</strong> is a US company, we must comply with US export control regulations. In an effort to comply with these, we now block all IP addresses that are located in Cuba, Iran, North Korea, Republic of Crimea, Sudan, and Syria. If you are not in one of these cities, countries, or regions and ...

Docker hub перестал работать в России — Разработка на vc.ru

KEY TAKEAWAYS WasmEdge functions as a stand-alone option. It replaces traditional OCI runtime systems with its own faster and lighter solution.
These containers provide faster startups and reduced resource consumption. So, the performance is faster alongside minimal overhead expenses.
The secure sandboxing of WasmEdge containers creates an improved security environment that cuts down system vulnerabilities while reducing the exposed attack area.
WasmEdge containers can run on various hardware architectures. Some of the common include x86 together with ARM and RISC-V while maintaining full portability without needing any modifications.
WasmEdge allows users to perform simplified Kubernetes deployments that work perfectly with edge computing alongside serverless applications. WasmEdge functions as a stand-alone option. It replaces traditional OCI runtime systems with its own faster and lighter solution. These containers provide faster startups and reduced resource consumption. So, the performance is faster alongside minimal overhead expenses. The secure sandboxing of WasmEdge containers creates an improved security environment that cuts down system vulnerabilities while reducing the exposed attack area. WasmEdge containers can run on various hardware architectures. Some of the common include x86 together with ARM and RISC-V while maintaining full portability without needing any modifications. WasmEdge allows users to perform simplified Kubernetes deployments that work perfectly with edge computing alongside serverless applications. INTRODUCTION The application running landscape has undergone a complete revolution through docker containers. These create isolated systems that can be easily managed. Kubernetes workloads traditionally depend on OCI runtimes, including containers and CRI-O, for executing container operations. A newer and more efficient solution known as WebAssembly (Wasm) containers with WasmEdge technology emerges as an alternative. In this guide, I will explain the practical implementation of running WebAssembly-based containers on Kubernetes through WasmEdge by avoiding traditional container runtimes. Moreover, it covers complete explanations, starting from configuration deployment to debugging tasks and performance evaluations. You will end up with complete knowledge about deploying WasmEdge to implement containers that deliver high efficiency and speed alongside architectural portability. WHY REPLACE OCI RUNTIMES WITH WEBASSEMBLY Before jumping into the setup and practical steps, let’s first understand why you might want to replace OCI runtimes with WasmEdge. PERFORMANCE GAINS WITH WASMEDGE WasmEdge provides superior performance. It should be the primary reason to choose the platform. The initialization and resource usage of traditional containers last a long time. WasmEdge exists for fast operation. These containers begin operations in milliseconds while requiring fewer system resources. So they become suitable for time-sensitive applications and devices that operate under resource limitations, such as edge systems, and IoT components. When developing an application that needs speed-based scaling according to user demand, you should consider it. WasmEdge containers can establish themselves practically instantaneously compared to how long traditional Docker containers take to begin their operation. They offer a breakthrough benefit for systems that must respond in critical periods like serverless computing and edge computing deployments. SECURITY Standard containers provide running security issues when deployed in operational systems. The main vulnerability of running your container environment exposes applications to potential attacks that could harm neighboring applications sharing the same infrastructure. Each WebAssembly workload receives an isolated sandbox environment through WasmEdge execution. A new security layer is created through WasmEdge implementation effectively. It minimizes the vulnerable points in your system. Each WasmEdge container provides a secure environment that makes it harder for malicious code to spread. Therefore, the host system remains secure. In addition, the security model proves critical for multi-tenant cloud deployments and microservice development. Since it provides necessary isolation between various applications. The implementation of WasmEdge benefits developers who aim to establish application resistance against attacks. CROSS-PLATFORM COMPATIBILITY Among the useful features of WasmEdge is its capability to work across multiple platforms. The normal deployment of containers between different platforms like x86 and ARM requires modifications either in container images/platform-specific configuration implementation. WasmEdge shows different behavior in this respect. The container engine operates without issues on multiple processor systems that include x86 ARM and RISC-V. WasmEdge operates containers on cloud servers, edge devices, and Raspberry Pi without any changes to the initial configuration. The high level of portability reduces deployment complexity. Because your application needs to function within diverse execution environments. IDEAL FOR EDGE AND SERVERLESS COMPUTING WasmEdge demonstrates exceptional performance within edge and serverless computing domains. It's compatible with handling small and quick booting containers. WasmEdge serves efficient workloads because engineers developed it specifically for lightweight, small systems that need fewer resources than traditional containers. WasmEdge works flawlessly in situations where equipment has restricted processing power, such as smart sensors, edge devices, and IoT setups. WasmEdge serves developers who construct microservices or serverless applications through its optimal features. The quick boot times of containers match their need for scarce resources while operating. Such performance attributes make WasmEdge ideal for applications that require fast responsiveness under various traffic conditions, such as event-driven workloads or stateless applications. SETTING UP KUBERNETES FOR WASMEDGE CONTAINERS This section will explain the necessary steps for WasmEdge installation on Kubernetes platforms, although the Kubernetes experience is beneficial. PREREQUISITES The following items should be checked before starting: A Kubernetes 1.26+ cluster. WasmEdge deployment requires a Kubernetes service provider.


Your cluster needs Kubectl installation with optimal configuration to access it from any location.


All Kubernetes nodes must have WasmEdge runtime running.


Container with Wasm support, and CRI-O with WasmEdge integration. A Kubernetes 1.26+ cluster. WasmEdge deployment requires a Kubernetes service provider. A Kubernetes 1.26+ cluster. WasmEdge deployment requires a Kubernetes service provider. Your cluster needs Kubectl installation with optimal configuration to access it from any location. Your cluster needs Kubectl installation with optimal configuration to access it from any location. All Kubernetes nodes must have WasmEdge runtime running. All Kubernetes nodes must have WasmEdge runtime running. Container with Wasm support, and CRI-O with WasmEdge integration. Container with Wasm support, and CRI-O with WasmEdge integration. Install the WasmEdge on Kubernetes nodes by using the command below: curl -sSf https://raw.githubusercontent.com/WasmEdge/WasmEdge/master/utils/install.sh | bash curl -sSf https://raw.githubusercontent.com/WasmEdge/WasmEdge/master/utils/install.sh | bash After installation, make sure that WasmEdge is installed rightly: wasmedge --version wasmedge --version This should return the version of WasmEdge you’ve installed. CONFIGURING KUBERNETES FOR WASMEDGE Your Kubernetes environment requires registration of WasmEdge as its runtime for containers. The process needs you to establish WasmEdge Runtime Class configuration for Kubernetes usage. WasmEdge can be used by Kubernetes instead of the default container, and CRI-O runtimes through this configuration. Create a RuntimeClass YAML file for WasmEdge: apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: wasmedge
runtimeHandler: wasmedge apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: wasmedge
runtimeHandler: wasmedge Apply this configuration using: kubectl apply -f runtimeclass-wasmedge.yaml kubectl apply -f runtimeclass-wasmedge.yaml This tells Kubernetes to use WasmEdge as a valid runtime for your containers. DEPLOYING A WEBASSEMBLY (WASM) CONTAINER FOR KUBERNETES Now, let’s discuss, how can you deploy a WebAssembly (Wasm) Container on Kubernetes: CREATING A WASMEDGE-BASED DOCKER IMAGE First, convert your application into a WebAssembly (Wasm) module. For example, try a Rust-based HTTP server and convert it into a Wasm module. Next, install the wasm-pack tool. It helps you compile Rust code into WebAssembly: cargo install wasm-pack cargo install wasm-pack Build the Rust app into a Wasm module: wasm-pack build --release --target=wasm32-wasi wasm-pack build --release --target=wasm32-wasi The Wasm module requires setup before it can be deployed to Docker Hub and ORAS container registries. Kubernetes can obtain Wasm images through this mechanism during its deployment process. Example: docker push myrepo/wasm-app:latest docker push myrepo/wasm-app:latest DEPLOYING THE WASMEDGE CONTAINER Now, you’re ready to deploy the Wasm-based container on Kubernetes. Create a deployment YAML file for Kubernetes to use: apiVersion: apps/v1
kind: Deployment
metadata:
  name: wasm-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: wasm-app
  template:
    metadata:
      labels:
        app: wasm-app
    spec:
      runtimeClassName: wasmedge
      containers:
        - name: wasm-container
          image: myrepo/wasm-app:latest
          command: ["/wasm-app.wasm"] apiVersion: apps/v1
kind: Deployment
metadata:
  name: wasm-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: wasm-app
  template:
    metadata:
      labels:
        app: wasm-app
    spec:
      runtimeClassName: wasmedge
      containers:
        - name: wasm-container
          image: myrepo/wasm-app:latest
          command: ["/wasm-app.wasm"] Now, apply this deployment file to Kubernetes: kubectl apply -f wasm-deployment.yaml kubectl apply -f wasm-deployment.yaml The WasmEdge container receives deployment through Kubernetes with two exact duplicates enabled. The deployment status can be checked by running this command: kubectl get pods -l app=wasm-app kubectl get pods -l app=wasm-app The above command provides a quick overview of the deployment status. RUNNING AND DEBUGGING WASM CONTAINERS Once deployed, it's imperative to verify that everything is working as intended. Let’s talk about some practical commands to monitor and troubleshoot your WasmEdge containers. VERIFYING THE DEPLOYMENT To check if your WasmEdge container is running, use: kubectl get pods -l app=wasm-app kubectl get pods -l app=wasm-app If everything is running in the right manner, this should show the pods associated with your deployment. To check the logs for a specific pod: kubectl logs <pod-name> kubectl logs <pod-name> TROUBLESHOOTING If you face any issues with your container, use the Kubectl describe pod to see detailed information: kubectl describe pod <pod-name> kubectl describe pod <pod-name> The command will show you the events related to the pod. It can help you debug issues such as image pull errors or misconfigurations. Additionally, WasmEdge provides a CLI tool to validate your Wasm module. To certify that your Wasm module is accurately built and ready to run, use wasmedge --validate my-wasm-app.wasm wasmedge --validate my-wasm-app.wasm This will check for any potential issues with the Wasm module before you deploy it. BENCHMARKING WASMEDGE VS. OCI CONTAINERS It’s important to compare the performance of WasmEdge containers with traditional OCI containers. Let’s look at a few performance benchmarks to see how WasmEdge performs. STARTUP TIME COMPARISON One of the most significant benefits of WasmEdge is the startup time. To evaluate the startup latency, run both WasmEdge and OCI containers using the time command: time docker run myrepo/wasm-app
time wasmedge run myrepo/wasm-app.wasm time docker run myrepo/wasm-app
time wasmedge run myrepo/wasm-app.wasm MEMORY & CPU USAGE To compare memory and CPU usage, use the kubectl top pods command: kubectl top pods kubectl top pods This will show you how much CPU and memory your containers are consuming. WasmEdge containers should consume far fewer resources than their OCI counterparts. For even more detailed profiling, use wasm-stats: wasm-stats my-wasm-app.wasm wasm-stats my-wasm-app.wasm It will help you understand how efficiently WasmEdge handles your workload. CONCLUSION WasmEdge offers users lightweight and secure container runtime functions. It replaces existing traditional OCI runtimes effectively. OCI runtime containers will become more effective when you migrate to WasmEdge because your applications will start faster and use fewer resources, while the sandbox security measures improve system safety. WasmEdge operates across multiple platforms. So, it means your containers can function on x86 devices as well as ARM and RISC-V devices without changing any configurations. The guide presents simple deployment and management instructions for WasmEdge containers on Kubernetes systems. WasmEdge proves to be an ideal solution for developers who create microservices and serverless applications as well as edge deployments.

Running Docker Containers on Kubernetes Without a Container Runtime Using WasmEdge

💡 Introduction 💡 Introduction Welcome Dev to the world of testing and Containerisation. Today, we will be diving into an exciting project, where we will be testing a Node Js application using cypress and this whole project will be inside a docker container. So without further ado, Let’s get started! 💡 Pre-requisites 💡 Pre-requisites Before we delve into the project, we need to ensure that we have following requirements met in our system: Docker and Docker-Compose: Since we are working inside a container, we will be needing docker and docker-compose for that.
Node installed: We are testing a node application, so we would be needing node to run in it.
Basic Understanding of Docker: We will be creating a dockerfile and Docker-compose file, a prior understanding of both would be appreciated. Docker and Docker-Compose: Since we are working inside a container, we will be needing docker and docker-compose for that. Docker and Docker-Compose: Node installed: We are testing a node application, so we would be needing node to run in it. Node installed: Basic Understanding of Docker: We will be creating a dockerfile and Docker-compose file, a prior understanding of both would be appreciated. Basic Understanding of Docker: 💡 Setting up the project 💡 Setting up the project We will start the project by creating a project dir named book-app-testing, inside the directory, we will be creating a docker-compose file with the following content: services:
 book-reader-app:
   image: rajudandigam/book-reader-node:v1
   container_name: book-reader-app
   ports:
     - "3000:3000" 
   networks:
     - test-network


 cypress:
   build:
     context: ./cypress-test
     dockerfile: Dockerfile
   container_name: cypress
   depends_on:
    - book-reader-app


   environment:
     - CYPRESS_baseUrl=http://book-reader-app:3000 
   volumes:
     - ./cypress-test:/e2e 
   networks:
     - test-network
   command: ["npx", "cypress", "run", "--spec", "cypress/e2e/book-app-heading.cy.js"]


networks:
 test-network:
   driver: bridge services:
 book-reader-app:
   image: rajudandigam/book-reader-node:v1
   container_name: book-reader-app
   ports:
     - "3000:3000" 
   networks:
     - test-network


 cypress:
   build:
     context: ./cypress-test
     dockerfile: Dockerfile
   container_name: cypress
   depends_on:
    - book-reader-app


   environment:
     - CYPRESS_baseUrl=http://book-reader-app:3000 
   volumes:
     - ./cypress-test:/e2e 
   networks:
     - test-network
   command: ["npx", "cypress", "run", "--spec", "cypress/e2e/book-app-heading.cy.js"]


networks:
 test-network:
   driver: bridge Now inside the project dir, we will create a new dir, named cyprus-test. We will navigate inside it and run the following commands: npm init -y
npm install cypress --save-dev
npx cypress open npm init -y
npm install cypress --save-dev
npx cypress open This will open a new Window. Here Select E2e Testing and then click on continue and start testing E2E testing in Electron. Now, Click on Create a new Spec File and in the name put cypress/e2e/book-app-heading.cy.js and click Okay and run the Spec file. cypress/e2e/book-app-heading.cy.js This will run a demo test. Now go to thebook-app-testing/cypress-test/cypress/e2e and here you will have your book-app-heading.cy.js file, remove the existing content and put the full information in it: book-app-testing/cypress-test/cypress/e2e book-app-heading.cy.js describe('Book Reader Node App', () => {
 it('should have a heading that includes "Book App"', () => {
   // Use the baseUrl from environment variable (preferred approach)
   cy.visit('/'); // This appends to CYPRESS_baseUrl (http://book-reader-app:3000)


   // Alternatively, hardcode the service name (less flexible):
   // cy.visit('http://book-reader-app:3000');


   // Check if any heading (h1, h2, etc.) contains "Book App"
   cy.get('h1, h2, h3, h4, h5, h6').should('contain.text', 'Book App');
 });
}); describe('Book Reader Node App', () => {
 it('should have a heading that includes "Book App"', () => {
   // Use the baseUrl from environment variable (preferred approach)
   cy.visit('/'); // This appends to CYPRESS_baseUrl (http://book-reader-app:3000)


   // Alternatively, hardcode the service name (less flexible):
   // cy.visit('http://book-reader-app:3000');


   // Check if any heading (h1, h2, etc.) contains "Book App"
   cy.get('h1, h2, h3, h4, h5, h6').should('contain.text', 'Book App');
 });
}); This is a simple test which will test whether the application heading has Book App in it. 💡 Creating the Dockerfile Now, we will be creating a dockerfile for Cypress with our Test in it, for that, go inside the  book-app-testing/cypress-test and create a Dockerfile and put the following content it: book-app-testing/cypress-test #Use Cypress base image #Use Cypress base image FROM cypress/included:13.0.0


WORKDIR /e2e


# Copy project files
COPY . .


# Install dependencies
RUN npm install


# Run Cypress tests
CMD ["npx", "cypress", "run"] FROM cypress/included:13.0.0


WORKDIR /e2e


# Copy project files
COPY . .


# Install dependencies
RUN npm install


# Run Cypress tests
CMD ["npx", "cypress", "run"] Now, we will navigate to our project dir and run the following command to run the docker-container: docker-compose up –build docker-compose up –build This will give you the following output and in that, we can see that the test we provided is passed. To enhance the testing process, you can add following test in the same file or create separate file for each test inside the cypress-test/cypress/e2e: cypress-test/cypress/e2e: // Render Header component Properly
describe('Book Reader Node App', () => {
 beforeEach(() => {
   cy.visit('/');
 });


 it('should render the header component', () => {
   cy.get('header').should('be.visible');
 });


 it('should render the sidebar component', () => {
   cy.get('aside').should('be.visible');
 });
}); // Render Header component Properly
describe('Book Reader Node App', () => {
 beforeEach(() => {
   cy.visit('/');
 });


 it('should render the header component', () => {
   cy.get('header').should('be.visible');
 });


 it('should render the sidebar component', () => {
   cy.get('aside').should('be.visible');
 });
}); // Check Animation Tap and Hover
describe('Book Reader Node App - Animations', () => {
 beforeEach(() => {
   cy.visit('/');
 });


 it('should scale up on hover', () => {
   cy.get('ul li').first().trigger('mouseover')
     .should('have.css', 'transform')
     .and('contain', 'matrix');
 });


 it('should scale down on tap', () => {
   cy.get('ul li').first().trigger('mousedown')
     .should('have.css', 'transform')
     .and('contain', 'matrix');
 });
}); // Check Animation Tap and Hover
describe('Book Reader Node App - Animations', () => {
 beforeEach(() => {
   cy.visit('/');
 });


 it('should scale up on hover', () => {
   cy.get('ul li').first().trigger('mouseover')
     .should('have.css', 'transform')
     .and('contain', 'matrix');
 });


 it('should scale down on tap', () => {
   cy.get('ul li').first().trigger('mousedown')
     .should('have.css', 'transform')
     .and('contain', 'matrix');
 });
}); // Check Test Responses in various Devices
describe('Book Reader Node App - Responsiveness', () => {
 beforeEach(() => {
   cy.visit('/');
 });


 const sizes = [
   { width: 1280, height: 720 },  // Desktop
   { width: 768, height: 1024 },  // Tablet
   { width: 375, height: 812 }    // Mobile
 ];


 sizes.forEach((size) => {
   it(`should display correctly on ${size.width}x${size.height}`, () => {
     cy.viewport(size.width, size.height);
     cy.get('main').should('be.visible');
   });
 });
}); // Check Test Responses in various Devices
describe('Book Reader Node App - Responsiveness', () => {
 beforeEach(() => {
   cy.visit('/');
 });


 const sizes = [
   { width: 1280, height: 720 },  // Desktop
   { width: 768, height: 1024 },  // Tablet
   { width: 375, height: 812 }    // Mobile
 ];


 sizes.forEach((size) => {
   it(`should display correctly on ${size.width}x${size.height}`, () => {
     cy.viewport(size.width, size.height);
     cy.get('main').should('be.visible');
   });
 });
}); // Test Naviation
describe('Book Reader Node App - Navigation', () => {
 beforeEach(() => {
   cy.visit('/');
 });


 it('should navigate to book details page on click', () => {
   cy.get('ul li a').first().click();
   cy.url().should('include', '/book/');
 });


 it('should display book details on the details page', () => {
   cy.get('ul li a').first().click();
   cy.get('h1').should('be.visible');
   cy.get('img').should('be.visible');
 });
}); // Test Naviation
describe('Book Reader Node App - Navigation', () => {
 beforeEach(() => {
   cy.visit('/');
 });


 it('should navigate to book details page on click', () => {
   cy.get('ul li a').first().click();
   cy.url().should('include', '/book/');
 });


 it('should display book details on the details page', () => {
   cy.get('ul li a').first().click();
   cy.get('h1').should('be.visible');
   cy.get('img').should('be.visible');
 });
}); You can also visit __http://localhost:300__0 where our application is running. http://localhost:300 💡 Conclusion 💡 Conclusion This marks the end of our blog. In today’s blog, we learnt how to use Cypress to test our Web Application built using Node. We started with a basic test to verify the heading contains "Book App" and gradually saw how we can expand our testing coverage with more complex and meaningful tests. By incorporating Cypress, we can ensure our app’s functionality, UI interactions, and navigation work as expected, making our web applications more robust and reliable.

Running Cypress Tests for Node.js Apps Inside a Docker Container

Your Docker Containers Are Probably Full of Security Holes—Here’s How to Fix Them Now

Firms increasingly make use of artificial intelligence (AI) infrastructures to host and manage autonomous workloads. Consequently, there has been significant demand for scalable as well as resilient infrastructures that will be able to meet heterogeneous application or cloud requirements. Organizations use Kubernetes and Docker to meet such needs because firms realize that both are highly effective use cases that deliver scalable AI infrastructures. Deploying AI infrastructure typically provides adequate computation power to execute and process large datasets. These demands can translate into the need for scalable methods that enable AI models to run on large workloads without hurting performance. Why Companies Need to Scale Up Their AI Infrastructure AI systems, nonetheless, are also resource-intensive, normally demanding both high computing capacity and the ability to process high levels of data. As more advanced AI applications and a larger scale become required, scalability becomes more critical. Scalability ensures that AI systems can handle increasing workloads without any loss of performance. Expanding Data Volumes The growing amount of data is a concern for AI systems in many facets. Most AI models, especially those based on deep learning, heavily depend on large amounts of data during training and inference. However, without adequate scalable infrastructure, processing and interpreting such enormous quantities of data is a roadblock. Optimized Performance Scalable AI hardware supports reliable and stable performance despite drastically overwhelming computational loads. With Kubernetes, horizontal scaling of AI jobs is a breeze, and the dynamic resizing of replica numbers can be done as a function of necessity. In contrast, Docker containers support lean, isolated environments for running AI models where resource conflict is not a performance bottleneck. Effective Resource Management Efficient use of resources is the key to cost-effective and sustainable AI deployment. Kubernetes' resource requests and limits allow for fine-grained CPU and memory resource management by avoiding underprovisioning and overprovisioning. Docker's resource management fills the gap by isolating container resources. Scaling AI Infrastructure With Kubernetes and Docker Containerization is one of the milestones in the evolution of scalable artificial intelligence infrastructure. Containerization of the AI application and its dependencies in a Docker container ensures consistency throughout the development, testing, and deployment environments. First, you must define a Dockerfile in order to install the environment. The Dockerfile is a series of instructions about how to build a Docker image. It declares a base image, the dependencies required, and the initial setup commands that apply to your app. The following is a basic Dockerfile for a Python machine-learning model: # Use an official Python runtime as a parent image
FROM python:3.9-slim
 
# Set the working directory in the container
WORKDIR /usr/src/app
 
# Copy the current directory contents into the container
COPY . .
 
# Install any needed packages specified in requirements.txt
RUN pip install --no-cache-dir -r requirements.txt
 
# Expose the port the app runs on
EXPOSE 5000
 
# Define environment variable
ENV NAME World
 
# Run the app
CMD ["python", "./app.py"] If the Dockerfile is ready, then you can build the Docker image and run the container. Run the following commands: # Build the Docker image
docker build -t ml-model:latest .
 
# Run the container
docker run -p 5000:5000 ml-model:latest Deploying the Dockerized AI Model to Kubernetes Kubernetes provides a wide range of orchestration features that enable efficient application management in the containerized infrastructure. Deployment of the Docker image on Kubernetes ensures that a specified number of application replicas is always running. The following is an example of deployment.yaml file that you can use to deploy your Dockerized machine learning model: apiVersion: apps/v1
kind: Deployment
metadata:
  name: ml-model-deployment
spec:
  replicas: 3  
  selector:
	matchLabels:
  	app: ml-model
  template:
	metadata:
  	labels:
    	app: ml-model
	spec:
  	containers:
  	- name: ml-model-container
    	image: ml-model:latest
    	ports:
    	- containerPort: 5000 The above code snippet shows how to deploy the AI model, but you also need to make the model externally accessible. You will need to expose it by defining a Kubernetes Service. The service.yaml below illustrates an example: apiVersion: v1
kind: Service
metadata:
  name: ml-model-service
spec:
  selector:
	app: ml-model
  ports:
	- protocol: TCP
  	port: 80
  	targetPort: 5000
  type: LoadBalancer Use the kubectl command-line tool to apply the deployment and service configurations: # Deploy the application
kubectl apply -f deployment.yaml
 
# Expose the service
kubectl apply -f service.yaml Scaling With Kubernetes Kubernetes provides excellent scaling capabilities to AI environments, maximizing resource utilization and performance. Horizontal scaling is done by adding additional containers, and vertical scaling involves adding additional resources like CPU or memory to a container. Horizontal Scaling Horizontal scaling is used to scale up the number of replicas (Pods) of an AI system to handle a higher workload. The process requires enabling dynamic scaling depending on the number of replicas. The command used to enable such a process is `kubectl scale`. The particular command is used to set up the deployment to function up to a maximum of five replicas: `kubectl scale --replicas=5 deployment/ml-model-deployment` The command scales up the ml-model-deployment to use five replicas of the machine-learning model container. The system dynamically provisions more Pods to meet the required number afterward. Automatic Scaling using the Horizontal Pod Autoscaler (HPA) Kubernetes facilitates auto-scaling using the Horizontal Pod Autoscaler (HPA). The HPA dynamically adjusts the number of replicas based on resource use, i.e., CPU or memory, in relation to set limits. The YAML configuration shown below is a relevant example of an HPA that dynamically scales for ml-model-deployment in response to CPU use: apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  name: ml-model-hpa
spec:
  scaleTargetRef:
	apiVersion: apps/v1
	kind: Deployment
	name: ml-model-deployment
  minReplicas: 2
  maxReplicas: 10
  targetCPUUtilizationPercentage: 50 In this setup, scaleTargetRef is used to define the Deployment to be scaled, i.e., ml-model-deployment. The minimum replica count is set using MinReplicas, while the maximum replica count is controlled using maxReplicas. In addition, the CPU utilization percentage is set using targetCPUUtilizationPercentage, i.e., to 50%. CPU utilization of more than 50% across all Pods results in scaling up the replica count to a maximum of 10 automatically. As soon as CPU utilization drops below the set percentage, Kubernetes automatically reduces the replica count in order to release resources. Vertical Scaling Horizontal scaling is mainly to cope with more traffic, whereas vertical scaling provides more resources (such as CPU or memory) to existing containers. The process is to scale up or down resource requests and limits in the Kubernetes Deployment. In order to scale up the CPU and memory limits of the ml-model-deployment, one would need to open the deployment.yaml file: apiVersion: apps/v1
kind: Deployment
metadata:
  name: ml-model-deployment
spec:
  replicas: 3
  selector:
	matchLabels:
  	app: ml-model
  template:
	metadata:
  	labels:
    	app: ml-model
	spec:
  	containers:
  	- name: ml-model-container
    	image: ml-model:latest
    	ports:
    	- containerPort: 5000
    	resources:
      	requests:
        	cpu: "1"
        	memory: "2Gi"
      	limits:
        	cpu: "2"
        	memory: "4Gi" In this updated configuration: requests specify the minimum resources required for the container.limits define the maximum resources the container can use. Firms increasingly make use of artificial intelligence (AI) infrastructures to host and manage autonomous workloads. Consequently, there has been significant demand for scalable as well as resilient infrastructures that will be able to meet heterogeneous application or cloud requirements. Organizations use Kubernetes and Docker to meet such needs because firms realize that both are highly effective use cases that deliver scalable AI infrastructures. there has been significant demand for scalable Kubernetes Docker Deploying AI infrastructure typically provides adequate computation power to execute and process large datasets. These demands can translate into the need for scalable methods that enable AI models to run on large workloads without hurting performance. Why Companies Need to Scale Up Their AI Infrastructure Why Companies Need to Scale Up Their AI Infrastructure AI systems , nonetheless, are also resource-intensive, normally demanding both high computing capacity and the ability to process high levels of data. As more advanced AI applications and a larger scale become required, scalability becomes more critical. Scalability ensures that AI systems can handle increasing workloads without any loss of performance. AI systems Expanding Data Volumes Expanding Data Volumes The growing amount of data is a concern for AI systems in many facets. Most AI models, especially those based on deep learning, heavily depend on large amounts of data during training and inference. However, without adequate scalable infrastructure, processing and interpreting such enormous quantities of data is a roadblock . enormous quantities of data is a roadblock Optimized Performance Optimized Performance Scalable AI hardware supports reliable and stable performance despite drastically overwhelming computational loads. With Kubernetes, horizontal scaling of AI jobs is a breeze, and the dynamic resizing of replica numbers can be done as a function of necessity. In contrast, Docker containers support lean, isolated environments for running AI models where resource conflict is not a performance bottleneck. Effective Resource Management Effective Resource Management Efficient use of resources is the key to cost-effective and sustainable AI deployment. Kubernetes' resource requests and limits allow for fine-grained CPU and memory resource management by avoiding underprovisioning and overprovisioning. Docker's resource management fills the gap by isolating container resources. Scaling AI Infrastructure With Kubernetes and Docker Scaling AI Infrastructure With Kubernetes and Docker Containerization is one of the milestones in the evolution of scalable artificial intelligence infrastructure. Containerization of the AI application and its dependencies in a Docker container ensures consistency throughout the development, testing, and deployment environments. First, you must define a Dockerfile in order to install the environment. The Dockerfile is a series of instructions about how to build a Docker image. It declares a base image, the dependencies required, and the initial setup commands that apply to your app. The following is a basic Dockerfile for a Python machine-learning model: # Use an official Python runtime as a parent image
FROM python:3.9-slim
 
# Set the working directory in the container
WORKDIR /usr/src/app
 
# Copy the current directory contents into the container
COPY . .
 
# Install any needed packages specified in requirements.txt
RUN pip install --no-cache-dir -r requirements.txt
 
# Expose the port the app runs on
EXPOSE 5000
 
# Define environment variable
ENV NAME World
 
# Run the app
CMD ["python", "./app.py"] If the Dockerfile is ready, then you can build the Docker image and run the container. Run the following commands: # Build the Docker image
docker build -t ml-model:latest .
 
# Run the container
docker run -p 5000:5000 ml-model:latest Deploying the Dockerized AI Model to Kubernetes Deploying the Dockerized AI Model to Kubernetes Kubernetes provides a wide range of orchestration features that enable efficient application management in the containerized infrastructure. Deployment of the Docker image on Kubernetes ensures that a specified number of application replicas is always running. The following is an example of deployment.yaml file that you can use to deploy your Dockerized machine learning model : Kubernetes deploy your Dockerized machine learning model apiVersion: apps/v1
kind: Deployment
metadata:
  name: ml-model-deployment
spec:
  replicas: 3  
  selector:
	matchLabels:
  	app: ml-model
  template:
	metadata:
  	labels:
    	app: ml-model
	spec:
  	containers:
  	- name: ml-model-container
    	image: ml-model:latest
    	ports:
    	- containerPort: 5000 The above code snippet shows how to deploy the AI model, but you also need to make the model externally accessible. You will need to expose it by defining a Kubernetes Service. The service.yaml below illustrates an example: apiVersion: v1
kind: Service
metadata:
  name: ml-model-service
spec:
  selector:
	app: ml-model
  ports:
	- protocol: TCP
  	port: 80
  	targetPort: 5000
  type: LoadBalancer Use the kubectl command-line tool to apply the deployment and service configurations: # Deploy the application
kubectl apply -f deployment.yaml
 
# Expose the service
kubectl apply -f service.yaml Scaling With Kubernetes Scaling With Kubernetes Kubernetes provides excellent scaling capabilities to AI environments, maximizing resource utilization and performance. Horizontal scaling is done by adding additional containers, and vertical scaling involves adding additional resources like CPU or memory to a container. Horizontal Scaling Horizontal Scaling Horizontal scaling is used to scale up the number of replicas (Pods) of an AI system to handle a higher workload. The process requires enabling dynamic scaling depending on the number of replicas. The command used to enable such a process is `kubectl scale`. The particular command is used to set up the deployment to function up to a maximum of five replicas: `kubectl scale --replicas=5 deployment/ml-model-deployment` The command scales up the ml-model-deployment to use five replicas of the machine-learning model container. The system dynamically provisions more Pods to meet the required number afterward. Automatic Scaling using the Horizontal Pod Autoscaler (HPA) Automatic Scaling using the Horizontal Pod Autoscaler (HPA) Kubernetes facilitates auto-scaling using the Horizontal Pod Autoscaler (HPA). The HPA dynamically adjusts the number of replicas based on resource use, i.e., CPU or memory, in relation to set limits. The YAML configuration shown below is a relevant example of an HPA that dynamically scales for ml-model-deployment in response to CPU use: apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  name: ml-model-hpa
spec:
  scaleTargetRef:
	apiVersion: apps/v1
	kind: Deployment
	name: ml-model-deployment
  minReplicas: 2
  maxReplicas: 10
  targetCPUUtilizationPercentage: 50 In this setup, scaleTargetRef is used to define the Deployment to be scaled, i.e., ml-model-deployment. The minimum replica count is set using MinReplicas, while the maximum replica count is controlled using maxReplicas. In addition, the CPU utilization percentage is set using targetCPUUtilizationPercentage, i.e., to 50%. CPU utilization of more than 50% across all Pods results in scaling up the replica count to a maximum of 10 automatically. As soon as CPU utilization drops below the set percentage, Kubernetes automatically reduces the replica count in order to release resources. Vertical Scaling Vertical Scaling Horizontal scaling is mainly to cope with more traffic, whereas vertical scaling provides more resources (such as CPU or memory) to existing containers. The process is to scale up or down resource requests and limits in the Kubernetes Deployment. In order to scale up the CPU and memory limits of the ml-model-deployment, one would need to open the deployment.yaml file: apiVersion: apps/v1
kind: Deployment
metadata:
  name: ml-model-deployment
spec:
  replicas: 3
  selector:
	matchLabels:
  	app: ml-model
  template:
	metadata:
  	labels:
    	app: ml-model
	spec:
  	containers:
  	- name: ml-model-container
    	image: ml-model:latest
    	ports:
    	- containerPort: 5000
    	resources:
      	requests:
        	cpu: "1"
        	memory: "2Gi"
      	limits:
        	cpu: "2"
        	memory: "4Gi" In this updated configuration: requests specify the minimum resources required for the container. limits define the maximum resources the container can use. requests specify the minimum resources required for the container. limits define the maximum resources the container can use.

How to Scale AI Infrastructure With Kubernetes and Docker

Com escalar la infraestructura d'IA amb Kubernetes i Docker

Πώς να κλιμακώσετε την υποδομή AI με το Kubernetes και το Docker

Cómo escalar la infraestructura de IA con Kubernetes y Docker

چگونه زیرساخت هوش مصنوعی را با Kubernetes و Docker مقیاس بندی کنیم

Kako skalirati AI infrastrukturu s Kubernetesom i Dockerom

Kubernetes と Docker で AI インフラストラクチャを拡張する方法

Kaip padidinti AI infrastruktūrą naudojant „Kubernetes“ ir „Docker“.

Kubernetes සහ Docker සමඟ AI යටිතල පහසුකම් පරිමාණය කරන්නේ කෙසේද?

Hur man skalar AI-infrastruktur med Kubernetes och Docker

Paano I-scale ang AI Infrastructure Gamit ang Kubernetes at Docker

کبرنیٹس اور ڈوکر کے ساتھ اے آئی انفراسٹرکچر کی پیمائش کیسے کریں۔

Cách mở rộng cơ sở hạ tầng AI với Kubernetes và Docker

Optimizing Docker Images is More Than Just a One and Done Thing

Optimitzar les imatges de Docker és més que una cosa feta

Optimizar las imágenes de Docker es más que una tarea que se realiza una sola vez

بهینه سازی تصاویر Docker چیزی فراتر از یک کار انجام شده است

Docker イメージの最適化は一度で終わるものではありません

Ny fanatsarana ny sary Docker dia mihoatra noho ny zavatra iray sy vita

Докерын зургуудыг оновчтой болгох нь зөвхөн нэг бөгөөд хийсэн зүйл биш юм

डकर छविहरू अप्टिमाइज गर्नु केवल एक र सम्पन्न चीज भन्दा बढी हो

Otimizar imagens do Docker é mais do que algo feito uma vez e pronto

Docker Imahinakuna allinchayqa Aswan Hukllamanta chaymanta Rurasqamanta

Hagaajinta Sawirrada Docker waa in ka badan hal iyo wax la sameeyay

Docker suratlaryny optimizirlemek diňe bir we ýerine ýetirilen zat däl

Ang Pag-optimize sa Mga Imahe ng Docker ay Higit pa sa Isa at Tapos na Bagay

This is the third part of the series Container Networking. I will explain a little bit about the docker container networking in this blog post. I followed the steps mentioned in this to install docker on Ubuntu. Post docker install, you can see the docker0 device in the list ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 02:fd:4d:34:55:76 brd ff:ff:ff:ff:ff:ff
11: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default 
    link/ether 02:42:cf:b4:c1:a8 brd ff:ff:ff:ff:ff:ff Let’s create a busybox container. docker run --name bb -dt busybox docker ps -a
CONTAINER ID   IMAGE     COMMAND   CREATED              STATUS              PORTS    NAMES
02429964e449   busybox   "sh"      About a minute ago   Up About a minute             bb In the device list, you see the veth7b01920@if16 interface is created with the master as docker0. ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 02:fd:4d:34:55:76 brd ff:ff:ff:ff:ff:ff
11: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether 02:42:cf:b4:c1:a8 brd ff:ff:ff:ff:ff:ff
17: veth7b01920@if16: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP mode DEFAULT group default 
    link/ether e2:15:50:88:11:03 brd ff:ff:ff:ff:ff:ff link-netnsid 2 Let’s check the network namespace. ip netns list And you won’t see any network namespaces…How come??? This is because: ip netns list command looks up network namespaces file in the 
/var/run/netns directory.

However, the Docker daemon doesn’t create a reference of
the network namespace file in the /var/run/netns directory
after the creation. Therefore, ip netns ls cannot resolve the
network namespace file

Ref: https://www.baeldung.com/linux/docker-network-namespace-invisible If you want ip netns list to show the namespace name docker has created, then follow the below steps export container_name=bb
container_pid=$(sudo docker inspect -f '' $container_name)
echo $container_pid

sudo touch /var/run/netns/$container_name
sudo mount -o bind /proc/$container_pid/ns/net /var/run/netns/$container_name Now, you can see the namespace name. ip netns list
bb (id: 0) sudo ip netns exec bb ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
16: eth0@if17: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0 The following diagram will help you visualize it better. With this, I’m signing off. I hope this blog series helps to de-clutter container networking. This is the third part of the series Container Networking . I will explain a little bit about the docker container networking in this blog post. Container Networking I followed the steps mentioned in this to install docker on Ubuntu. this Post docker install, you can see the docker0 device in the list docker0 ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 02:fd:4d:34:55:76 brd ff:ff:ff:ff:ff:ff
11: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default 
    link/ether 02:42:cf:b4:c1:a8 brd ff:ff:ff:ff:ff:ff ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 02:fd:4d:34:55:76 brd ff:ff:ff:ff:ff:ff
11: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default 
    link/ether 02:42:cf:b4:c1:a8 brd ff:ff:ff:ff:ff:ff Let’s create a busybox container. docker run --name bb -dt busybox docker run --name bb -dt busybox docker ps -a
CONTAINER ID   IMAGE     COMMAND   CREATED              STATUS              PORTS    NAMES
02429964e449   busybox   "sh"      About a minute ago   Up About a minute             bb docker ps -a
CONTAINER ID   IMAGE     COMMAND   CREATED              STATUS              PORTS    NAMES
02429964e449   busybox   "sh"      About a minute ago   Up About a minute             bb In the device list, you see the veth7b01920@if16 interface is created with the master as docker0. veth7b01920@if16 ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 02:fd:4d:34:55:76 brd ff:ff:ff:ff:ff:ff
11: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether 02:42:cf:b4:c1:a8 brd ff:ff:ff:ff:ff:ff
17: veth7b01920@if16: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP mode DEFAULT group default 
    link/ether e2:15:50:88:11:03 brd ff:ff:ff:ff:ff:ff link-netnsid 2 ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 02:fd:4d:34:55:76 brd ff:ff:ff:ff:ff:ff
11: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether 02:42:cf:b4:c1:a8 brd ff:ff:ff:ff:ff:ff
17: veth7b01920@if16: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP mode DEFAULT group default 
    link/ether e2:15:50:88:11:03 brd ff:ff:ff:ff:ff:ff link-netnsid 2 Let’s check the network namespace. ip netns list ip netns list And you won’t see any network namespaces…How come??? This is because: ip netns list command looks up network namespaces file in the 
/var/run/netns directory.

However, the Docker daemon doesn’t create a reference of
the network namespace file in the /var/run/netns directory
after the creation. Therefore, ip netns ls cannot resolve the
network namespace file

Ref: https://www.baeldung.com/linux/docker-network-namespace-invisible ip netns list command looks up network namespaces file in the 
/var/run/netns directory.

However, the Docker daemon doesn’t create a reference of
the network namespace file in the /var/run/netns directory
after the creation. Therefore, ip netns ls cannot resolve the
network namespace file

Ref: https://www.baeldung.com/linux/docker-network-namespace-invisible If you want ip netns list to show the namespace name docker has created, then follow the below steps ip netns list export container_name=bb
container_pid=$(sudo docker inspect -f '' $container_name)
echo $container_pid

sudo touch /var/run/netns/$container_name
sudo mount -o bind /proc/$container_pid/ns/net /var/run/netns/$container_name export container_name=bb
container_pid=$(sudo docker inspect -f '' $container_name)
echo $container_pid

sudo touch /var/run/netns/$container_name
sudo mount -o bind /proc/$container_pid/ns/net /var/run/netns/$container_name Now, you can see the namespace name. ip netns list
bb (id: 0) ip netns list
bb (id: 0) sudo ip netns exec bb ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
16: eth0@if17: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0 sudo ip netns exec bb ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
16: eth0@if17: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0 The following diagram will help you visualize it better. With this, I’m signing off. I hope this blog series helps to de-clutter container networking.

A Guide to Container Networking: Part III - Docker Container Networking

When building Python applications with Poetry in a Docker container, we sometimes encounter issues accessing private packages stored in the Google Artifact Registry (GAR). Locally, this challenge arises because docker build cannot directly handle the Google Cloud credentials in the same way as our CI/CD pipeline, where we leverage service accounts and the Kaniko action for secure builds. To resolve this for local development, we'll configure Google Cloud credentials within Docker using Docker Compose, enabling secure access to the GAR repository during the build process. Non-Containerized Pre-requirements Python ^3.12
Poetry ^1.7.1
Google Cloud SDK ^489.0.0
Keyring ^24.0.0
keyrings.google-artifactregistry-auth ^1.0.0
Read access to your GAR Python repository
Logged into GCP (gcloud auth application-default login) Setup Keyring Setup
Install keyring support for Google Artifact Registry:
pip install keyring
pip install keyrings.google-artifactregistry-auth Connect Poetry to your GAR repo
Access to the private repository in the Google Artifact Registry can be managed through Poetry. First, configure a custom source in Poetry for the GAR repository by running:
poetry source add --priority=explicit <PACKAGE_NAME> https://<REGION>-python.pkg.dev/<PROJECT>/<REGISTRY>/simple Be sure to append /simple to the repository URL for compatibility. Now you can install packages from your private repo: poetry add --source <PACKAGE_NAME> some-package@1.0.0 Notes In some cases, accessing the repository may require setting an explicit OAuth token for authentication in Poetry. Use the following command to configure this globally in Poetry: poetry config http-basic.<PACKAGE_NAME> oauth2accesstoken $(gcloud auth print-access-token) Containerized Pre-requirements Docker ^20.10
Docker Compose ^2.0
Google Cloud SDK ^489.0.0
Read access to your GAR Python repository
Logged into GCP (gcloud auth application-default login) Setup Secrets Configuration
First, define a secret in docker-compose.yaml using the local path to your credentials file:
secrets:
  gcloud_credentials:
    file: ~/.config/gcloud/application_default_credentials.json



We define gcloud_credentials.file as a consistent path for Unix-like environments in docker-compose.yaml:


This configuration securely passes the credentials file from your local machine to the build context without exposing sensitive data.




Dockerfile Adjustments
In the Dockerfile, we handle credentials with the following setup:
ARG GOOGLE_APPLICATION_CREDENTIALS
ENV GOOGLE_APPLICATION_CREDENTIALS=${GOOGLE_APPLICATION_CREDENTIALS}

RUN --mount=type=secret,id=gcloud_credentials \
    mkdir -p $(dirname ${GOOGLE_APPLICATION_CREDENTIALS}) && \
    cp /run/secrets/gcloud_credentials ${GOOGLE_APPLICATION_CREDENTIALS}



--mount=type=secret,id=gcloud_credentials: Securely mounts the credentials during the build process.


GOOGLE_APPLICATION_CREDENTIALS: Specifies the credential file's path within the container.




Service Configurations
Here, gcloud_credentials is the secret mounted at build time, as specified in the secrets configuration.
some-service:
  build:
    context: .
    dockerfile: ./Dockerfile.local
    args:
      GOOGLE_APPLICATION_CREDENTIALS: /tmp/application_default_credentials.json
    secrets:
      - gcloud_credentials Usage Just run: docker-compose up --build Conclusion This approach allows local Docker builds to access private GAR resources securely, ensuring that credentials are handled appropriately and remain protected. When building Python applications with Poetry in a Docker container, we sometimes encounter issues accessing private packages stored in the Google Artifact Registry (GAR). Locally, this challenge arises because docker build cannot directly handle the Google Cloud credentials in the same way as our CI/CD pipeline, where we leverage service accounts and the Kaniko action for secure builds. To resolve this for local development, we'll configure Google Cloud credentials within Docker using Docker Compose, enabling secure access to the GAR repository during the build process. docker build Docker Non-Containerized Pre-requirements Python ^3.12 Poetry ^1.7.1 Google Cloud SDK ^489.0.0 Keyring ^24.0.0 keyrings.google-artifactregistry-auth ^1.0.0 Read access to your GAR Python repository Logged into GCP (gcloud auth application-default login) Python ^3.12 Poetry ^1.7.1 Google Cloud SDK ^489.0.0 Keyring ^24.0.0 keyrings.google-artifactregistry-auth ^1.0.0 Read access to your GAR Python repository Logged into GCP ( gcloud auth application-default login ) gcloud auth application-default login Setup Keyring Setup
Install keyring support for Google Artifact Registry:
pip install keyring
pip install keyrings.google-artifactregistry-auth Keyring Setup Install keyring support for Google Artifact Registry: pip install keyring
pip install keyrings.google-artifactregistry-auth Keyring Setup Keyring Setup Install keyring support for Google Artifact Registry: keyring pip install keyring
pip install keyrings.google-artifactregistry-auth pip install keyring
pip install keyrings.google-artifactregistry-auth Connect Poetry to your GAR repo
Access to the private repository in the Google Artifact Registry can be managed through Poetry. First, configure a custom source in Poetry for the GAR repository by running:
poetry source add --priority=explicit <PACKAGE_NAME> https://<REGION>-python.pkg.dev/<PROJECT>/<REGISTRY>/simple Connect Poetry to your GAR repo Access to the private repository in the Google Artifact Registry can be managed through Poetry. First, configure a custom source in Poetry for the GAR repository by running: poetry source add --priority=explicit <PACKAGE_NAME> https://<REGION>-python.pkg.dev/<PROJECT>/<REGISTRY>/simple Connect Poetry to your GAR repo Connect Poetry to your GAR repo Access to the private repository in the Google Artifact Registry can be managed through Poetry. First, configure a custom source in Poetry for the GAR repository by running: poetry source add --priority=explicit <PACKAGE_NAME> https://<REGION>-python.pkg.dev/<PROJECT>/<REGISTRY>/simple poetry source add --priority=explicit <PACKAGE_NAME> https://<REGION>-python.pkg.dev/<PROJECT>/<REGISTRY>/simple Be sure to append /simple to the repository URL for compatibility. Be sure to append /simple to the repository URL for compatibility. /simple Now you can install packages from your private repo: poetry add --source <PACKAGE_NAME> some-package@1.0.0 poetry add --source <PACKAGE_NAME> some-package@1.0.0 Notes In some cases, accessing the repository may require setting an explicit OAuth token for authentication in Poetry. Use the following command to configure this globally in Poetry: poetry config http-basic.<PACKAGE_NAME> oauth2accesstoken $(gcloud auth print-access-token) poetry config http-basic.<PACKAGE_NAME> oauth2accesstoken $(gcloud auth print-access-token) Containerized Pre-requirements Docker ^20.10 Docker Compose ^2.0 Google Cloud SDK ^489.0.0 Read access to your GAR Python repository Logged into GCP (gcloud auth application-default login) Docker ^20.10 Docker Compose ^2.0 Google Cloud SDK ^489.0.0 Read access to your GAR Python repository Logged into GCP ( gcloud auth application-default login ) gcloud auth application-default login Setup Secrets Configuration
First, define a secret in docker-compose.yaml using the local path to your credentials file:
secrets:
  gcloud_credentials:
    file: ~/.config/gcloud/application_default_credentials.json



We define gcloud_credentials.file as a consistent path for Unix-like environments in docker-compose.yaml:


This configuration securely passes the credentials file from your local machine to the build context without exposing sensitive data. Dockerfile Adjustments
In the Dockerfile, we handle credentials with the following setup:
ARG GOOGLE_APPLICATION_CREDENTIALS
ENV GOOGLE_APPLICATION_CREDENTIALS=${GOOGLE_APPLICATION_CREDENTIALS}

RUN --mount=type=secret,id=gcloud_credentials \
    mkdir -p $(dirname ${GOOGLE_APPLICATION_CREDENTIALS}) && \
    cp /run/secrets/gcloud_credentials ${GOOGLE_APPLICATION_CREDENTIALS}



--mount=type=secret,id=gcloud_credentials: Securely mounts the credentials during the build process.


GOOGLE_APPLICATION_CREDENTIALS: Specifies the credential file's path within the container. Service Configurations
Here, gcloud_credentials is the secret mounted at build time, as specified in the secrets configuration.
some-service:
  build:
    context: .
    dockerfile: ./Dockerfile.local
    args:
      GOOGLE_APPLICATION_CREDENTIALS: /tmp/application_default_credentials.json
    secrets:
      - gcloud_credentials Secrets Configuration First, define a secret in docker-compose.yaml using the local path to your credentials file: secrets:
  gcloud_credentials:
    file: ~/.config/gcloud/application_default_credentials.json We define gcloud_credentials.file as a consistent path for Unix-like environments in docker-compose.yaml:


This configuration securely passes the credentials file from your local machine to the build context without exposing sensitive data. Secrets Configuration Secrets Configuration First, define a secret in docker-compose.yaml using the local path to your credentials file: docker-compose.yaml secrets:
  gcloud_credentials:
    file: ~/.config/gcloud/application_default_credentials.json secrets:
  gcloud_credentials:
    file: ~/.config/gcloud/application_default_credentials.json We define gcloud_credentials.file as a consistent path for Unix-like environments in docker-compose.yaml: This configuration securely passes the credentials file from your local machine to the build context without exposing sensitive data. We define gcloud_credentials.file as a consistent path for Unix-like environments in docker-compose.yaml: We define gcloud_credentials.file as a consistent path for Unix-like environments in docker-compose.yaml : gcloud_credentials.file docker-compose.yaml This configuration securely passes the credentials file from your local machine to the build context without exposing sensitive data. This configuration securely passes the credentials file from your local machine to the build context without exposing sensitive data. Dockerfile Adjustments In the Dockerfile, we handle credentials with the following setup: ARG GOOGLE_APPLICATION_CREDENTIALS
ENV GOOGLE_APPLICATION_CREDENTIALS=${GOOGLE_APPLICATION_CREDENTIALS}

RUN --mount=type=secret,id=gcloud_credentials \
    mkdir -p $(dirname ${GOOGLE_APPLICATION_CREDENTIALS}) && \
    cp /run/secrets/gcloud_credentials ${GOOGLE_APPLICATION_CREDENTIALS} --mount=type=secret,id=gcloud_credentials: Securely mounts the credentials during the build process.


GOOGLE_APPLICATION_CREDENTIALS: Specifies the credential file's path within the container. Dockerfile Adjustments Dockerfile Adjustments In the Dockerfile, we handle credentials with the following setup: ARG GOOGLE_APPLICATION_CREDENTIALS
ENV GOOGLE_APPLICATION_CREDENTIALS=${GOOGLE_APPLICATION_CREDENTIALS}

RUN --mount=type=secret,id=gcloud_credentials \
    mkdir -p $(dirname ${GOOGLE_APPLICATION_CREDENTIALS}) && \
    cp /run/secrets/gcloud_credentials ${GOOGLE_APPLICATION_CREDENTIALS} ARG GOOGLE_APPLICATION_CREDENTIALS
ENV GOOGLE_APPLICATION_CREDENTIALS=${GOOGLE_APPLICATION_CREDENTIALS}

RUN --mount=type=secret,id=gcloud_credentials \
    mkdir -p $(dirname ${GOOGLE_APPLICATION_CREDENTIALS}) && \
    cp /run/secrets/gcloud_credentials ${GOOGLE_APPLICATION_CREDENTIALS} --mount=type=secret,id=gcloud_credentials: Securely mounts the credentials during the build process. GOOGLE_APPLICATION_CREDENTIALS: Specifies the credential file's path within the container. --mount=type=secret,id=gcloud_credentials: Securely mounts the credentials during the build process. --mount=type=secret,id=gcloud_credentials : Securely mounts the credentials during the build process. --mount=type=secret,id=gcloud_credentials GOOGLE_APPLICATION_CREDENTIALS: Specifies the credential file's path within the container. GOOGLE_APPLICATION_CREDENTIALS : Specifies the credential file's path within the container. GOOGLE_APPLICATION_CREDENTIALS Service Configurations Here, gcloud_credentials is the secret mounted at build time, as specified in the secrets configuration. some-service:
  build:
    context: .
    dockerfile: ./Dockerfile.local
    args:
      GOOGLE_APPLICATION_CREDENTIALS: /tmp/application_default_credentials.json
    secrets:
      - gcloud_credentials Service Configurations Service Configurations Here, gcloud_credentials is the secret mounted at build time, as specified in the secrets configuration. gcloud_credentials some-service:
  build:
    context: .
    dockerfile: ./Dockerfile.local
    args:
      GOOGLE_APPLICATION_CREDENTIALS: /tmp/application_default_credentials.json
    secrets:
      - gcloud_credentials some-service:
  build:
    context: .
    dockerfile: ./Dockerfile.local
    args:
      GOOGLE_APPLICATION_CREDENTIALS: /tmp/application_default_credentials.json
    secrets:
      - gcloud_credentials Usage Just run: docker-compose up --build docker-compose up --build Conclusion This approach allows local Docker builds to access private GAR resources securely, ensuring that credentials are handled appropriately and remain protected. Docker

Accessing Private Google Artifact Registry with Poetry: Local and Docker Setup

How to Use Docker to Template Local Database

Are you here because of the error message above when building the Dockerfile? You're not alone. This is a common issue that many developers encounter. Understanding the Issue Dockerfile keywords can be written in uppercase or lowercase, but it’s best not to mix them for clarity and consistency. The error message "WARN: FromAsCasing: 'as' and 'FROM' keywords' casing do not match" specifically points to a mismatch in the casing of the 'FROM' and 'AS' keywords. This rule highlights instances where mixed case is used in a FROM instruction that includes the AS keyword to specify a stage name. For example, FROM ubuntu as a builder would trigger this warning because 'FROM' is in uppercase while 'as' is in lowercase. Why Casing Matters Consistency in casing improves readability and reduces the potential for errors. When working in teams or on open-source projects, maintaining a standard practice for casing can prevent misunderstandings and ensure that scripts and Dockerfiles are more predictable and easier to manage. In Docker, the 'FROM' keyword is used to specify the base image, and the 'AS' keyword is used to name the build stage in multi-stage builds. The inconsistency in casing can lead to warnings and, in some cases, errors that can disrupt your build process. Recent Changes in Docker Some recent versions of Docker enforce stricter guidelines for keyword casing. This means that Docker now expects the 'FROM' and 'AS' keywords to match in casing, typically both being in uppercase. This enforcement helps maintain a standard across Dockerfiles and aids in avoiding potential issues that arise from mixed casing. Quick Fix To resolve this issue, you need to ensure that both 'FROM' and 'AS' keywords are in the same case. The recommended approach is to use uppercase for both, as shown below: # Use the Ubuntu image and name the stage "builder"
FROM ubuntu:latest AS builder This will prevent the warning from appearing during your build process. Additional Tips Consistency Across the Dockerfile: Ensure that all Dockerfile instructions follow the same casing convention. This includes other keywords like RUN, CMD, COPY, and ENV.
Linting Tools: Use Dockerfile linting tools to automatically check for issues related to casing and other best practices. These tools can be integrated into your CI/CD pipeline to catch errors early.
Documentation and Team Standards: Document the casing standards in your project's guidelines and ensure that all team members are aware of and follow these standards.
Updating Existing Dockerfiles: If you have existing Dockerfiles with mixed casing, take the time to update them. This can prevent future warnings and ensure that your Dockerfiles are compliant with the latest Docker versions. Consistency is key, and following these guidelines will help maintain the quality and reliability of your Docker projects. Are you here because of the error message above when building the Dockerfile? You're not alone. This is a common issue that many developers encounter. Understanding the Issue Understanding the Issue Dockerfile keywords can be written in uppercase or lowercase, but it’s best not to mix them for clarity and consistency. The error message "WARN: FromAsCasing: 'as' and 'FROM' keywords' casing do not match" specifically points to a mismatch in the casing of the 'FROM' and 'AS' keywords. This rule highlights instances where mixed case is used in a FROM instruction that includes the AS keyword to specify a stage name. For example, FROM ubuntu as a builder would trigger this warning because 'FROM' is in uppercase while 'as' is in lowercase. FROM ubuntu as a builder Why Casing Matters Why Casing Matters Consistency in casing improves readability and reduces the potential for errors. When working in teams or on open-source projects, maintaining a standard practice for casing can prevent misunderstandings and ensure that scripts and Dockerfiles are more predictable and easier to manage. In Docker, the 'FROM' keyword is used to specify the base image, and the 'AS' keyword is used to name the build stage in multi-stage builds. The inconsistency in casing can lead to warnings and, in some cases, errors that can disrupt your build process. Recent Changes in Docker Some recent versions of Docker enforce stricter guidelines for keyword casing. This means that Docker now expects the 'FROM' and 'AS' keywords to match in casing, typically both being in uppercase. This enforcement helps maintain a standard across Dockerfiles and aids in avoiding potential issues that arise from mixed casing. Quick Fix To resolve this issue, you need to ensure that both 'FROM' and 'AS' keywords are in the same case. The recommended approach is to use uppercase for both, as shown below: # Use the Ubuntu image and name the stage "builder"
FROM ubuntu:latest AS builder # Use the Ubuntu image and name the stage "builder"
FROM ubuntu:latest AS builder This will prevent the warning from appearing during your build process. Additional Tips Consistency Across the Dockerfile: Ensure that all Dockerfile instructions follow the same casing convention. This includes other keywords like RUN, CMD, COPY, and ENV. Linting Tools: Use Dockerfile linting tools to automatically check for issues related to casing and other best practices. These tools can be integrated into your CI/CD pipeline to catch errors early. Documentation and Team Standards: Document the casing standards in your project's guidelines and ensure that all team members are aware of and follow these standards. Updating Existing Dockerfiles: If you have existing Dockerfiles with mixed casing, take the time to update them. This can prevent future warnings and ensure that your Dockerfiles are compliant with the latest Docker versions. Consistency Across the Dockerfile : Ensure that all Dockerfile instructions follow the same casing convention. This includes other keywords like RUN , CMD , COPY , and ENV . Consistency Across the Dockerfile RUN CMD COPY ENV Linting Tools : Use Dockerfile linting tools to automatically check for issues related to casing and other best practices. These tools can be integrated into your CI/CD pipeline to catch errors early. Linting Tools CI/CD Documentation and Team Standards : Document the casing standards in your project's guidelines and ensure that all team members are aware of and follow these standards. Documentation and Team Standards Updating Existing Dockerfiles : If you have existing Dockerfiles with mixed casing, take the time to update them. This can prevent future warnings and ensure that your Dockerfiles are compliant with the latest Docker versions. Updating Existing Dockerfiles Consistency is key, and following these guidelines will help maintain the quality and reliability of your Docker projects.

Fix Dockerfile Error: WARN: FromAsCasing: 'as' and 'FROM' Keywords' Casing Do Not match

There is a requirement to use Spark Operator in a K8s cluster to run a spark job. The official image contains many vulnerabilities, including those due to Hadoop libraries. Let's build our own Spark Operator image. To build our image, we'll need a spark image as a base image and a Golang image to build Spark Operator itself. Spark image Building a Spark image without Hadoop using a specific version of Spark RUN curl -L https://dlcdn.apache.org/spark/spark-3.5.1/spark-3.5.1-bin-without-hadoop.tgz -o spark-3.5.1-bin-without-hadoop.tgz \
    && tar -xvzf spark-3.5.1-bin-without-hadoop.tgz \
    && mv spark-3.5.1-bin-without-hadoop /opt/spark \
    && rm spark-3.5.1-bin-without-hadoop.tgz Spark-operator image We build the Spark Operator image, we will need several Hadoop libraries to run submit commands. For example, the FIPS version build is given, the differences in the build and run commands. For building on Go, the parameter GOEXPERIMENT=boringcrypto is used For running spark-submit, the java parameter for Bouncy Castle is used Djavax.net.ssl.trustStorePassword=password You can build an image without FIPS changes. To run spark-submit, we will add Hadoop libraries during the build process: hadoop-client-runtime
hadoop-client-api
slf4j-api entrypoint.sh is used from the official Kubeflow repository https://github.com/kubeflow/spark-operator/blob/master/entrypoint.sh Example Dockerfile for building Spark Operator ARG SPARK_IMAGE=spark-3.5.1-bin-without-hadoop
ARG GOLANG_IMAGE=golang-1.21
ARG SPARK_OPERATOR_VERSION=1.3.1
ARG HADOOP_VERSION_DEFAULT=3.4.0
ARG HADOOP_TMP_HOME="/opt/hadoop"
ARG TARGETARCH=amd64

# Prepare spark-operator build
FROM ${GOLANG_IMAGE} as builder
WORKDIR /app/spark-operator

ARG SPARK_OPERATOR_VERSION
RUN curl -Ls https://github.com/kubeflow/spark-operator/archive/refs/tags/spark-operator-chart-${SPARK_OPERATOR_VERSION}.tar.gz | tar -xz --strip-components 1 -C /app/spark-operator

RUN GOTOOLCHAIN=go1.22.3 go mod download

# Build
ARG TARGETARCH
RUN CGO_ENABLED=0 GOOS=linux GOARCH=${TARGETARCH} GO111MODULE=on GOTOOLCHAIN=go1.22.3 GOEXPERIMENT=boringcrypto go build -a -o /app/spark-operator/spark-operator main.go

#Install Hadoop jars
ARG HADOOP_VERSION_DEFAULT
ARG HADOOP_TMP_HOME
RUN mkdir -p ${HADOOP_TMP_HOME}
RUN curl -Ls https://archive.apache.org/dist/hadoop/common/hadoop-${HADOOP_VERSION_DEFAULT}/hadoop-${HADOOP_VERSION_DEFAULT}.tar.gz | tar -xz --strip-components 1 -C ${HADOOP_TMP_HOME}

# Prepare spark-operator image
FROM ${ECR_URL}:${SPARK_IMAGE}
WORKDIR /opt/spark-operator
USER root

ENV PATH $JAVA_HOME/bin:$PATH
ENV SPARK_HOME="/opt/spark"
ENV JAVA_HOME="/opt/jdk-11.0.21"
ENV SPARK_SUBMIT_OPTS="${SPARK_SUBMIT_OPTS} -Djavax.net.ssl.trustStorePassword=password"
ENV PATH=${PATH}:${SPARK_HOME}/bin:${JAVA_HOME}/bin:

RUN yum update -y && \
    yum install --setopt=tsflags=nodocs -y openssl && \
    yum clean all

ARG HADOOP_TMP_HOME
COPY --from=builder ${HADOOP_TMP_HOME}/share/hadoop/client/hadoop-client-runtime-*.jar ${HADOOP_TMP_HOME}/share/hadoop/client/hadoop-client-api-*.jar ${HADOOP_TMP_HOME}/share/hadoop/common/lib/slf4j-api-*.jar /opt/spark/jars/

COPY --from=builder /app/spark-operator/spark-operator /opt/spark-operator/
COPY --from=builder /app/spark-operator/hack/gencerts.sh /usr/bin/

COPY entrypoint.sh /opt/spark-operator/
RUN chmod a+x /opt/spark-operator/entrypoint.sh
ENTRYPOINT ["/opt/spark-operator/entrypoint.sh"] Conclusion After the build, we still have several vulnerabilities in the Hadoop library hadoop-client-runtime: org.apache.avro:avro (hadoop-client-runtime-3.4.0.jar) – CVE-2023-39410
org.apache.commons:commons-compress – CVE-2024-25710, CVE-2024-26308 Since without this library we'll not be able to run spark-submit, but the rest of the huge part of the vulnerabilities is removed along with the main Hadoop libraries. There is a requirement to use Spark Operator in a K8s cluster to run a spark job. The official image contains many vulnerabilities, including those due to Hadoop libraries. Let's build our own Spark Operator image. To build our image, we'll need a spark image as a base image and a Golang image to build Spark Operator itself. Spark image Building a Spark image without Hadoop using a specific version of Spark RUN curl -L https://dlcdn.apache.org/spark/spark-3.5.1/spark-3.5.1-bin-without-hadoop.tgz -o spark-3.5.1-bin-without-hadoop.tgz \
    && tar -xvzf spark-3.5.1-bin-without-hadoop.tgz \
    && mv spark-3.5.1-bin-without-hadoop /opt/spark \
    && rm spark-3.5.1-bin-without-hadoop.tgz RUN curl -L https://dlcdn.apache.org/spark/spark-3.5.1/spark-3.5.1-bin-without-hadoop.tgz -o spark-3.5.1-bin-without-hadoop.tgz \
    && tar -xvzf spark-3.5.1-bin-without-hadoop.tgz \
    && mv spark-3.5.1-bin-without-hadoop /opt/spark \
    && rm spark-3.5.1-bin-without-hadoop.tgz Spark-operator image We build the Spark Operator image, we will need several Hadoop libraries to run submit commands. For example, the FIPS version build is given, the differences in the build and run commands. For building on Go, the parameter GOEXPERIMENT=boringcrypto is used GOEXPERIMENT=boringcrypto For running spark-submit, the java parameter for Bouncy Castle is used Djavax.net.ssl.trustStorePassword=password Djavax.net.ssl.trustStorePassword=password You can build an image without FIPS changes. To run spark-submit, we will add Hadoop libraries during the build process: hadoop-client-runtime hadoop-client-api slf4j-api hadoop-client-runtime hadoop-client-runtime hadoop-client-api hadoop-client-api slf4j-api slf4j-api entrypoint.sh is used from the official Kubeflow repository https://github.com/kubeflow/spark-operator/blob/master/entrypoint.sh entrypoint.sh https://github.com/kubeflow/spark-operator/blob/master/entrypoint.sh Example Dockerfile for building Spark Operator ARG SPARK_IMAGE=spark-3.5.1-bin-without-hadoop
ARG GOLANG_IMAGE=golang-1.21
ARG SPARK_OPERATOR_VERSION=1.3.1
ARG HADOOP_VERSION_DEFAULT=3.4.0
ARG HADOOP_TMP_HOME="/opt/hadoop"
ARG TARGETARCH=amd64

# Prepare spark-operator build
FROM ${GOLANG_IMAGE} as builder
WORKDIR /app/spark-operator

ARG SPARK_OPERATOR_VERSION
RUN curl -Ls https://github.com/kubeflow/spark-operator/archive/refs/tags/spark-operator-chart-${SPARK_OPERATOR_VERSION}.tar.gz | tar -xz --strip-components 1 -C /app/spark-operator

RUN GOTOOLCHAIN=go1.22.3 go mod download

# Build
ARG TARGETARCH
RUN CGO_ENABLED=0 GOOS=linux GOARCH=${TARGETARCH} GO111MODULE=on GOTOOLCHAIN=go1.22.3 GOEXPERIMENT=boringcrypto go build -a -o /app/spark-operator/spark-operator main.go

#Install Hadoop jars
ARG HADOOP_VERSION_DEFAULT
ARG HADOOP_TMP_HOME
RUN mkdir -p ${HADOOP_TMP_HOME}
RUN curl -Ls https://archive.apache.org/dist/hadoop/common/hadoop-${HADOOP_VERSION_DEFAULT}/hadoop-${HADOOP_VERSION_DEFAULT}.tar.gz | tar -xz --strip-components 1 -C ${HADOOP_TMP_HOME}

# Prepare spark-operator image
FROM ${ECR_URL}:${SPARK_IMAGE}
WORKDIR /opt/spark-operator
USER root

ENV PATH $JAVA_HOME/bin:$PATH
ENV SPARK_HOME="/opt/spark"
ENV JAVA_HOME="/opt/jdk-11.0.21"
ENV SPARK_SUBMIT_OPTS="${SPARK_SUBMIT_OPTS} -Djavax.net.ssl.trustStorePassword=password"
ENV PATH=${PATH}:${SPARK_HOME}/bin:${JAVA_HOME}/bin:

RUN yum update -y && \
    yum install --setopt=tsflags=nodocs -y openssl && \
    yum clean all

ARG HADOOP_TMP_HOME
COPY --from=builder ${HADOOP_TMP_HOME}/share/hadoop/client/hadoop-client-runtime-*.jar ${HADOOP_TMP_HOME}/share/hadoop/client/hadoop-client-api-*.jar ${HADOOP_TMP_HOME}/share/hadoop/common/lib/slf4j-api-*.jar /opt/spark/jars/

COPY --from=builder /app/spark-operator/spark-operator /opt/spark-operator/
COPY --from=builder /app/spark-operator/hack/gencerts.sh /usr/bin/

COPY entrypoint.sh /opt/spark-operator/
RUN chmod a+x /opt/spark-operator/entrypoint.sh
ENTRYPOINT ["/opt/spark-operator/entrypoint.sh"] ARG SPARK_IMAGE=spark-3.5.1-bin-without-hadoop
ARG GOLANG_IMAGE=golang-1.21
ARG SPARK_OPERATOR_VERSION=1.3.1
ARG HADOOP_VERSION_DEFAULT=3.4.0
ARG HADOOP_TMP_HOME="/opt/hadoop"
ARG TARGETARCH=amd64

# Prepare spark-operator build
FROM ${GOLANG_IMAGE} as builder
WORKDIR /app/spark-operator

ARG SPARK_OPERATOR_VERSION
RUN curl -Ls https://github.com/kubeflow/spark-operator/archive/refs/tags/spark-operator-chart-${SPARK_OPERATOR_VERSION}.tar.gz | tar -xz --strip-components 1 -C /app/spark-operator

RUN GOTOOLCHAIN=go1.22.3 go mod download

# Build
ARG TARGETARCH
RUN CGO_ENABLED=0 GOOS=linux GOARCH=${TARGETARCH} GO111MODULE=on GOTOOLCHAIN=go1.22.3 GOEXPERIMENT=boringcrypto go build -a -o /app/spark-operator/spark-operator main.go

#Install Hadoop jars
ARG HADOOP_VERSION_DEFAULT
ARG HADOOP_TMP_HOME
RUN mkdir -p ${HADOOP_TMP_HOME}
RUN curl -Ls https://archive.apache.org/dist/hadoop/common/hadoop-${HADOOP_VERSION_DEFAULT}/hadoop-${HADOOP_VERSION_DEFAULT}.tar.gz | tar -xz --strip-components 1 -C ${HADOOP_TMP_HOME}

# Prepare spark-operator image
FROM ${ECR_URL}:${SPARK_IMAGE}
WORKDIR /opt/spark-operator
USER root

ENV PATH $JAVA_HOME/bin:$PATH
ENV SPARK_HOME="/opt/spark"
ENV JAVA_HOME="/opt/jdk-11.0.21"
ENV SPARK_SUBMIT_OPTS="${SPARK_SUBMIT_OPTS} -Djavax.net.ssl.trustStorePassword=password"
ENV PATH=${PATH}:${SPARK_HOME}/bin:${JAVA_HOME}/bin:

RUN yum update -y && \
    yum install --setopt=tsflags=nodocs -y openssl && \
    yum clean all

ARG HADOOP_TMP_HOME
COPY --from=builder ${HADOOP_TMP_HOME}/share/hadoop/client/hadoop-client-runtime-*.jar ${HADOOP_TMP_HOME}/share/hadoop/client/hadoop-client-api-*.jar ${HADOOP_TMP_HOME}/share/hadoop/common/lib/slf4j-api-*.jar /opt/spark/jars/

COPY --from=builder /app/spark-operator/spark-operator /opt/spark-operator/
COPY --from=builder /app/spark-operator/hack/gencerts.sh /usr/bin/

COPY entrypoint.sh /opt/spark-operator/
RUN chmod a+x /opt/spark-operator/entrypoint.sh
ENTRYPOINT ["/opt/spark-operator/entrypoint.sh"] Conclusion After the build, we still have several vulnerabilities in the Hadoop library hadoop-client-runtime: org.apache.avro:avro (hadoop-client-runtime-3.4.0.jar) – CVE-2023-39410 org.apache.commons:commons-compress – CVE-2024-25710, CVE-2024-26308 org.apache.avro:avro (hadoop-client-runtime-3.4.0.jar) – CVE-2023-39410 org.apache.commons:commons-compress – CVE-2024-25710, CVE-2024-26308 Since without this library we'll not be able to run spark-submit, but the rest of the huge part of the vulnerabilities is removed along with the main Hadoop libraries.

Building a Custom Docker Image for K8s Spark Operator to Fix Vulnerabilities

Developers are always concerned about their code quality. And they try their best to ensure the best quality of their code. If they find some tools that can show them exactly where they need to make changes to improve the quality of the code. There are many of them, but I will explain how you can set up SonarQube locally with docker so that you can evaluate your project code quality. Without wasting any time let’s begin. Prerequisite You must have docker installed in your system. If you don’t have it installed, then please feel free to set up Docker using this link. Docker Container On your root project, create a docker-compose.yml file, and then we will need a database so that Sonarqube can store its analytical data. postgres:
    image: postgres
    restart: always
    environment:
      POSTGRES_PASSWORD: example
    networks:
      - lookup
    ports:
      - 5432:5432
    volumes:
      - postgres:/var/lib/postgresql/data After that, we will need Sonarqube service, sonarqube:
    image: sonarqube:latest
    restart: always
    ports:
      - 9000:9000
    environment:
      SONARQUBE_JDBC_URL: jdbc:postgresql://postgres:5432/postgres
      SONARQUBE_JDBC_USERNAME: postgres
      SONARQUBE_JDBC_PASSWORD: example
    networks:
      - lookup
    depends_on:
      - postgres Don’t worry; I will explain each line. sonarqube:
This defines a service in your Docker Compose file. The name of the service is sonarqube, meaning that this section is describing how the SonarQube container will run.


image: sonarqube:latest
This specifies the Docker image to use for the SonarQube service. In this case, it pulls the latest version of the official SonarQube image from Docker Hub.


restart: always
This ensures that the SonarQube container will automatically restart if it crashes or if the Docker host is rebooted.


ports:
- 9000:9000:
This maps port 9000 on the host machine to port 9000 inside the container. SonarQube’s web interface runs on port 9000, so this makes it accessible from the host machine's browser at http://localhost:9000.


SONARQUBE_JDBC_URL: jdbc:postgresql://postgres:5432/postgres
Defines the JDBC URL to connect SonarQube to a PostgreSQL database.


SONARQUBE_JDBC_USERNAME: postgres:
The username used to authenticate with the PostgreSQL database. In this case, it's postgres.


SONARQUBE_JDBC_PASSWORD: example:
The password used to authenticate the PostgreSQL database, set to example. The final version of our docker-compose.yml file will be the following services:
  postgres:
    image: postgres
    restart: always
    environment:
      POSTGRES_PASSWORD: example
    networks:
      - lookup
    ports:
      - 5432:5432
    volumes:
      - postgres:/var/lib/postgresql/data

  sonarqube:
    image: sonarqube:latest
    restart: always
    ports:
      - 9000:9000
    environment:
      SONARQUBE_JDBC_URL: jdbc:postgresql://postgres:5432/postgres
      SONARQUBE_JDBC_USERNAME: postgres
      SONARQUBE_JDBC_PASSWORD: example
    networks:
      - lookup
    depends_on:
      - postgres

networks:
  lookup:

volumes:
  postgres:
    driver: local Inside your project directory, open a terminal, and to start and run the services do docker-compose up . Setup SonarQube Once the services are running on a browser, go to localhost:9000, and it will open the Sonarqube log-in page. The default username and password is admin. Once you are logged in, it will ask you to set up a new password. After completing these steps, you will be able to see the Sonarqube dashboard. As we are running the Sonarqube locally, we will create a local project. In the below, you will see the marked section that you have to click. It will ask for a project name and a branch name. I will leave the branch name as main. You can choose any branch that you have and create a project. Then we will see a new section where we have to choose Use the global setting  for now. Once we click on  create project, it should ask to generate a token. I will only update the Expires in section and make it to No Expiration. Once we generate a token, it will ask us to select our project coding language; after choosing that, we need to download the scanner zip folder. I have marked the section where you will find it below. Let’s unzip the folder and open the folder; you will find the bin folder, and inside, you will find a sonar-scanner file. Copy the path of that file, and save it in a text file. ~./sonar-scanner-cli-6.1.0.4477-linux-x64/sonar-scanner-6.1.0.4477-linux-x64/bin After that, you need to copy the command given in the Execute the Scanner section. Once you copy that, then concat with the path that you have saved in the file as shown below. ~./sonar-scanner-cli-6.1.0.4477-linux-x64/sonar-scanner-6.1.0.4477-linux-x64/bin/sonar-scanner \
   -Dsonar.projectKey=demo   \
   -Dsonar.sources=.   \
   -Dsonar.host.url=http://localhost:9000   \
   -Dsonar.token=<Your_Token> Once you execute the command in the terminal, it will start analyzing your code. Once it finishes the operation, you should see some analytical information in the Sonarqube dashboard. And then, you should be able to see some metrics, and if you have any issues, you should be able to see the reasons for marking it as an issue. Great, we have completed the setup of our Sonarqube on our project locally. Developers are always concerned about their code quality. And they try their best to ensure the best quality of their code. If they find some tools that can show them exactly where they need to make changes to improve the quality of the code. There are many of them, but I will explain how you can set up SonarQube locally with docker so that you can evaluate your project code quality. Without wasting any time let’s begin. Prerequisite You must have docker installed in your system. If you don’t have it installed, then please feel free to set up Docker using this link . link Docker Container On your root project, create a docker-compose.yml file, and then we will need a database so that Sonarqube can store its analytical data. postgres:
    image: postgres
    restart: always
    environment:
      POSTGRES_PASSWORD: example
    networks:
      - lookup
    ports:
      - 5432:5432
    volumes:
      - postgres:/var/lib/postgresql/data postgres:
    image: postgres
    restart: always
    environment:
      POSTGRES_PASSWORD: example
    networks:
      - lookup
    ports:
      - 5432:5432
    volumes:
      - postgres:/var/lib/postgresql/data After that, we will need Sonarqube service, sonarqube:
    image: sonarqube:latest
    restart: always
    ports:
      - 9000:9000
    environment:
      SONARQUBE_JDBC_URL: jdbc:postgresql://postgres:5432/postgres
      SONARQUBE_JDBC_USERNAME: postgres
      SONARQUBE_JDBC_PASSWORD: example
    networks:
      - lookup
    depends_on:
      - postgres sonarqube:
    image: sonarqube:latest
    restart: always
    ports:
      - 9000:9000
    environment:
      SONARQUBE_JDBC_URL: jdbc:postgresql://postgres:5432/postgres
      SONARQUBE_JDBC_USERNAME: postgres
      SONARQUBE_JDBC_PASSWORD: example
    networks:
      - lookup
    depends_on:
      - postgres Don’t worry; I will explain each line. sonarqube:
This defines a service in your Docker Compose file. The name of the service is sonarqube, meaning that this section is describing how the SonarQube container will run. image: sonarqube:latest
This specifies the Docker image to use for the SonarQube service. In this case, it pulls the latest version of the official SonarQube image from Docker Hub. restart: always
This ensures that the SonarQube container will automatically restart if it crashes or if the Docker host is rebooted. ports:
- 9000:9000:
This maps port 9000 on the host machine to port 9000 inside the container. SonarQube’s web interface runs on port 9000, so this makes it accessible from the host machine's browser at http://localhost:9000. SONARQUBE_JDBC_URL: jdbc:postgresql://postgres:5432/postgres
Defines the JDBC URL to connect SonarQube to a PostgreSQL database. SONARQUBE_JDBC_USERNAME: postgres:
The username used to authenticate with the PostgreSQL database. In this case, it's postgres. SONARQUBE_JDBC_PASSWORD: example:
The password used to authenticate the PostgreSQL database, set to example. sonarqube: This defines a service in your Docker Compose file. The name of the service is sonarqube, meaning that this section is describing how the SonarQube container will run. sonarqube: sonarqube: This defines a service in your Docker Compose file. The name of the service is sonarqube , meaning that this section is describing how the SonarQube container will run. sonarqube image: sonarqube:latest This specifies the Docker image to use for the SonarQube service. In this case, it pulls the latest version of the official SonarQube image from Docker Hub. image: sonarqube:latest image: sonarqube:latest This specifies the Docker image to use for the SonarQube service. In this case, it pulls the latest version of the official SonarQube image from Docker Hub. restart: always This ensures that the SonarQube container will automatically restart if it crashes or if the Docker host is rebooted. restart: always restart: always This ensures that the SonarQube container will automatically restart if it crashes or if the Docker host is rebooted. ports: - 9000:9000: This maps port 9000 on the host machine to port 9000 inside the container. SonarQube’s web interface runs on port 9000, so this makes it accessible from the host machine's browser at http://localhost:9000. ports: ports: - 9000:9000 : - 9000:9000 This maps port 9000 on the host machine to port 9000 inside the container. SonarQube’s web interface runs on port 9000, so this makes it accessible from the host machine's browser at http://localhost:9000 . http://localhost:9000 SONARQUBE_JDBC_URL: jdbc:postgresql://postgres:5432/postgres Defines the JDBC URL to connect SonarQube to a PostgreSQL database. SONARQUBE_JDBC_URL: jdbc:postgresql://postgres:5432/postgres SONARQUBE_JDBC_URL: jdbc:postgresql://postgres:5432/postgres Defines the JDBC URL to connect SonarQube to a PostgreSQL database. SONARQUBE_JDBC_USERNAME: postgres: The username used to authenticate with the PostgreSQL database. In this case, it's postgres. SONARQUBE_JDBC_USERNAME: postgres : SONARQUBE_JDBC_USERNAME: postgres The username used to authenticate with the PostgreSQL database. In this case, it's postgres . postgres SONARQUBE_JDBC_PASSWORD: example: The password used to authenticate the PostgreSQL database, set to example. SONARQUBE_JDBC_PASSWORD: example : SONARQUBE_JDBC_PASSWORD: example The password used to authenticate the PostgreSQL database, set to example . example The final version of our docker-compose.yml file will be the following services:
  postgres:
    image: postgres
    restart: always
    environment:
      POSTGRES_PASSWORD: example
    networks:
      - lookup
    ports:
      - 5432:5432
    volumes:
      - postgres:/var/lib/postgresql/data

  sonarqube:
    image: sonarqube:latest
    restart: always
    ports:
      - 9000:9000
    environment:
      SONARQUBE_JDBC_URL: jdbc:postgresql://postgres:5432/postgres
      SONARQUBE_JDBC_USERNAME: postgres
      SONARQUBE_JDBC_PASSWORD: example
    networks:
      - lookup
    depends_on:
      - postgres

networks:
  lookup:

volumes:
  postgres:
    driver: local services:
  postgres:
    image: postgres
    restart: always
    environment:
      POSTGRES_PASSWORD: example
    networks:
      - lookup
    ports:
      - 5432:5432
    volumes:
      - postgres:/var/lib/postgresql/data

  sonarqube:
    image: sonarqube:latest
    restart: always
    ports:
      - 9000:9000
    environment:
      SONARQUBE_JDBC_URL: jdbc:postgresql://postgres:5432/postgres
      SONARQUBE_JDBC_USERNAME: postgres
      SONARQUBE_JDBC_PASSWORD: example
    networks:
      - lookup
    depends_on:
      - postgres

networks:
  lookup:

volumes:
  postgres:
    driver: local Inside your project directory, open a terminal, and to start and run the services do docker-compose up . docker-compose up Setup SonarQube Once the services are running on a browser, go to localhost:9000 , and it will open the Sonarqube log-in page. The default username and password is admin . Once you are logged in, it will ask you to set up a new password. localhost:9000 admin After completing these steps, you will be able to see the Sonarqube dashboard. As we are running the Sonarqube locally, we will create a local project. In the below, you will see the marked section that you have to click. It will ask for a project name and a branch name. I will leave the branch name as main . You can choose any branch that you have and create a project. Then we will see a new section where we have to choose Use the global setting for now. main Use the global setting Once we click on create project , it should ask to generate a token. I will only update the Expires in section and make it to No Expiration . Once we generate a token, it will ask us to select our project coding language; after choosing that, we need to download the scanner zip folder. I have marked the section where you will find it below. create project Expires in No Expiration Let’s unzip the folder and open the folder; you will find the bin folder, and inside, you will find a sonar-scanner file. Copy the path of that file, and save it in a text file. bin sonar-scanner ~./sonar-scanner-cli-6.1.0.4477-linux-x64/sonar-scanner-6.1.0.4477-linux-x64/bin ~./sonar-scanner-cli-6.1.0.4477-linux-x64/sonar-scanner-6.1.0.4477-linux-x64/bin After that, you need to copy the command given in the Execute the Scanner section. Execute the Scanner Once you copy that, then concat with the path that you have saved in the file as shown below. ~./sonar-scanner-cli-6.1.0.4477-linux-x64/sonar-scanner-6.1.0.4477-linux-x64/bin/sonar-scanner \
   -Dsonar.projectKey=demo   \
   -Dsonar.sources=.   \
   -Dsonar.host.url=http://localhost:9000   \
   -Dsonar.token=<Your_Token> ~./sonar-scanner-cli-6.1.0.4477-linux-x64/sonar-scanner-6.1.0.4477-linux-x64/bin/sonar-scanner \
   -Dsonar.projectKey=demo   \
   -Dsonar.sources=.   \
   -Dsonar.host.url=http://localhost:9000   \
   -Dsonar.token=<Your_Token> Once you execute the command in the terminal, it will start analyzing your code. Once it finishes the operation, you should see some analytical information in the Sonarqube dashboard. And then, you should be able to see some metrics, and if you have any issues, you should be able to see the reasons for marking it as an issue. Great, we have completed the setup of our Sonarqube on our project locally.

How to Setup SonarQube Locally Using Docker

A Deep Dive into Docker Compose

You may be wondering what DOCKERIZE is. It means using Docker to containerize your dApp project. Docker is an open platform for developing, shipping, and running applications. Docker enables you to separate your applications from your infrastructure so you can deliver software quickly. Applications include web apps, APIs, mobile apps, and backend services. It includes the code you write, the libraries you use, and the configurations needed for it to run. Infrastructure like operating system(OS), Network setting, web servers, database, and other services your application interacts with. It also involves the hardware or virtual machines where your application runs. Docker allows you to package and run your application as a single entity in a loosely isolated environment called a CONTAINER. Container Containers are lightweight and contain everything needed to run the application, so you don't need to rely on what's installed on the host. You can share containers while you work, and be sure that everyone you share with gets the same container that works in the same way. For example, you and your colleagues are working on a dApp project that uses Node.js and Hardhat. Initially, you set up the project on your local machine, knowing that before your colleagues can run the application locally, they would need to install Node.js and Hardhat on their systems. Now, let’s imagine one of your colleagues is using a machine that isn't compatible with Node.js and Hardhat. How would you solve this problem? With Docker, you can package your application, along with all its dependencies, into a single container that can run on any operating system. This solves the issue of installing software or dependencies separately on different machines. Images After you’ve containerized your application, how do you get it to run? As a frontend developer, when starting a new React app, you usually run npx create-react-app or npm init vite@latest. For an existing project, you'd typically run git pull followed by npm install to get it running locally. Similarly, to run a container, you need something called a container image. A container image is a standardized package that includes all the files, binaries, libraries, and configurations needed to run your application inside a container. It’s like having everything bundled together so it can run consistently across any environment. See below for a container image: Practicals In this section, you will learn how to containerize and share your app. Prerequisites You have installed the latest version of Docker Desktop.
You have installed a Git client.
You have an IDE or a text editor to edit files. Docker recommends using Visual Studio Code. Containerize Your Application In your root directory, initialize docker using this command prompt: docker init Provide answers to the prompts. ? What application platform does your project use? Node  
? What version of Node do you want to use? 20.16.0  
? Which package manager do you want to use? npm  
? Do you want to run "npm run build" before starting your server? No  
? What command do you want to use to start the app? ["npm", "run", "dev"]  
? What port does your server listen on? 5173 You will see three additional files in your directory, namely .dockerignore, compose.yaml and DockerFile. Dockerfile: A Dockerfile is a text document that contains all the commands a user could call on the command line to assemble an image.
compose.yaml: The Compose file, or compose.yaml file follows the rules provided by the Compose Specification in how to define multi-container applications.
.dockerignore: This file excludes files and directories from the build context. Inside Dockerfile, clear out the file and add the following: # syntax=docker/dockerfile:1  
ARG NODE_VERSION=20.16.0  
FROM node:${NODE_VERSION}-alpine  
# Use development node environment by default.  
ENV NODE_ENV development  
WORKDIR /app  
# Copy package.json and package-lock.json to the working directory  
COPY package.json package-lock.json ./  
# Install dependencies  
RUN npm install  
# Copy the rest of the source files into the image  
COPY . .  
# Change ownership of the /app directory to the node user  
RUN chown -R node:node /app  
# Switch to the node user  
USER node  
# Ensure node_modules/.bin is in the PATH  
ENV PATH /app/node_modules/.bin:$PATH  
# Expose the port that the application listens on  
EXPOSE 5173  
# Run the application  
CMD ["npm", "run", "dev"] Build your container image using this: docker build -t <your-image-name> . To see your container image, go to your docker desktop, click on Images, as shown below  or you can use this command prompt, docker images to see your container image in your terminal.  6. Run the image: docker run -p 5173:5173 <your-image-name> You should see an output like this: > vite-project@0.0.0 dev
> vite

  VITE v5.4.2  ready in 222 ms

  ➜  Local:   http://localhost:5173/
  ➜  Network: http://172.17.0.2:5173/ Sharing the Application Sign up or Sign in to Docker Hub.
Select the Create Repository button.  
Name your repository with the same container image name. Make sure the Visibility is Public.
Select Create. Sign in to Docker Hub using the command line docker login -u <YOUR USERNAME> Enter your Password Your Password won’t be visible while typing Use the docker tag command to give the getting-started image a new name. Replace YOUR-USER-NAME with your Docker ID: You may encounter this denied: requested access to the resource is denied while pushing.
To fix this,

relogin using docker login, press enter.
After that, you will be navigated to your browser,
copy your OTP the terminal and input it,
press CONFIRM
repush the image

You will see an output like: This shows your image has been built and pushed into a registry. Your team can now pull this image and run the application on their local machine. You can check by searching for your container image on the search bar on your docker desktop or docker hub. Troubleshooting Port Conflict:

Error: if the port 8000 was already in use.
Solution: Identified the process using port 8000 and terminated it using the following commands: # Find the process using port 8000  
lsof -i :8000  
# Kill the process (replace <PID> with the actual process ID found from the previous command)  
kill -9 <PID> Alternatively, you could run the Docker container on a different port: docker run -p 8001:8000 my-vite-app Update an Application Anything you update your app. e.g., you change a syntax or function,

Stop your docker desktop, click on Containers
Stop the port, and click on delete.  
Rebuild the Application using the docker build command. docker build -t <your-image-name> . Rerun the app. docker run -p 5173:5173 <your-image-name> Conclusion By following this guide, you can easily package your application and its dependencies into a container, share it with your team, and run it easily on any machine. This not only enhances collaboration but also mitigates issues related to environment setup and compatibility. You may be wondering what DOCKERIZE is. It means using Docker to containerize your dApp project. DOCKERIZE DOCKERIZE Docker Docker is an open platform for developing, shipping, and running applications. Docker enables you to separate your applications from your infrastructure so you can deliver software quickly. applications infrastructure Applications include web apps, APIs, mobile apps, and backend services. It includes the code you write, the libraries you use, and the configurations needed for it to run. Applications include web apps, APIs, mobile apps, and backend services. It includes the code you write, the libraries you use, and the configurations needed for it to run. Applications Infrastructure like operating system(OS), Network setting, web servers, database, and other services your application interacts with. It also involves the hardware or virtual machines where your application runs. Infrastructure like operating system(OS), Network setting, web servers, database, and other services your application interacts with. It also involves the hardware or virtual machines where your application runs. Infrastructure Docker allows you to package and run your application as a single entity in a loosely isolated environment called a CONTAINER. CONTAINER. Container Containers are lightweight and contain everything needed to run the application, so you don't need to rely on what's installed on the host. You can share containers while you work, and be sure that everyone you share with gets the same container that works in the same way. For example, you and your colleagues are working on a dApp project that uses Node.js and Hardhat. Initially, you set up the project on your local machine, knowing that before your colleagues can run the application locally, they would need to install Node.js and Hardhat on their systems. Now, let’s imagine one of your colleagues is using a machine that isn't compatible with Node.js and Hardhat. How would you solve this problem? With Docker, you can package your application, along with all its dependencies, into a single container that can run on any operating system. This solves the issue of installing software or dependencies separately on different machines. Images After you’ve containerized your application, how do you get it to run? As a frontend developer, when starting a new React app, you usually run npx create-react-app or npm init vite@latest . For an existing project, you'd typically run git pull followed by npm install to get it running locally. Similarly, to run a container, you need something called a container image . npx create-react-app npm init vite@latest git pull npm install container image A container image is a standardized package that includes all the files, binaries, libraries, and configurations needed to run your application inside a container. It’s like having everything bundled together so it can run consistently across any environment. See below for a container image: Practicals In this section, you will learn how to containerize and share your app. Prerequisites You have installed the latest version of Docker Desktop. You have installed a Git client. You have an IDE or a text editor to edit files. Docker recommends using Visual Studio Code. You have installed the latest version of Docker Desktop . Docker Desktop You have installed a Git client . Git client You have an IDE or a text editor to edit files. Docker recommends using Visual Studio C ode. Visual Studio C ode. Containerize Your Application In your root directory, initialize docker using this command prompt: In your root directory, initialize docker using this command prompt: docker init docker init Provide answers to the prompts. Provide answers to the prompts. ? What application platform does your project use? Node  
? What version of Node do you want to use? 20.16.0  
? Which package manager do you want to use? npm  
? Do you want to run "npm run build" before starting your server? No  
? What command do you want to use to start the app? ["npm", "run", "dev"]  
? What port does your server listen on? 5173 ? What application platform does your project use? Node  
? What version of Node do you want to use? 20.16.0  
? Which package manager do you want to use? npm  
? Do you want to run "npm run build" before starting your server? No  
? What command do you want to use to start the app? ["npm", "run", "dev"]  
? What port does your server listen on? 5173 You will see three additional files in your directory, namely .dockerignore, compose.yaml and DockerFile. You will see three additional files in your directory, namely .dockerignore , compose.yaml and DockerFile. .dockerignore compose.yaml DockerFile. Dockerfile: A Dockerfile is a text document that contains all the commands a user could call on the command line to assemble an image. compose.yaml: The Compose file, or compose.yaml file follows the rules provided by the Compose Specification in how to define multi-container applications. .dockerignore: This file excludes files and directories from the build context. Dockerfile : A Dockerfile is a text document that contains all the commands a user could call on the command line to assemble an image. Dockerfile compose.yaml : The Compose file, or compose.yaml file follows the rules provided by the Compose Specification in how to define mul ti-container applications. compose.yaml compose.yaml Compose Specification in how to define mul .dockerignore : This file excludes files and directories from the build context. .dockerignore Inside Dockerfile, clear out the file and add the following: Inside Dockerfile , clear out the file and add the following: Dockerfile # syntax=docker/dockerfile:1  
ARG NODE_VERSION=20.16.0  
FROM node:${NODE_VERSION}-alpine  
# Use development node environment by default.  
ENV NODE_ENV development  
WORKDIR /app  
# Copy package.json and package-lock.json to the working directory  
COPY package.json package-lock.json ./  
# Install dependencies  
RUN npm install  
# Copy the rest of the source files into the image  
COPY . .  
# Change ownership of the /app directory to the node user  
RUN chown -R node:node /app  
# Switch to the node user  
USER node  
# Ensure node_modules/.bin is in the PATH  
ENV PATH /app/node_modules/.bin:$PATH  
# Expose the port that the application listens on  
EXPOSE 5173  
# Run the application  
CMD ["npm", "run", "dev"] # syntax=docker/dockerfile:1  
ARG NODE_VERSION=20.16.0  
FROM node:${NODE_VERSION}-alpine  
# Use development node environment by default.  
ENV NODE_ENV development  
WORKDIR /app  
# Copy package.json and package-lock.json to the working directory  
COPY package.json package-lock.json ./  
# Install dependencies  
RUN npm install  
# Copy the rest of the source files into the image  
COPY . .  
# Change ownership of the /app directory to the node user  
RUN chown -R node:node /app  
# Switch to the node user  
USER node  
# Ensure node_modules/.bin is in the PATH  
ENV PATH /app/node_modules/.bin:$PATH  
# Expose the port that the application listens on  
EXPOSE 5173  
# Run the application  
CMD ["npm", "run", "dev"] Build your container image using this: Build your container image using this: docker build -t <your-image-name> . docker build -t <your-image-name> . To see your container image, go to your docker desktop, click on Images , as shown below or you can use this command prompt, Images docker images docker images to see your container image in your terminal. 6. Run the image: docker run -p 5173:5173 <your-image-name> docker run -p 5173:5173 <your-image-name> You should see an output like this: > vite-project@0.0.0 dev
> vite

  VITE v5.4.2  ready in 222 ms

  ➜  Local:   http://localhost:5173/
  ➜  Network: http://172.17.0.2:5173/ > vite-project@0.0.0 dev
> vite

  VITE v5.4.2  ready in 222 ms

  ➜  Local:   http://localhost:5173/
  ➜  Network: http://172.17.0.2:5173/ Sharing the Application Sign up or Sign in to Docker Hub. Select the Create Repository button. Name your repository with the same container image name. Make sure the Visibility is Public. Select Create. Sign up or Sign in to Docker Hub. Sign up Docker Hub. Select the Create Repository button. Create Repository Name your repository with the same container image name. Make sure the Visibility is Public . Visibility Public Select Create . Create Sign in to Docker Hub using the command line Sign in to Docker Hub using the command line docker login -u <YOUR USERNAME> docker login -u <YOUR USERNAME> Enter your Password Enter your Password Your Password won’t be visible while typing Your Password won’t be visible while typing Use the docker tag command to give the getting-started image a new name. Replace YOUR-USER-NAME with your Docker ID: Use the docker tag command to give the getting-started image a new name. Replace YOUR-USER-NAME with your Docker ID: docker tag getting-started YOUR-USER-NAME You may encounter this denied: requested access to the resource is denied while pushing. To fix this, relogin using docker login, press enter.
After that, you will be navigated to your browser,
copy your OTP the terminal and input it,
press CONFIRM
repush the image You will see an output like: You may encounter this denied: requested access to the resource is denied while pushing. denied: requested access to the resource is denied To fix this, relogin using docker login, press enter. After that, you will be navigated to your browser, copy your OTP the terminal and input it, press CONFIRM repush the image relogin using docker login , press enter. docker login After that, you will be navigated to your browser, copy your OTP the terminal and input it, OTP press CONFIRM CONFIRM repush the image You will see an output like: This shows your image has been built and pushed into a registry. Your team can now pull this image and run the application on their local machine. You can check by searching for your container image on the search bar on your docker desktop or docker hub . docker hub Troubleshooting Port Conflict:

Error: if the port 8000 was already in use.
Solution: Identified the process using port 8000 and terminated it using the following commands: Port Conflict : Error: if the port 8000 was already in use.
Solution: Identified the process using port 8000 and terminated it using the following commands: Port Conflict Error: if the port 8000 was already in use. Solution: Identified the process using port 8000 and terminated it using the following commands: Error : if the port 8000 was already in use. Error 8000 Solution : Identified the process using port 8000 and terminated it using the following commands: Solution 8000 # Find the process using port 8000  
lsof -i :8000  
# Kill the process (replace <PID> with the actual process ID found from the previous command)  
kill -9 <PID> # Find the process using port 8000  
lsof -i :8000  
# Kill the process (replace <PID> with the actual process ID found from the previous command)  
kill -9 <PID> Alternatively, you could run the Docker container on a different port: Alternatively, you could run the Docker container on a different port: docker run -p 8001:8000 my-vite-app docker run -p 8001:8000 my-vite-app Update an Application Anything you update your app. e.g., you change a syntax or function,

Stop your docker desktop, click on Containers
Stop the port, and click on delete.  
Rebuild the Application using the docker build command. Update an Application Anything you update your app. e.g., you change a syntax or function, Stop your docker desktop, click on Containers
Stop the port, and click on delete.  
Rebuild the Application using the docker build command. Update an Application Anything you update your app. e.g., you change a syntax or function, Update an Application Stop your docker desktop, click on Containers Stop the port, and click on delete. Rebuild the Application using the docker build command. Stop your docker desktop, click on Containers Containers Stop the port, and click on delete. Rebuild the Application using the docker build command. docker build docker build -t <your-image-name> . docker build -t <your-image-name> . Rerun the app. Rerun the app. docker run -p 5173:5173 <your-image-name> docker run -p 5173:5173 <your-image-name> Conclusion By following this guide, you can easily package your application and its dependencies into a container, share it with your team, and run it easily on any machine. This not only enhances collaboration but also mitigates issues related to environment setup and compatibility.

A Beginner's Guide to Dockerizing a Vite dApp Project

የ Vite dApp ፕሮጀክትን ለመቆጠብ የጀማሪ መመሪያ

Guía para principiantes sobre cómo dockerizar un proyecto de dApp de Vite

Unha guía para principiantes para dockerizar un proxecto Vite dApp

Vite dApp プロジェクトを Docker 化する初心者向けガイド

დამწყებთათვის გზამკვლევი Vite dApp პროექტის დოკერიზაციისთვის

Panduan Pemula untuk Melabuhkan Projek Vite dApp

Vite dApp परियोजना डकराइज गर्नको लागि एक शुरुआती गाइड

Huk qallariqpa yanapakuynin huk Vite dApp Proyecto Dockerizing kaqpaq

Mwongozo wa Waanzilishi wa Kuweka Mradi wa Vite dApp

ንጀመርቲ መምርሒ ንዶከርይዝድ ፕሮጀክት ቫይት ዲኣፕ

“Vite dApp” taslamasyny dokerlemek üçin başlangyç gollanma

Hướng dẫn cho người mới bắt đầu về Dockerizing một dự án dApp Vite

An Intro to Kubernetes for Docker Developers

Wazuh on Docker: Secure Your Dashboard with Let's Encrypt and Deploy Agent with Kubernetes Daemonset

Docker deployment aims to start and deploy Apache Dolphinscheduler services quickly in a container. Prerequisites docker-compose Docker Single-node Deployment of Dolphinscheduler Using Containers Please download the source package apache-dolphinscheduler — src.tar.gz from Download First, ensure the ports required for service startup are not occupied: port_list=(12345 25333 5432)
for port in ${port_list[@]}; do
  netstat -an | grep $port
done
# No output means the ports are not occupied If the ports are occupied: vim docker-compose.yml Find dolphinscheduler-api, and modify the ports. # Default content
ports:
   - "12345:12345"
   - "25333:25333"

# Modify as needed, e.g.:
ports:
   - "22345:12345"
   - "35333:25333" 2. Install Postgresql and Dolphinscheduler services: tar -zxvf apache-dolphinscheduler-<version>-src.tar.gz
cd apache-dolphinscheduler-<version>-src/deploy/docker
# Initialize database, etc.
docker-compose --profile schema up -d 
# Start all services
docker-compose --profile all up -d Distributed Deployment of Dolphinscheduler Using Containers After modifying the .env file in the deploy folder, copy it to other servers.


On other servers, where you want to start Dolphinscheduler services (e.g., Dolphinscheduler-api), execute the startup command: docker-compose up -d dolphinscheduler-api Using Non-containerized Postgresql and Zookeeper Modify environment variables: # Enter the Deploy directory
cd apache-dolphinscheduler-<version>-src/deploy/docker
vim .env
## Modify the following contents
# Database name
DATABASE=dolp
# Database connection address
SPRING_DATASOURCE_URL=jdbc:postgresql://10.0.8.14:15432/dolp
# Zookeeper connection address
REGISTRY_ZOOKEEPER_CONNECT_STRING=10.0.8.14:12181
# Username for connecting to the database
SPRING_DATASOURCE_USERNAME=dolp
# Password for connecting to the database
SPRING_DATASOURCE_PASSWORD=111111 Initialization: docker-compose --profile schema up -d Start Dolphinscheduler services: # Start the worker service
docker-compose up -d dolphinscheduler-worker
# Start the master service
docker-compose up -d dolphinscheduler-master
# Start the alert service
docker-compose up -d dolphinscheduler-alert
# Start the API service
docker-compose up -d dolphinscheduler-api Docker deployment aims to start and deploy Apache Dolphinscheduler services quickly in a container. Prerequisites docker-compose docker-compose docker-compose Docker Docker Single-node Deployment of Dolphinscheduler Using Containers Please download the source package apache-dolphinscheduler — src.tar.gz from Download Please download the source package apache-dolphinscheduler — src.tar.gz from Download Download First, ensure the ports required for service startup are not occupied: First, ensure the ports required for service startup are not occupied: port_list=(12345 25333 5432)
for port in ${port_list[@]}; do
  netstat -an | grep $port
done
# No output means the ports are not occupied port_list=(12345 25333 5432)
for port in ${port_list[@]}; do
  netstat -an | grep $port
done
# No output means the ports are not occupied If the ports are occupied: If the ports are occupied: vim docker-compose.yml vim docker-compose.yml Find dolphinscheduler-api, and modify the ports. Find dolphinscheduler-api , and modify the ports. dolphinscheduler-api # Default content
ports:
   - "12345:12345"
   - "25333:25333"

# Modify as needed, e.g.:
ports:
   - "22345:12345"
   - "35333:25333" # Default content
ports:
   - "12345:12345"
   - "25333:25333"

# Modify as needed, e.g.:
ports:
   - "22345:12345"
   - "35333:25333" 2. Install Postgresql and Dolphinscheduler services: tar -zxvf apache-dolphinscheduler-<version>-src.tar.gz
cd apache-dolphinscheduler-<version>-src/deploy/docker
# Initialize database, etc.
docker-compose --profile schema up -d 
# Start all services
docker-compose --profile all up -d tar -zxvf apache-dolphinscheduler-<version>-src.tar.gz
cd apache-dolphinscheduler-<version>-src/deploy/docker
# Initialize database, etc.
docker-compose --profile schema up -d 
# Start all services
docker-compose --profile all up -d Distributed Deployment of Dolphinscheduler Using Containers After modifying the .env file in the deploy folder, copy it to other servers. On other servers, where you want to start Dolphinscheduler services (e.g., Dolphinscheduler-api), execute the startup command: After modifying the .env file in the deploy folder, copy it to other servers. After modifying the .env file in the deploy folder, copy it to other servers. On other servers, where you want to start Dolphinscheduler services (e.g., Dolphinscheduler-api), execute the startup command: On other servers, where you want to start Dolphinscheduler services (e.g., Dolphinscheduler-api), execute the startup command: docker-compose up -d dolphinscheduler-api docker-compose up -d dolphinscheduler-api Using Non-containerized Postgresql and Zookeeper Modify environment variables: Modify environment variables: # Enter the Deploy directory
cd apache-dolphinscheduler-<version>-src/deploy/docker
vim .env
## Modify the following contents
# Database name
DATABASE=dolp
# Database connection address
SPRING_DATASOURCE_URL=jdbc:postgresql://10.0.8.14:15432/dolp
# Zookeeper connection address
REGISTRY_ZOOKEEPER_CONNECT_STRING=10.0.8.14:12181
# Username for connecting to the database
SPRING_DATASOURCE_USERNAME=dolp
# Password for connecting to the database
SPRING_DATASOURCE_PASSWORD=111111 # Enter the Deploy directory
cd apache-dolphinscheduler-<version>-src/deploy/docker
vim .env
## Modify the following contents
# Database name
DATABASE=dolp
# Database connection address
SPRING_DATASOURCE_URL=jdbc:postgresql://10.0.8.14:15432/dolp
# Zookeeper connection address
REGISTRY_ZOOKEEPER_CONNECT_STRING=10.0.8.14:12181
# Username for connecting to the database
SPRING_DATASOURCE_USERNAME=dolp
# Password for connecting to the database
SPRING_DATASOURCE_PASSWORD=111111 Initialization: Initialization: docker-compose --profile schema up -d docker-compose --profile schema up -d Start Dolphinscheduler services: Start Dolphinscheduler services: # Start the worker service
docker-compose up -d dolphinscheduler-worker
# Start the master service
docker-compose up -d dolphinscheduler-master
# Start the alert service
docker-compose up -d dolphinscheduler-alert
# Start the API service
docker-compose up -d dolphinscheduler-api # Start the worker service
docker-compose up -d dolphinscheduler-worker
# Start the master service
docker-compose up -d dolphinscheduler-master
# Start the alert service
docker-compose up -d dolphinscheduler-alert
# Start the API service
docker-compose up -d dolphinscheduler-api

A Guide to Deploying Dolphinscheduler With Docker

Redirecting DNS Traffic in Docker Containers Using Tc-BPF

You can see the hands-on video here: https://blog.sofianeouafir.com/docker/video-hands-on-introduction-to-docker-stack Introduction to Docker Stack Docker Stack is an essential tool for managing Docker services across multiple nodes in a Docker Swarm using a docker-compose.yml file. In this tutorial, I dive into Docker Stack’s deployment capabilities and the secure management of secrets, using a practical example involving Drupal and PostgreSQL. Docker-Compose File Setup Here’s our docker-compose.yml file that sets up the Drupal and PostgreSQL services: version: '3.8' services: drupal: image: drupal:8.2 ports: - "8080:80" deploy: replicas: 2 update_config: parallelism: 2 delay: 10s restart_policy: condition: on-failure postgres: image: postgres:14 secrets: - psql-pw environment: POSTGRES_PASSWORD_FILE: /run/secrets/psql-pw POSTGRES_DB: drupal POSTGRES_USER: user deploy: placement: constraints: [node.role == manager] secrets: psql-pw: external: true Understanding the Deploy Section The deploy section of the Docker Compose file specifies deployment strategies in a swarm environment, including: Replicas: Defines the number of instances for the service.
Update_config:
Parallelism: Sets how many service tasks are updated simultaneously.
Delay: Adds a delay between updates to different service tasks to ensure smooth roll-outs.
Restart_policy:
Condition: Determines under what conditions the service should be restarted. Managing Secrets in Docker Stack What is a Docker Secret? A Docker secret securely stores and manages sensitive data within Docker services, encrypted during transit and at rest, and only accessible by specifically authorized services. Managing Secrets with Docker Stack Secrets can be managed in Docker Stack in two ways: Using Secret Files: Ideal for development, storing sensitive data in local files securely transferred into Docker Swarm during deployment.
secrets: psql-pw: file: ./your-file.txt
External Secrets: For production, secrets are created directly in Docker using the CLI, avoiding the storage of sensitive information in files.
secrets: psql-pw: external: true Hands-On: Deploying the Stack in Docker Swarm To deploy your stack in Docker Swarm, follow these steps: Step 1: Prepare Your Docker Environment Ensure Docker is running and connected to your Docker Swarm. Refer to this guide for setting up a 3-node Docker Swarm. Step 2: Create and Edit the docker-compose.yml File On your swarm node, initialize and edit the docker-compose file to inclue our own docker-compose file: touch docker-compose.yml; vim docker-compose.yml; Step 3: Create the Secret echo "myverysecretpassword" | docker secret create psql-pw - Step 4: Deploy the Stack docker stack deploy -c docker-compose.yml demo-stack Step 5: Verify the Deployment List Services: docker stack services demo-stack
List Tasks: docker stack ps demo-stack Step 6: View the Secrets in the Container Get inside the container docker exec -it [container-name] bash See the secret values cat /run/secrets/psql-pw Step 7: View Logs and Debug docker service logs drupal
docker service logs postgres Step 8: Play with your configs For example, you can scale up and down your services by editing the replicas values in your docker-compose file and re-deploy using the same deploy command. Step 9: Play with your Drupal app! Your drupal application should now be up and running on port 8080! Conclusion You have successfully configured a robust Docker Swarm with three nodes. This setup is now running efficient instances of Drupal and PostgreSQL, demonstrating Docker Stack’s powerful capabilities for real-world applications. Happy deploying! You can see the hands-on video here: https://blog.sofianeouafir.com/docker/video-hands-on-introduction-to-docker-stack https://blog.sofianeouafir.com/docker/video-hands-on-introduction-to-docker-stack Introduction to Docker Stack Docker Stack is an essential tool for managing Docker services across multiple nodes in a Docker Swarm using a docker-compose.yml file . In this tutorial, I dive into Docker Stack’s deployment capabilities and the secure management of secrets, using a practical example involving Drupal and PostgreSQL. using a docker-compose.yml file Docker-Compose File Setup Here’s our docker-compose.yml file that sets up the Drupal and PostgreSQL services: docker-compose.yml version: '3.8' services: drupal: image: drupal:8.2 ports: - "8080:80" deploy: replicas: 2 update_config: parallelism: 2 delay: 10s restart_policy: condition: on-failure version: '3.8' services: drupal: image: drupal:8.2 ports: - "8080:80" deploy: replicas: 2 update_config: parallelism: 2 delay: 10s restart_policy: condition: on-failure postgres: image: postgres:14 secrets: - psql-pw environment: POSTGRES_PASSWORD_FILE: /run/secrets/psql-pw POSTGRES_DB: drupal POSTGRES_USER: user deploy: placement: constraints: [node.role == manager] postgres: image: postgres:14 secrets: - psql-pw environment: POSTGRES_PASSWORD_FILE: /run/secrets/psql-pw POSTGRES_DB: drupal POSTGRES_USER: user deploy: placement: constraints: [node.role == manager] secrets: psql-pw: external: true secrets: psql-pw: external: true Understanding the Deploy Section The deploy section of the Docker Compose file specifies deployment strategies in a swarm environment, including: deploy Replicas: Defines the number of instances for the service. Update_config: Parallelism: Sets how many service tasks are updated simultaneously. Delay: Adds a delay between updates to different service tasks to ensure smooth roll-outs. Restart_policy: Condition: Determines under what conditions the service should be restarted. Replicas : Defines the number of instances for the service. Replicas Update_config : Update_config Parallelism : Sets how many service tasks are updated simultaneously. Parallelism Delay : Adds a delay between updates to different service tasks to ensure smooth roll-outs. Delay Restart_policy : Restart_policy Condition : Determines under what conditions the service should be restarted. Condition Managing Secrets in Docker Stack What is a Docker Secret? A Docker secret securely stores and manages sensitive data within Docker services, encrypted during transit and at rest, and only accessible by specifically authorized services. Managing Secrets with Docker Stack Secrets can be managed in Docker Stack in two ways: Using Secret Files: Ideal for development, storing sensitive data in local files securely transferred into Docker Swarm during deployment. secrets: psql-pw: file: ./your-file.txt External Secrets: For production, secrets are created directly in Docker using the CLI, avoiding the storage of sensitive information in files. secrets: psql-pw: external: true Using Secret Files : Ideal for development, storing sensitive data in local files securely transferred into Docker Swarm during deployment. Using Secret Files secrets: psql-pw: file: ./your-file.txt External Secrets : For production, secrets are created directly in Docker using the CLI, avoiding the storage of sensitive information in files. External Secrets secrets: psql-pw: external: true Hands-On: Deploying the Stack in Docker Swarm To deploy your stack in Docker Swarm, follow these steps: Step 1: Prepare Your Docker Environment Ensure Docker is running and connected to your Docker Swarm. Refer to this guide for setting up a 3-node Docker Swarm. this guide Step 2: Create and Edit the docker-compose.yml File On your swarm node, initialize and edit the docker-compose file to inclue our own docker-compose file: touch docker-compose.yml; vim docker-compose.yml; Step 3: Create the Secret echo "myverysecretpassword" | docker secret create psql-pw - Step 4: Deploy the Stack docker stack deploy -c docker-compose.yml demo-stack Step 5: Verify the Deployment List Services: docker stack services demo-stack List Tasks: docker stack ps demo-stack List Services : docker stack services demo-stack List Services docker stack services demo-stack List Tasks : docker stack ps demo-stack List Tasks docker stack ps demo-stack Step 6: View the Secrets in the Container Get inside the container docker exec -it [container-name] bash See the secret values cat /run/secrets/psql-pw Step 7: View Logs and Debug docker service logs drupal
docker service logs postgres Step 8: Play with your configs For example, you can scale up and down your services by editing the replicas values in your docker-compose file and re-deploy using the same deploy command. Step 9: Play with your Drupal app! Your drupal application should now be up and running on port 8080! Conclusion You have successfully configured a robust Docker Swarm with three nodes. This setup is now running efficient instances of Drupal and PostgreSQL, demonstrating Docker Stack’s powerful capabilities for real-world applications. Happy deploying!

An Introduction to Docker Stack: Deployment and Secrets Management

Stories, News & Mentions about DOCKER

Docker’s Stories on HackerNoon

Docker's News & Mentions