Cluster helps companies hire 30+ types of mechanical and electrical engineers for full-time roles

Cluster

cluster

Hello friends, The Oracle DBA occasionally needs to update some of the Oracle Clusterware network configurations. Today let's check how to update the private network part. The Environment Oracle RAC with 2 nodes: ol8-19-rac1 and ol8-19-rac2

Oracle Linux Server release 8.8

Grid Version
34318175;TOMCAT RELEASE UPDATE 19.0.0.0.0 (34318175)
34160635;OCW RELEASE UPDATE 19.16.0.0.0 (34160635)
34139601;ACFS RELEASE UPDATE 19.16.0.0.0 (34139601)
34133642;Database Release Update : 19.16.0.0.220719 (34133642)
33575402;DBWLM RELEASE UPDATE 19.0.0.0.0 (33575402)

DB Version
34086870;OJVM RELEASE UPDATE: 19.16.0.0.220719 (34086870)
34160635;OCW RELEASE UPDATE 19.16.0.0.0 (34160635)
34133642;Database Release Update : 19.16.0.0.220719 (34133642)

When you see <dbenv>, load the DB HOME variables.
When you see <gridenv>, load the GRID HOME variables.

IPs
# Public
192.168.56.161  ol8-19-rac1.localdomain  ol8-19-rac1
192.168.56.162  ol8-19-rac2.localdomain  ol8-19-rac2

# Virtual
192.168.56.163  ol8-19-rac1-vip.localdomain    ol8-19-rac1-vip
192.168.56.164  ol8-19-rac2-vip.localdomain    ol8-19-rac2-vip

# Private
10.1.2.161  ol8-19-rac1-priv.localdomain   ol8-19-rac1-priv
10.1.2.162  ol8-19-rac2-priv.localdomain   ol8-19-rac2-priv As the grid user, take a backup of profile.xml on all cluster nodes before proceeding. [oracle@ol8-19-rac1 ~]$ cp -p $ORACLE_HOME/gpnp/`hostname`/profiles/peer/profile.xml $ORACLE_HOME/gpnp/`hostname`/profiles/peer/profile.xml.bkp

[oracle@ol8-19-rac1 ~]$ ll $ORACLE_HOME/gpnp/`hostname`/profiles/peer/profile.xml.bkp

-rw-r--r--. 1 oracle oinstall 2171 May 27 20:09 /u01/app/19.0.0/grid/gpnp/ol8-19-rac1/profiles/peer/profile.xml.bkp Now, let’s check the Oracle CRS networks and the Linux networks. It’s possible to identify that Oracle is using the network as his private one. “eth2 10.1.2.0” [oracle@ol8-19-rac1 ~]$ oifcfg getif
eth1  192.168.56.0  global  public
eth2  10.1.2.0  global  cluster_interconnect,asm

[oracle@ol8-19-rac1 ~]$ ip -c -br a
lo               UNKNOWN        127.0.0.1/8 ::1/128
eth0             UP             10.0.2.15/24 fe80::a00:27ff:fe63:d73f/64
eth1             UP             192.168.56.161/24 192.168.56.163/24 192.168.56.166/24 192.168.56.165/24 fe80::a00:27ff:fe4f:b8aa/64
eth2             UP             10.1.2.161/24 169.254.1.211/19 fe80::a00:27ff:fe01:a7de/64
eth3             UP             10.1.3.161/24 fe80::a00:27ff:fef3:172b/64

[oracle@ol8-19-rac2 ~]$ ip -c -br a
lo               UNKNOWN        127.0.0.1/8 ::1/128
eth0             UP             10.0.2.15/24 fe80::a00:27ff:fe63:d73f/64
eth1             UP             192.168.56.162/24 192.168.56.164/24 192.168.56.167/24 fe80::a00:27ff:fe76:b954/64
eth2             UP             10.1.2.162/24 169.254.26.118/19 fe80::a00:27ff:fe7f:8663/64
eth3             UP             10.1.3.162/24 fe80::a00:27ff:fea8:d7c3/64 Let’s also check the ASM Listener configuration and take note. [oracle@ol8-19-rac1 ~]$ srvctl config listener -asmlistener
Name: ASMNET1LSNR_ASM
Type: ASM Listener
Owner: oracle
Subnet: 10.1.2.0
Home: <CRS home>
End points: TCP:1525
Listener is enabled.
Listener is individually enabled on nodes:
Listener is individually disabled on nodes:

[oracle@ol8-19-rac1 ~]$ srvctl status listener -l ASMNET1LSNR_ASM
Listener ASMNET1LSNR_ASM is enabled
Listener ASMNET1LSNR_ASM is running on node(s): ol8-19-rac2,ol8-19-rac1

[oracle@ol8-19-rac1 ~]$ srvctl config asm
ASM home: <CRS home>
Password file: +CRS/orapwASM
Backup of Password file: +CRS/orapwASM_backup
ASM listener: LISTENER
ASM instance count: 2
Cluster ASM listener: ASMNET1LSNR_ASM Firstly, let’s check if both nodes are able to reach the network we intend to use. [oracle@ol8-19-rac1 tmp]$ ping -c 3 10.1.3.161
PING 10.1.3.161 (10.1.3.161) 56(84) bytes of data.
64 bytes from 10.1.3.161: icmp_seq=1 ttl=64 time=0.010 ms
64 bytes from 10.1.3.161: icmp_seq=2 ttl=64 time=0.014 ms
64 bytes from 10.1.3.161: icmp_seq=3 ttl=64 time=0.018 ms

--- 10.1.3.161 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2052ms
rtt min/avg/max/mdev = 0.010/0.014/0.018/0.003 ms

[oracle@ol8-19-rac1 tmp]$ ping -c 3 10.1.3.162
PING 10.1.3.162 (10.1.3.162) 56(84) bytes of data.
64 bytes from 10.1.3.162: icmp_seq=1 ttl=64 time=0.168 ms
64 bytes from 10.1.3.162: icmp_seq=2 ttl=64 time=0.151 ms
64 bytes from 10.1.3.162: icmp_seq=3 ttl=64 time=0.171 ms

--- 10.1.3.162 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2068ms
rtt min/avg/max/mdev = 0.151/0.163/0.171/0.013 ms

[oracle@ol8-19-rac2 tmp]$ ping -c 3 10.1.3.161
PING 10.1.3.161 (10.1.3.161) 56(84) bytes of data.
64 bytes from 10.1.3.161: icmp_seq=1 ttl=64 time=0.153 ms
64 bytes from 10.1.3.161: icmp_seq=2 ttl=64 time=0.156 ms
64 bytes from 10.1.3.161: icmp_seq=3 ttl=64 time=0.176 ms

--- 10.1.3.161 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2081ms
rtt min/avg/max/mdev = 0.153/0.161/0.176/0.017 ms

[oracle@ol8-19-rac2 tmp]$ ping -c 3 10.1.3.162
PING 10.1.3.162 (10.1.3.162) 56(84) bytes of data.
64 bytes from 10.1.3.162: icmp_seq=1 ttl=64 time=0.010 ms
64 bytes from 10.1.3.162: icmp_seq=2 ttl=64 time=0.025 ms
64 bytes from 10.1.3.162: icmp_seq=3 ttl=64 time=0.015 ms

--- 10.1.3.162 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2037ms
rtt min/avg/max/mdev = 0.010/0.016/0.025/0.007 ms Since everything seems OK, now we will start changing the private network. We will change it from the subnet to . As the grid user, proceed as below in only one of the nodes. eth2:10.1.2.0 eth3:10.1.3.0 [oracle@ol8-19-rac1 ~]$ oifcfg setif -global eth3/10.1.3.0:cluster_interconnect,asm
[oracle@ol8-19-rac1 ~]$ srvctl add listener -asmlistener -l ASMNEWLSNR_ASM -subnet 10.1.3.0
[oracle@ol8-19-rac1 ~]$ srvctl update listener -listener ASMNET1LSNR_ASM -asm -remove -force

--The error below is expected, as we are only testing if the previous listener still exists.
[oracle@ol8-19-rac1 ~]$ lsnrctl stop ASMNET1LSNR_ASM

LSNRCTL for Linux: Version 19.0.0.0.0 - Production on 28-MAY-2023 07:55:09

Copyright (c) 1991, 2023, Oracle.  All rights reserved.

TNS-01101: Could not find listener name or service name ASMNET1LSNR_ASM Check if the ASM Listener was created correctly. [oracle@ol8-19-rac1 ~]$ srvctl config asm
ASM home: <CRS home>
Password file: +CRS/orapwASM
Backup of Password file: +CRS/orapwASM_backup
ASM listener: LISTENER
ASM instance count: 2
Cluster ASM listener: ASMNEWLSNR_ASM

[oracle@ol8-19-rac1 ~]$ srvctl config listener -asmlistener
Name: ASMNEWLSNR_ASM
Type: ASM Listener
Owner: oracle
Subnet: 10.1.3.0
Home: <CRS home>
End points: TCP:1526
Listener is enabled.
Listener is individually enabled on nodes:
Listener is individually disabled on nodes: In the 19c version, we also have one more resource to handle, the ASM Network. [oracle@ol8-19-rac1 ~]$ srvctl config asmnetwork
ASM network 1 exists
Subnet IPv4: 10.1.2.0//
Subnet IPv6:
Network is enabled
Network is individually enabled on nodes:
Network is individually disabled on nodes:

[oracle@ol8-19-rac1 ~]$ srvctl status asmnetwork
ASM network is running on ol8-19-rac2,ol8-19-rac1 As the root user, proceed as below. [root@ol8-19-rac1 scripts]# srvctl remove asmnetwork -netnum 1 -force
[root@ol8-19-rac1 scripts]# srvctl add asmnetwork -netnum 1 -subnet 10.1.3.0 Now let’s restart the entire cluster. [root@ol8-19-rac1 ~]# crsctl stop cluster -all -f
CRS-2673: Attempting to stop 'ora.crsd' on 'ol8-19-rac2'
CRS-2673: Attempting to stop 'ora.crsd' on 'ol8-19-rac1'
...
CRS-2673: Attempting to stop 'ora.cssd' on 'ol8-19-rac1'
CRS-2677: Stop of 'ora.cssd' on 'ol8-19-rac1' succeeded

[root@ol8-19-rac1 ~]# crsctl stop crs -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'ol8-19-rac1'
...
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'ol8-19-rac1' has completed
CRS-4133: Oracle High Availability Services has been stopped.

[root@ol8-19-rac2 ~]# crsctl stop crs -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'ol8-19-rac2'
...
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'ol8-19-rac2' has completed
CRS-4133: Oracle High Availability Services has been stopped.

[root@ol8-19-rac1 ~]# crsctl start crs
CRS-4123: Oracle High Availability Services has been started.

[root@ol8-19-rac2 ~]# crsctl start crs
CRS-4123: Oracle High Availability Services has been started. Check if everything is OK. [oracle@ol8-19-rac1 tmp]$ crsctl status res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.LISTENER.lsnr
               ONLINE  ONLINE       ol8-19-rac1              STABLE
               ONLINE  ONLINE       ol8-19-rac2              STABLE
ora.chad
               ONLINE  ONLINE       ol8-19-rac1              STABLE
               ONLINE  ONLINE       ol8-19-rac2              STABLE
ora.net1.network
               ONLINE  ONLINE       ol8-19-rac1              STABLE
               ONLINE  ONLINE       ol8-19-rac2              STABLE
ora.ons
               ONLINE  ONLINE       ol8-19-rac1              STABLE
               ONLINE  ONLINE       ol8-19-rac2              STABLE
ora.proxy_advm
               OFFLINE OFFLINE      ol8-19-rac1              STABLE
               OFFLINE OFFLINE      ol8-19-rac2              STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.ASMNEWLSNR_ASM.lsnr(ora.asmgroup)
      1        ONLINE  ONLINE       ol8-19-rac1              STABLE
      2        ONLINE  ONLINE       ol8-19-rac2              STABLE
ora.CRS.dg(ora.asmgroup)
      1        ONLINE  ONLINE       ol8-19-rac1              STABLE
      2        ONLINE  ONLINE       ol8-19-rac2              STABLE
ora.DATA.dg(ora.asmgroup)
      1        ONLINE  ONLINE       ol8-19-rac1              STABLE
      2        ONLINE  ONLINE       ol8-19-rac2              STABLE
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       ol8-19-rac2              STABLE
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       ol8-19-rac1              STABLE
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       ol8-19-rac1              STABLE
ora.RECO.dg(ora.asmgroup)
      1        ONLINE  ONLINE       ol8-19-rac1              STABLE
      2        ONLINE  ONLINE       ol8-19-rac2              STABLE
ora.asm(ora.asmgroup)
      1        ONLINE  ONLINE       ol8-19-rac1              Started,STABLE
      2        ONLINE  ONLINE       ol8-19-rac2              Started,STABLE
ora.asmnet1.asmnetwork(ora.asmgroup)
      1        ONLINE  ONLINE       ol8-19-rac1              STABLE
      2        OFFLINE OFFLINE                               STABLE
ora.cdbrac.db
      1        ONLINE  ONLINE       ol8-19-rac1              Open,HOME=/u01/app/o
                                                             racle/product/19.0.0
                                                             /dbhome_1,STABLE
      2        ONLINE  ONLINE       ol8-19-rac2              Open,HOME=/u01/app/o
                                                             racle/product/19.0.0
                                                             /dbhome_1,STABLE
ora.cvu
      1        ONLINE  ONLINE       ol8-19-rac1              STABLE
ora.ol8-19-rac1.vip
      1        ONLINE  ONLINE       ol8-19-rac1              STABLE
ora.ol8-19-rac2.vip
      1        ONLINE  ONLINE       ol8-19-rac2              STABLE
ora.qosmserver
      1        ONLINE  ONLINE       ol8-19-rac1              STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       ol8-19-rac2              STABLE
ora.scan2.vip
      1        ONLINE  ONLINE       ol8-19-rac1              STABLE
ora.scan3.vip
      1        ONLINE  ONLINE       ol8-19-rac1              STABLE
-------------------------------------------------------------------------------- In my case, the ASM Network was not running on node 2, so I started it manually, and I will check in future reboots if I face the same issue again. [oracle@ol8-19-rac2 tmp]$ srvctl start asmnetwork

[oracle@ol8-19-rac2 tmp]$ srvctl status asmnetwork
ASM network is running on ol8-19-rac2,ol8-19-rac1 Since everything is running smoothly, let’s remove the previous network from the Oracle CRS. As the grid user, proceed as below. [oracle@ol8-19-rac1 tmp]$ oifcfg delif -global eth2/10.1.2.0

[oracle@ol8-19-rac1 tmp]$ oifcfg getif
eth1  192.168.56.0  global  public
eth3  10.1.3.0  global  cluster_interconnect,asm In the next article, I will explain how to update the IP from the other layers of the Oracle CRS. That’s it!

How to Modify Private Network Information in Oracle Clusterware 19c

A Raspberry Pi is a mini-computer board to which you can connect a monitor, mouse, and keyboard, and install a Linux-based operating system with a GUI. Or you can use it in “headless” mode with no GUI and run, for example, a database server. There are you can give a Raspberry Pi—from building a Minecraft server to smart mirrors, the possibilities are endless. many usages Since I started to discover MariaDB and learned about database clusters, , and , the idea of building a Raspberry Pi cluster has been in the back of my head. A cluster like this is a great way to experiment with distributed systems. Distributed SQL Xpand In this article, I’ll show you how to build a Raspberry Pi cluster with: 8 nodes 32 cores 64 GB of RAM 2TB of storage What You Need If you want to build your own cluster, you don’t have to buy 8 Raspberry Pi devices. Even one device is enough to learn stuff. The instructions in this article are useful even if you plan to configure only one My advice, however, is to have at least three devices to build your cluster if you truly want to boost your skills in administration, , , , … you name it! Raspberry Pi. Linux Ansible Docker Kubernetes database clusters With that in mind, please replace 8 with whatever is your number. Do the same with the specifications of the Raspberry Pi devices if you have different models with less or more RAM or storage. Here’s what I used: 8 x (8 GB RAM) Raspberry Pi 4 Model B 1 x (10-Ports) Anker 60W Desktop Charger 8 x (5A, 40W, 23cm) Short USB Type C cable 8 x (256 GB) MicroSD card 2 x (4 layers with cooling fan) Raspberry Pi cluster case Adapt the quantities to your setup. You can either get one or more if you have very few devices (say 1 or 2) or a multi-USB charger. If you go with a multi-USB charger, make sure that each port can deliver at least 2.4Am (5V). Pick microSD cards that fit your goals and budget as well. I recommend going with at least 32 GB SD cards. Raspberry Pi power supplies On the cables and case side, pick anything that accommodates your setup. You don’t necessarily need cooling fans, but I recommend them if you plan to leave your devices on for prolonged periods of time. I also recommend getting all the ingredients before you start “cooking," especially if you want to use multiple Raspberry Pi devices. Installing Raspberry Pi OS (Headless) The operating system (OS) we’ll use is , a Debian-based OS optimized for Raspberry Pi boards. Raspberry Pi OS is installed on the microSD cards using your computer. You later connect these microSD cards to your Raspberry Pi devices and boot them. Raspberry Pi OS Get all the microSD cards and Raspberry Pi boxes on your desk. You’ll need an SD card adapter (they usually come with microSD cards) or an SD card USB reader if your working computer doesn’t have a slot for SD cards. Bring a Sharpie as well. Download the application, and install it on your computer. Take a microSD card and connect it to your computer. Open the Raspberry Pi Imager application, and click on . From the list, click on and select if your Raspberry Pi devices are 64-bit or otherwise: Raspberry Pi Imager CHOOSE OS Raspberry Pi OS (other), Raspberry Pi OS Lite (64-bit) Raspberry Pi OS Lite (32-bit) Click on and select the microSD card. Double-check that you selected the correct drive, and click on the “gear” icon (advanced options). For hostname use , or something similar. You’ll be naming the devices as , , , etc. CHOOSE STORAGE, rpi01 rpi01 rp02 rp03 Check the and options. Set a username (I’ll leave the default ), and set a secure password. Enable SSH, Use password authentication pi Check the option, and enter the name and password of your WiFi connection. Configure your locale settings as well. Configure wireless LAN Since we have to do this once with each microSD card, make sure to set the options field to: . This way the settings will be saved, and the process will be easier for the next cards: Image customization to always use Now, you can click and then to start the process. Once completed, eject the card and connect it to one of the Raspberry Pi devices, put it back into its box, and mark the box with the number . Repeat this process for all of the devices. SAVE WRITE 01 Assembling the Cluster Depending on what kind of cluster case you have, the assembling process might vary. Follow the instructions that come with your case, or look for photographs online to get a clear idea of how the devices should be placed in the layers. I recommend looking ahead to decide which parts to assemble first. For example, with the case I picked, I had to install the cooling fans before mounting the Raspberry Pi devices on the layers. Once you mount a Raspberry Pi, use a Sharpie to write its number on the layer. This will help you in the future if you have to take a microSD card out for reconfiguration or any other administrative tasks that involve touching the hardware. Once you have all the devices mounted, connect the USB cables to the Raspberry Pi devices and the multi-USB charger. You can optionally use and a instead of the USB charger if you prefer, but I’ll leave that for you to explore; it’s definitely worth checking. Power over Ethernet network switch You can either have the charger (or network switch if you go for it) next to your Raspberry Pi rack or attached to it. I used rubber bands to keep the charger attached to the side of the case. It worked fine. With the USB cables plugged in, you are ready to start all those mini-computers! Connect the charger, and switch it on. Depending on the Raspberry Pi models that you use, they might take some time to boot, so be patient. Meanwhile, enjoy the LEDs flashing and fans starting. Connecting Through SSH If everything went well, you should be able to reach the Raspberry Pi devices through SSH. Give it a try by running the following from your computer: ssh pi@rpi01.local Introduce your password, and answer to add the device to the list of known hosts. Congrats! Your cluster is live now. yes You may want to run commands to check the hardware information. For example: : Information about the CPU architecture lscpu : File system disk space usage df -H : Partition information sudo fdisk -l : Amount of used, free and total amount of RAM free -m : Linux kernel information cat /proc/version Manual Configuration With raspi-config If you have one Raspberry Pi, you can connect to it through SSH as described above, and use the utility program to configure various settings. You can modify the WiFi connection ( ), change the hostname ( ) and user password ( ), and expand the filesystem ( ), among many other things. raspi-config S1 S4 S3 A1 Automated Configuration With Ansible If you have more than one Raspberry Pi in your cluster, you might want to automate the configuration process using a tool like . With Ansible, you can run a command on multiple machines without having to perform repetitive tasks. For example, let’s say we want to expand the filesystem on each Raspberry Pi. This can be done from the command line as follows: Ansible sudo raspi-config --expand-rootfs Without automation, you’d have to SSH to , run the command above, and end the SSH session. You’d have to repeat all these steps for , , , and so forth. Instead, you can simply tell Ansible to run the command for you on all the machines. Let’s see how to do this. rpi01.local rpi02.local pr03.local rp04.local You have to install Ansible on a that is connected to your local network. It can be one of the Raspberry Pi devices, your working laptop, or any other machine as long as it runs a Unix-based operating system like Linux or macOS. control node I happen to have an old laptop that I connected to my network, so I used it as the control node (I’ll show how to install a database with replication in the Raspberry Pi devices in a future article). repurposed as a dedicated database I won’t go through the instructions on installing Ansible since the process is different on different operating systems. Check the and install it on your laptop or any device that you want to assign as a control node. official documentation Before you start, I recommend generating an example configuration file that you can tweak later as you wish: sudo su
ansible-config init --disabled -t all > /etc/ansible/ansible.cfg
exit In this file, disable host key checking to simplify the process. Don’t do that in production environments! Change the following line: ;host_key_checking=True to this: host_key_checking=False Next, you need to define an inventory. This is a list of machines that you want to control with Ansible. The inventory is defined in the file. For example, you could add the following machines to the inventory using their IP addresses or hostnames: /etc/ansible/hosts [mymachines]
foo.example.com
bar.example.com
192.0.2.50
192.0.2.51 If the machines have some kind of pattern in their IP addresses or hostnames, you can alternatively use ranges. This is what you can do with your Raspberry Pi devices. Edit the file by adding the following to the end: /etc/ansible/hosts [rpis]
rpi[01-08].local Change the pattern to match your hostnames and numbers. This is equivalent to: [rpis]
rpi01.local
rpi02.local
rpi03.local
rpi04.local
rpi05.local
rpi06.local
rpi07.local
rpi08.local In this inventory, is an arbitrary name that you can use to refer to all the Raspberry Pi devices when running commands on them using Ansible. rpis You need to configure the SSH username that Ansible will use when connecting to the machines. Add the following to the file: /etc/ansible/hosts [rpis:vars]
ansible_user=pi Time to control de machines. A good start is to them: ping ansible rpis -m ping --ask-pass Type the SSH password. You should see in the output how the “ping” is answered by a “pong” for each machine (I’m showing only one here): rpi01.local | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python3"
    },
    "changed": false,
    "ping": "pong"
} Let’s back to the filesystem expansion (although, I think nowadays this is not required anymore, let me know in the comments if you can confirm this). To perform this action on all the machines, run: ansible rpis -m shell -a "raspi-config --expand-rootfs" --become --ask-pass And as simple as that, all your Raspberry Pi devices have an expanded filesystem. This is a good time to update the system on all the machines: ansible rpis -m shell -a "apt update -y" --become --ask-pass
ansible rpis -m shell -a "apt upgrade -y" --become --ask-pass You can reboot all the devices as follows: ansible rpis -m shell -a "reboot" --become --ask-pass And once you are done and want to safely turn all the machines off, just run: ansible rpis -m shell -a "poweroff" --become --ask-pass What's Next? This is only the beginning. You now have a cluster of small computers ready to be controlled by Ansible. You can try setting up any kind of server software, including web servers or databases ( ). There also is much more about Ansible. see this example I merely scratched the surface here using , but you can for example create that contain the state in which you want your cluster to be. Explore the , and have fun! ad-hoc commands playbooks documentation Also published here

How to Build a 32-Core Raspberry Pi Cluster From Scratch

Cómo construir un clúster Raspberry Pi de 32 núcleos desde cero

Comment construire un cluster Raspberry Pi à 32 cœurs à partir de zéro

स्क्रैच से 32-कोर रास्पबेरी पाई क्लस्टर कैसे बनाएं

32 コアの Raspberry Pi クラスターをゼロから構築する方法

Como construir um cluster Raspberry Pi de 32 núcleos a partir do zero

Cách xây dựng cụm Raspberry Pi 32 lõi từ Scratch

如何从零开始构建 32 核 Raspberry Pi 集群

is an open source system for automatically deploying, scaling, and managing containerized applications. Containerization is a lightweight alternative to full machine virtualization that involves encapsulating an application in a container with its own operating environment. Containerization has matured over the past few years and has led to greater adoption of Kubernetes across the tech industry. Kubernetes Kubernetes greatly increases a user’s ability to run testing and simulations. At a high level, it allows users to expand their technical resources outside of their traditional limits; for example, users can simulate a 101-node blockchain cluster without needing to own 101 physical machines. This is ideal for quickly standing up staging environments for a blockchain cluster. It is also particularly useful for simulating blockchain networks with proof of stake (PoS) or delegated proof of stake (DPoS) consensus mechanisms, which involve a finite amount (e.g., 101) of geographically distributed block producing / verifying nodes. Starting with our , IoTeX has used Kubernetes to deploy and optimize our Testnet infrastructure. Overall, Kubernetes has greatly improved our operations — in this blog, we share our experience and some tips to run your own containerized applications and environments on Kubernetes. Testnet Beta “Epik” release Setting up the IoTeX cluster on Kubernetes We begin by defining two type of Kubernetes : services used to boot up the whole IoTeX cluster. It is the entry point of any new iotex nodes and helps new nodes join the P2P network. All nodes identify each other through their internal Pod’s IP address. bootnode service: exposes the JSON-RPC API from all iotex nodes. The JSON-RPC interface is used by our and our action injection tool. iotex service: open source IoTeX Explorer All node configurations are deployed through the Kubernetes , including the genesis block. This allows us to redeploy a new cluster without waiting for a new docker image to be built. ConfigMap Applications running in Kubernetes Pods are stateless by default, which means saved data from any applications will not be available after a redeployment. To counteract this, we use the following setup process. In our scenario each IoTeX node will utilize the same copy of data (i.e., distributed ledger), so it is not necessary to persist every nodes’ data. Therefore, we mount the Kubernetes to only one iotex node — in our case, we mounted it to our boot node. We also back up our persisted data to object storage spaces like S3 and Digital Ocean. Since all nodes except the boot node will not have saved data after a redeployment, they will need to first download the backup before startup. We use init container to achieve this. This process enables our cluster to restart from a crash our failure without worrying about losing previous data. Persistent Volume Cluster Monitoring It is not trivial to track the status of 21 nodes (let alone 101 nodes down the road), and it becomes more complex with a larger amount of nodes. To have better observability within our IoTeX cluster, we set up monitoring capabilities in Kubernetes, which consists of a logging stack, metrics stack, and alert manager. For the logging stack, we use + + + . Fluentd sends logs from each node to the Elastic Search client, and the Elastic Search master server indexes the logs which are query-able through Kibana. Finally, we use the Elastic Search Curator to clean up the outdated logs. Fluentd Elastic Search Kibana Elastic Search Curator Refer from Log aggregation with ElasticSearch, Fluentd and Kibana stack on ARM64 Kubernetes cluster For the metrics stack, we use + + . provides an easy way to configure Prometheus running in a Kubernetes cluster. We can hook up the IoTeX node’s Prometheus client with the Prometheus server by simply exposing a metrics service. Prometheus Prometheus Operator Grafana CoreOS Prometheus Operator Refer from CoreOS: Prometheus Operator We are also working on setting up alerts with Prometheus Alert Manager to send out alerts to our on-call engineer when we have abnormal metrics. Using Helm to Manage Multiple Configurations Using yaml configuration to deploy different types of applications on Kubernetes is fairly straightforward for a single environment. However, when you have multiple environments (e.g., testing, staging, production) and different node twists / configurations (e.g., 21 vs. 101 delegates), users encounter higher overhead and redundancy to manage various Kubernetes yaml configurations for the same application. To solve this issue, we use Helm. is a package manager for Kubernetes applications. It allows us to create our IoTeX cluster as a chart package with version and default values. With Helm, we don’t need to manage duplicate Kubernetes yaml files; rather, we only need to manage a smaller subset of configurations for different environments and twists which overwrite the default value. It also simplifies the command we need to run to start a fresh IoTeX cluster into one single command: Helm Helm also allows us to publish a chart easily. In the future, we will publish our IoTeX Helm chart to the community, so you can setup an IoTeX cluster within minutes. More to come! With the power of Kubernetes and other operational tools, IoTeX’s Development team not only saves time on cluster operation, but also digests issues quicker and iterates even faster than before. We look forward to sharing more technical perspectives in the future — feel free to reach out with any questions to support@iotex.io. About IoTeX IoTeX is the world’s first privacy-centric blockchain platform that is fast, flexible, and Internet of Things (IoT) friendly. IoTeX’s global team is comprised of Ph.Ds in Cryptography, Distributed Systems, and Machine Learning, top tier engineers, and experienced ecosystem builders. Designed and optimized for IoT, IoTeX uses state-of-the-art privacy, consensus, and subchain innovations to capture the full potential of IoT. By enabling trusted data, interoperability, and M2M automation, IoTeX connects the physical and digital worlds and brings trusted machine economies to the masses. Stay connected with us! Website: Twitter: Telegram Announcement Channel: Telegram Group: Medium: Reddit: Join us: https://iotex.io/ https://twitter.com/iotex_io https://t.me/iotexchannel https://t.me/IoTeXGroup https://medium.com/@iotex https://www.reddit.com/r/IoTeX/ https://iotex.io/careers

Deploy, operate, and monitor IoTeX cluster with Kubernetes

This post is part of “ ” series explaining how to use concepts with . In this part, I will show you how to setup an cluster on using & as shown in the diagram below : IaC Infrastracture as Code Terraform etcd AWS CoreOS Terraform All the templates used in this demo can be found on my . Github So let’s start with “ ” file which contains the global variables such as AWS region, cluster instances type … variables.tf Note: As of writing this article, the latest stable version is . CoreOS 1465.6.0 So make sure to find an that is as close to the latest version as possible. AMI Next, we need to define a for our cluster. For simplicity, Im going to make this open to the world. Even though security is important, this tutorial serves an educational purposes and you should never have all ports open in production. security group security group And finally, we will define our which consists of : cluster 3 Nodes In order to bring up an , I used a file that I passed as a parameter to attribut: etcd cluster cloud config user_data Note: Make sure to grab the and place it into the discovery parameter: discovery token, $ curl https://discovery.etcd.io/new?size=3 https://discovery.etcd.io/3e86b59982e49066c5d813af1c2e2579cbf573de Once you defined all templates required, just type the following command to bring up the : etcd cluster terraform apply Note: Don’t forget to set the as an before: AWS credentials envrionment variables export AWS_ACCESS_KEY_ID=”YOUR ACCESS KEY ID” export AWS_SECRET_ACCESS_KEY=”YOUR SECRET ACCESS KEY” Setting up an in action is shown below: etcd cluster Once done, go to your then navigate to your : AWS Management Console EC2 Dashboard Congratulations ! You have your . CoreOS cluster To verify the c , you can either point your browser to the discovery url you generated earlier: luster health or to one of your cluster nodes using the command: SSH ssh core@<NODE-IP> Then, use the to fetch the : etcd command line cluster status Now we have an ready to use. Let’s see what we can do with it: etcd cluster Through etcdctl: etcdctl set nginx/port 80 etcdctl get nginx/port 80 etcdctl ls nginx etcdctl rm nginx/port Through HTTP API: curl -sS -X PUT -d value=”80" | jq ‘.’ # Create http://localhost:2379/v2/keys/nginx/port curl -sS | jq ‘.’ # Get http://localhost:2379/v2/keys/nginx/port curl -sS -X DELETE | jq ‘.’ # Delete http://localhost:2379/v2/keys/nginx/port

Setting up an etcd cluster on AWS using CoreOS & Terraform

In this post, I'm going to show you how to potentially triple your application's performance by managing multiple threads. This is an important tutorial, where the methods and the examples shown will give you what you need to set up production-ready thread management. Node Watch The Video on YouTube Child Processes, Clustering, and Worker Threads For the longest time, Node's had the ability to be multi-threaded, by using either , , or the more recent preferred method of a module called . Child Processes Clustering Worker Threads Child processes were the initial means of creating multiple threads for your application and have been available since version 0.10. This was achieved by spawning a node process for every additional thread you wanted to be created. Clustering, which has been a stable release since around version 4, allows us to simplify the creation and management of Child Processes. It works brilliantly when combined with . PM2 Now before we get into multithreading our app, there are a few points that you need to fully understand: 1. Multithreading already exists for I/O tasks There is a layer of Node that's already multithreaded and that is the thread-pool. I/O tasks such as files and folder management, TCP/UDP transactions, compression, and encryption are handed off to libuv, and if not asynchronous by nature, get handled in the libuv's thread-pool. libuv 2. Child Processes/Worker Threads only work for synchronous JavaScript logic Implementing multithreading using Child Processes or Worker Threads will only be effective for your synchronous JavaScript code that's performing heavy-duty operations, such as looping, calculations, etc. If you try to offload I/O tasks to Worker Threads as an example, you will not see a performance improvement. 3. Creating one thread is easy. Managing multiple threads dynamically is hard Creating one additional thread in your app is easy enough, as there are tons of tutorials on how to do so. However, creating threads equivalent to the number of logical cores your machine or VM is running and managing the distribution of work to these threads is way more advanced, and to code this, logic is above most of our pay grades 😎. Thank goodness we are in a world of open source and brilliant contributions from the Node community. Meaning, there is already a module that will give us the full capability of dynamically creating and managing threads based on the CPU availability of our machine or VM. Worker Pool The module we will work with today is called . Created by , Worker Pool offers an easy way to create a pool of workers for both dynamically offloading computations as well as managing a pool of dedicated workers. It's basically a thread-pool manager for Node JS, supporting Worker Threads, Child Processes, and Web Workers for browser-based implementations. Worker Pool Jos de Jong To make use of the Worker Pool module in our application, the following tasks will need to be performed: Install Worker Pool First, we need to install the Worker Pool module - npm install workerpool Init Worker Pool Next, we'll need to initialize the Worker Pool on the launch of our App Create Middleware Layer We'll then need to create a middleware layer between our heavy-duty JavaScript logic and the Worker Pool that will manage it Update Existing Logic Finally, we need to update our App to hand off heavy-duty tasks to the Worker Pool when required Managing Multiple Threads Using Worker Pool At this point, you have 2 options: Use your own NodeJS app (and install and modules), or download the from GitHub for this tutorial and my video series. workerpool bcryptjs source code NodeJS Performance Optimization If going for the latter, the files for this tutorial will exist inside the folder . Once downloaded, enter into the root project folder and run npm install. After that, enter into the folder to follow along. 06-multithreading 06-multithreading In the folder, we have 2 files: one is the controller logic for the Worker Pool (controller.js). The other holds the functions that will be triggered by the threads…aka the middleware layer I mentioned earlier (thread-functions.js). worker-pool worker-pool/controller.js WorkerPool = ( ) Path = ( ) poolProxy = init = (options) => { pool = WorkerPool.pool(Path.join(__dirname, ), options)
  poolProxy = pool.proxy() .log( )
} get = { poolProxy
} exports.init = init
exports.get = get 'use strict' const require 'workerpool' const require 'path' let null // FUNCTIONS const async const './thread-functions.js' await console `Worker Threads Enabled - Min Workers: - Max Workers: - Worker Type: ` ${pool.minWorkers} ${pool.maxWorkers} ${pool.workerType} const => () return // EXPORTS The controller.js is where we require the module. We also have 2 functions that we export, called and . The function will be executed once during a load of our application. It instantiates the Worker Pool with options we'll provide and a reference to the . It also creates a proxy that will be held in memory for as long as our application is running. The function simply returns the in-memory proxy. workerpool init get init thread-functions.js get worker-pool/thread-functions.js WorkerPool = ( ) Utilities = ( ) bcryptHash = { Utilities.bcryptHash(password)
} WorkerPool.worker({
  bcryptHash
}) 'use strict' const require 'workerpool' const require '../2-utilities' // MIDDLEWARE FUNCTIONS const ( ) => password return // CREATE WORKERS In the file, we create worker functions that will be managed by the Worker Pool. For our example, we're going to be using to hash passwords. This usually takes around 10 milliseconds to run, depending on the speed of one's machine, and makes for a good use case when it comes to heavy duty tasks. Inside the file is the function and logic that hashes the password. All we are doing in the thread-functions is executing this via the workerpool function. This allows us to keep code centralized and avoid duplication or confusion of where certain operations exist. thread-functions.js BcryptJS utilities.js bcryptHash 2-utilities.js BCrypt = ( ) bcryptHash = (password) => { BCrypt.hash(password, )
}

exports.bcryptHash = bcryptHash 'use strict' const require 'bcryptjs' const async return await 8 .env NODE_ENV= PORT= WORKER_POOL_ENABLED= "production" 6000 "1" The .env file holds the port number and sets the variable to "production". It's also where we specify if we want to enable or disable the Worker Pool, by setting the to "1" or "0". NODE_ENV WORKER_POOL_ENABLED 1-app.js ( ).config() Express = ( ) App = Express() HTTP = ( ) Utilities = ( ) WorkerCon = ( ) App.get( , (req, res) => { password = result = workerPool = (process.env.WORKER_POOL_ENABLED === ) {
    workerPool = WorkerCon.get()
    result = workerPool.bcryptHash(password)
  } {
    result = Utilities.bcryptHash(password)
  }

  res.send(result)
}) port = process.env.PORT server = HTTP.createServer(App)

; { (process.env.WORKER_POOL_ENABLED === ) { options = { : } WorkerCon.init(options)
  } server.listen(port, () => { .log( , port)
  })
})() 'use strict' require 'dotenv' const require 'express' const const require 'http' const require './2-utilities' const require './worker-pool/controller' // Router Setup '/bcrypt' async const 'This is a long password' let null let null if '1' await else await // Server Setup const const ( ) => ( async // Init Worker Pool if '1' const minWorkers 'max' await // Start Server console 'NodeJS Performance Optimizations listening on: ' Finally, our holds the code that will be executed on the launch of our App. First, we initialize the variables in the file. We then set up an server and create a route called . When this route is triggered, we will check to see if the Worker Pool is enabled. If yes, we get a handle on the Worker Pool proxy and execute the function that we declared in the file. This will in turn execute the function in and return us the result. If the Worker Pool is disabled, we simply execute the function directly in . 1-app.js .env Express /bcrypt bcryptHash thread-functions.js bcryptHash Utilities bcryptHash Utilities At the bottom of our , you'll see we have a self-calling function. We're doing this to support async/await, which we are using when interacting with the Worker Pool. Here is where we initialize the Worker Pool if it's enabled. The only config we want to override is setting the to "max". This will ensure that the Worker Pool will spawn as many threads as there are logical cores on our machine, with the exception of 1 logical core, which is used for our main thread. In my case, I have 6 physical cores with hyperthreading, meaning I have 12 logical cores. So with set to "max", the Worker Pool will create and manage 11 threads. Finally, the last piece of code is where we start our server and listen on port 6000. 1-app.js minWorkers minWorkers Testing the Worker Pool Testing the Worker Pool is as simple as starting the application and while it's running, preforming a get request to . If you have a load testing tool like , you can have some fun seeing the difference in performance when the Worker Pool is enabled/disabled. AutoCannon is very easy to use. http://localhost:6000/bcrypt AutoCannon Conclusion I hope this tutorial has provided insight into managing multiple threads in your Node application. The embedded video at the top of this article provides a live demo of testing the Node App. Till next time, cheers :) Previously published at http://bleedingcode.com/managing-multiple-threads-nodejs/

How to Manage Multithreaded Node JS Applications for Better Performance

Cómo administrar aplicaciones JS de nodo multiproceso para un mejor rendimiento

Introduction With ever-increasing demands from other business units, IT departments have to be constantly looking for service improvements and cost-saving opportunities. This article showcases several concrete use-cases for companies that are investigating or already using Kafka, in particular, Kafka Connect. Kafka Connect is an enterprise-grade solution for integrating a plethora of applications, ranging from traditional databases to business applications like Salesforce and SAP. Possible integration scenarios range from continuously streaming events and data between applications to large-scale, configurable batch jobs that can be used to replace manual data transfers. Why Kafka and Heroku started as an in-house project at LinkedIn intended to replace the various queuing and message-passing systems that had evolved over the years. The goal was to create a horizontally scalable system, able to reliably store and handle many millions of messages per second. Today, the Kafka community provides a whole ecosystem of tools and features ranging from simple log-message passing to performing basic on-the-fly ETL jobs to complex data-pipeline processing. Its versatility and robustness in addition to its scalability make Kafka a great choice for the enterprise. Apache Kafka The power of Kafka comes at a price: While it's easy to use Kafka from a client perspective, the setup and operation of Kafka is a difficult task. Setting up a reliable Kafka cluster is a challenge that requires experience. If your company does not have the experience, it can take weeks, even months to set up and fine-tune a production-ready cluster. That's where Kafka on Heroku shines: , which makes it feasible for IT departments to focus on the services they want to provide instead of operating a Kafka cluster. Setting up a private Kafka cluster literally takes minutes Adding Kafka Connect is the logical next step, once you are using Kafka. But is a huge time-saver, even if you are not planning to use Kafka directly from your application. With little operational overhead, Kafka Connect provides an integration point for any number of applications. Instead of running and maintaining different solutions for recurring data transfer, on-demand data transfer, and continuous updates between systems, Kafka Connect provides a unified solution to those problems. Kafka Connect on Heroku In the next section you will learn about some typical use cases, and also find out which applications Kafka Connect is able to integrate. Typical Business Use Cases for Kafka Connect IT departments constantly struggle with requirements from business units that must connect and integrate all kinds of custom and off-the-shelf applications. These integrations often use less-reliable mechanisms, like recurring batch jobs, instead of continuously "streaming" updates across systems. Kafka Connect provides a scalable, reliable solution that can often replace slow batch jobs by instead "streaming" updates. Common scenarios include: Your sales department regularly needs to update an internal SQL database with data from Salesforce. Use Kafka Connect to continuously stream the changes from Salesforce to your on-premise database. Your compliance department needs an audit trail for updates to certain databases or applications. Use Kafka Connect to stream the relevant changes to an on-premise file server, to S3 for offsite storage, or even to Elasticsearch for quick retrieval. Replace your unreliable, slow-running ETL batch jobs with a robust solution. Use Kafka Connect to: update data across multiple applications while making them faster and more reliable; continuously stream changes from structured files (CSV, JSON, logs, etc.) into databases and applications; reliably and continuously stream updates between heterogeneous applications and databases IT departments could even go as far as creating new self-service opportunities, where manual or labor-intensive requests for data transfer or extraction are replaced by self-service processes that reduce the IT department's workload while improving response time for business units. Consider a system in which your company's marketing department creates a ticket in your service-desk software, which automatically creates and triggers a data-transfer that otherwise the IT service desk would have had to handle manually. An (Incomplete) List of Systems That Kafka Connect Can Integrate Kafka Connect can an ever-increasing number of applications and services, including enterprise applications, like SAP and Salesforce: read from and write to any SQL database: Oracle, Microsoft SQL Server, IBM Db2, MySQL, PostgreSQL, MariaDB, and others NoSQL databases like MongoDB, Redis, Cassandra, InfluxDB, and others message queues like ActiveMQ, RabbitMQ, IBM MQ, Amazon SQS, Azure Event Hub, Google Pub/Sub, and others IT service-desk applications like Jira, ServiceNow, and Zendesk logging and monitoring services, applications, and protocols like SNMP, Syslog, Elasticsearch, Amazon CloudWatch, Appdynamics, Metrics, Splunk, and others generic "low-level" data sources and protocols, including HTTP, SFTP, MQTT, JMS, the file system, and others robust custom connectors can be easily written using Java, taking full advantage of the reliable Kafka Connect framework and the underlying infrastructure since Kafka Connect uses Kafka as the underlying infrastructure, any application that uses Kafka as a message bus is automatically available for integration Setting Up a Kafka Connect Integration Is Fast and Easy Let's take a concrete example. A common integration scenario is this: You have two SQL databases and you need to update one database with information from the other database. For example, when the status of order changes in your fulfillment system, you might need to update the internal CRM database to reflect this change. Here is the concrete scenario: whenever the STATUS table changes in the fulfillment database write the changes to the ORDER_STATUS table in the CRM database For simplicity, we'll assume both are MySQL databases. The steps for other connectors would be the same, only with different configurations. It takes two steps to set up this integration, assuming you have a working Kafka Cluster with Kafka Connect: create the "source-connection" to read data from the STATUS table of fulfillment database create the "sink-connection" to write data to the ORDER_STATUS table of the CRM database Here are the corresponding cURL calls, and an explanation of what each of them does. They use the to create the source and sink. Note that these calls are not specific to Heroku. They will work with any Kafka Connect installation: Kafka Connect REST API 1. Creating the source-connection. curl -X POST : , : { : , : , : , : , : , : , : , : , : }   
} http: //localhost:8083/connectors -H "Content-Type: application/json" -d '{ "name" "jdbc_source_mysql_01" "config" "connector.class" "io.confluent.connect.jdbc.JdbcSourceConnector" "connection.url" "jdbc:mysql://fulfillmentdbhost:3306/fulfillmentdb" "connection.user" "fullfilment_user" "connection.password" "<password>" "topic.prefix" "order-status-update-" "mode" "timestamp" "table.whitelist" "fulfullmentdb.status" "timestamp.column.name" "LAST_UPDATED" "validate.non.null" false ' 2. Creating the sink-connection. curl -X POST : , : { : , : , : , : , : , : }
} http: //localhost:8083/connectors -H "Content-Type: application/json" -d '{ "name" "jdbc_sink_mysql_01" "config" "connector.class" "io.confluent.connect.jdbc.JdbcSinkConnector" "connection.url" "jdbc:mysql://crmdbhost:3306/crmdb" "connection.user" "crm_user" "connection.password" "<password>" "topics" "order-status-update-status" "table.name.format" "crmdb.order_status" ' And you're done! From this point on, any new entries in the STATUS table of the fulfillment database will automatically be written to the ORDER_STATUS table of the CRM database. Let's dive a little into what each call is doing. cURL call no. 1: Connecting the Source The first cURL command tells Kafka Connect to use a specific type of source connector, namely JdbcSourceConnector, to connect to the MySQL database at fulfillmentdbhost:3306/fulfillmentdb using the supplied credentials. It also configures this connector to monitor the table called "status" in that database for new entries based on the timestamp in the column "LAST_UPDATED." As a result, whenever a new row is written to the table, Kafka Connect will automatically write a new message into the Kafka topic "order-status-update-status" for each new row. Next, we need a sink connector that monitors this Kafka topic for changes. That's what the second cURL call does. cURL call no. 2: Connecting the Sink The second cURL command creates a sink connector (JdbcSinkConnector) that connects to a different database, namely crmdbhost:3306/crmdb. This connector will monitor the topic "order-status-update-status" for new messages and will write any new messages to the table "order_status" in the crmdb-database. This example uses the JdbcSourceConnector and JdbcSinkConnector to connect two relational databases. Other types of connectors can be installed and then used in the same way. Each connector comes with its own documentation and set of options. An unofficial list of available connectors with links to their documentation can be found on the . Confluent website Taking It to the Next Level You have seen how to set up a continuously streaming integration between two SQL databases with just two commands. In a similar fashion, you could integrate any of the applications that Kafka Connect provides connectors for -- or even write your own. Just set up a source and a sink connector and you are ready to stream your updates. But this is not all. Once you have set up a source connector, you can use the same source to update multiple destinations ("sinks"). In our example, you could run a third cURL command to write all order-status updates to another database as well. Conclusion If your team faces any of the problems described in this article, you should give Kafka Connect a try. Kafka Connect adds a whole new set of capabilities to an existing Kafka cluster, that will make your team's life easier in the long run. And if you are currently not running a Kafka cluster, this article has shown you how to set it up within minutes. Once Kafka Connect is up and running, your team can focus on solving actual business problems. Once your team has mastered the basic workflows described in this article, you might want also want to check out Kafka Connect's ability to data on the fly, which can be used to solve a whole new set of problems. transform

Tips About Kafka Connect On Heroku You Can't Afford To Miss

If you've spent days (or even weeks?) trying to spin up a Kubernetes cluster for learning purposes or to test your application, then your worries are over. Spawned from a Kubernetes Special Interest Group, KIND is a tool that provisions a Kubernetes cluster running IN Docker. From the docs: kind is a tool for running local Kubernetes clusters using Docker container "nodes". kind is primarily designed for testing Kubernetes 1.11+, initially targeting the . conformance tests Installing KIND As it is built using go, you will need to make sure you have the latest version of installed on your machine. golang According to the k8s , is preferred. To install kind, run these commands (it takes a while): docs golang -v 1.11.5 go get -u sigs.k8s.io/kind
kind create cluster Then confirm kind cluster is available: kind get clusters Setting up kubectl Also, install the latest using or . The latest Docker has Kubernetes feature but it may come with older . Check its version by running this command: kubernetes-cli Homebrew Chocolatey kubectl kubectl version Make sure it shows or above. If you find you are running from Docker, try or reorder path environment variable. GitVersion: "v1.14.1" kubectl brew link Once and kind are ready, open bash console and run these commands: kubectl KUBECONFIG=”$(kind get kubeconfig-path)”
kubectl cluster-info export If is properly set up, some information will be shown. Now you are ready to proceed. Yay! kind Deploying first application What should we deploy on the cluster? We are going to attempt deploying Cassandra since the docs have a pretty decent walk-through on it. First of all, download and for later. Then create kustomization.yaml by running two cat commands. cassandra-service.yaml cassandra-statefulset.yaml Once those files are prepared, layout them as following: yaml k8s-wp/
  kustomization.yaml
  mysql-deployment.yaml
  wordpress-deployment.yaml Then apply them to your cluster: k8s-wp
kubectl apply -k ./ cd Validating (optional) Get the Cassandra Service. kubectl get svc cassandra The response is: NAME        TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
cassandra   ClusterIP   None         <none>        9042/TCP   45s Note that Service creation might have failed if anything else is returned. Read for common issues. Debug Services Finishing up That's really all you need to know to get started with KIND, I hope this makes your life a little easier and lets you play with Kubernetes a little bit more :) About the author - Sudip is a Solution Architect with more than 15 years of working experience, and is the founder of Javelynn . He likes sharing his knowledge by regularly writing for Hackernoon , DZone , Appfleet and many more. And while he is not doing that, he must be fishing or playing chess. Previously posted at https://appfleet.com/ .

How To Manage Local Kubernetes Testing with KIND Tool

Customise your EC2 to your deployment needs Picture the scene. You’ve got a few small, simple microservices. They all run beautifully on 200mb of ram and scale instantly. Engineering bliss. But wait… in this blue sky, a scar in the scenery, lies an omen. A storm cloud in the shape of a monolith. You need to get this beast running in your Kubernetes cluster but it’s dramatically larger than your other applications. In the words of the internet, “wut do?” Is your Microservice EC2 Instance Type enough? Your Microservices can be packed into small boxes, but your monolith needs some juice. There simply isn’t enough horsepower in a single instance to run your application as a pod. The purists are going to tell you “Well then, refactor it into microservices”. Reality isn’t often so black and white. So do you deploy everything onto big boxes? Your first port of call might be to reassess everything — drain the existing nodes and deploy onto bigger boxes. That way you don’t need to run multiple node pools. This will work! But it comes at a cost. There is a greater chance of wasted resources in your cluster. Bigger boxes mean you’re more likely to have very underutilized instances, costing you money. Okay, so can I have multiple node types? Yes you can, you lucky devil. It’s straightforward and requires only a few steps. For the sake of this tutorial, I’m going to assume you’re familiar with Kubernetes concepts like or . If you’re not, you should read through the . Additionally, I’ll assume you’re comfortable with AWS terminology like Autoscaling Group (ASG) and EC2. If you’re not, . For the sake of brevity and clarity, I won’t be including changes to IAM roles and security groups. The includes these details. pod node Kubernetes documentation have a read through the AWS documentation first official documentation Okay! The Cluster Autoscaler The first thing you’ll need to hook up is the . This has an AWS specific installation mode. It will detect when there is no free space in your EC2 instances to deploy pods, at which point it will increase the number of EC2 instances in your ASG. It works by adjusting the “desired” field in the ASG, up to a predefined maximum that you’ve set. I won’t go over the installation in detail here because . When you’re following these instructions, be sure to set it up for “Autodiscovery”. That way, when you add new ASGs, it will detect them and you won’t need to reconfigure your cluster. cluster autoscaler the documentation is pretty good Next, your Autoscaling Group Spinning up an ASG is straight forward. For the sake of example, we’ll be using Terraform. In the following code snippets, we will create an auto scaling group and a run configuration. The run configuration is the template that your ASG should use when it’s spinning up new EC2 instances. There are some key things to note here. We’re ignoring changes to . This is because the Cluster autoscaler will be using this value to scale the cluster up and down. We don’t want to reset it back to 2 every time we do a . desired_capacity terraform apply There are two highlighted tag keys. is to ensure that the cluster autoscaler can detect your Auto scaling groups. The second, isn’t mandatory but is a good idea if you’re running multiple clusters. This will prevent two clusters picking up the same ASG. k8s.io/cluster-autoscaler/enabled mandatory k8s.io/cluster-autoscaler/cluster_name resource "aws_autoscaling_group" "worker_node_asg" {desired_capacity = 2max_size         = 10min_size         = 1name_prefix      = "worker-node-"lifecycle {ignore_changes = ["desired_capacity"]}tag {key = "Name"value = "worker-node"propagate_at_launch = true}tag {key = "k8s.io/cluster-autoscaler/enabled"value = "true"propagate_at_launch = true}tag {key = "k8s.io/cluster-autoscaler/cluster_name"value = "true"propagate_at_launch = true}} And the launch configuration (template) for your ASG. Note the field highlighted in bold. instance_type resource "aws_launch_configuration" "worker_launch_configuration" {image_id             = "${data.aws_ami.worker_ami.id}"instance_type        = "t2.medium"name_prefix          = "worker-node-"security_groups      = ["${some_security_group_ID}"]create_before_destroy = true} Create this infrastructure for each of the EC2 instance types you would like to create. We’ll assume, from here on out, you’ve done it twice for two different instance types. Let’s make the Pods more picky Your pods are pretty laid back at the moment — they sigh and say “oh just put me anywhere” as they’re flung at the k8s api. We need to give them a bit of choice and we can do that with . This will allow the pod to specify the type of node it would like to run on. node selectors Find your label If you run you’ll see some EC2 instances. Look for the label, that’ll let you know what the EC2 instance type is. You can use this in your resource yaml to pick which box you want your applications to deploy to. There are other ways to go about this that don’t pollute your yaml with AWS specific stuff —as always, be mindful. A simple example yaml might look something like this: kubectl get node -o yaml beta.kubernetes.io/instance-type apiVersion: v1kind: Podmetadata:  name: nginx  labels:    env: testspec:  containers:  - name: nginx    image: nginx  nodeSelector:    beta.kubernetes.io/instance-type: t2.medium Using the flag has given this pod a preference — it’s picky. Once you run and push it to the k8s API, kubernetes will find a box with the correct instance type and deploy it. If it doesn’t exist, it will trigger the autoscaling group to create an instance for you. nodeSelector kubectl apply -f myfile.yaml And there you have it! You’ve got a cluster that supports multiple EC2 instance types. You can add as many ASGs as you like. Go and take over the world you little scamp! If you enjoyed this tutorial, I’m regularly writing and throwing other articles . about on twitter

Configuring Multiple Node Pools

Community detection is a topic that can be applied to datasets that have an inherent order among themselves. For example, redwood, birch and oak are types of trees. Fish, algae and octopus are types of underwater creatures. These entities can be grouped under a common category, and the task of doing that is called as community detection. Sometimes we have a dataset that is unstructured and we want to get some value out of it, in those cases community detection can be useful. In this article I will be talking about how to detect communities using LPA, label propagation algorithm in GraphFrames package of Spark. The code for the article can be found . here Label Propagation Algorithm Label Propagation Algorithm is a fast algorithm that detects communities in a graph. It does not require prior information about the communities, and it uses network structure to detect them. We label some initial communities and it takes it from there. LPA was proposed by Raghavan in , and it works by labelling the nodes, and propagating these labels throughout the network and forming communities based on the process of label propagation. this paper The idea is that a label will become dominant in a densely connected group of nodes, but will have trouble propagating in a sparsely connected region. When the algorithm finishes, the nodes with the same label are considered as part of the community. The algorithm works as follows: a) Every node is initialized with a unique label. b) Label are propagated through the network c) At every iteration of the propagation, each node updates its label to the one that the maximum number of its neighbours belongs to. d) LPA reaches convergence when each node has the majority label of its neighbours. e) LPA stops if either convergence or the user defined maximum number of iterations are achieved. Implementation Dataset and Processing I am using the dataset provided by , it is a dataset that contains a word, and a reference to the wikipedia url about that word. There are 10 files, 600 MB each, and i am working with one file. It contains 18 million rows. UMASS It is in the following format: URL: ftp://217.219.170.14/Computer%20Group/Faani/vaset%20fani/second/sattari/word/2007/source/s%20crt.docx MENTION vacuum tube 421 http://en.wikipedia.org/wiki/Vacuum_tube MENTION vacuum tubes 10838 http://en.wikipedia.org/wiki/Vacuum_tube MENTION electron gun 598 http://en.wikipedia.org/wiki/Electron_gun MENTION fluorescent 790 http://en.wikipedia.org/wiki/Fluorescent MENTION oscilloscope 1307 http://en.wikipedia.org/wiki/Oscilloscope MENTION computer monitor 1503 http://en.wikipedia.org/wiki/Computer_monitor MENTION computer monitors 3066 http://en.wikipedia.org/wiki/Computer_monitor MENTION radar 1657 http://en.wikipedia.org/wiki/Radar MENTION plasma screens 2162 http://en.wikipedia.org/wiki/Plasma_screen URL is the link to a particular topic, and mention contains the word that was mentioned in the article along with the Wikipedia link (also mentioned in the article). I used the following code to preprocess the data to arrive at a meaningful format: After preprocessing, i have the data in the following format: ftp://ftp.pcbsd.org/pub/handbook 9.0/handbook_en_ver9.0.html,Zfs ftp://ftp.pcbsd.org/pub/handbook/9.0/handbook_en_ver9.0.html,BSD_UNIX ftp://ftp.pcbsd.org/pub/handbook/9.0/handbook_en_ver9.0.html,Acpi ftp://ftp.pcbsd.org/pub/handbook/9.0/handbook_en_ver9.0.html,Softupdates ftp://ftp.pcbsd.org/pub/handbook/9.0/handbook_en_ver9.0.html,Compiz ftp://ftp.pcbsd.org/pub/handbook/9.0/handbook_en_ver9.0.html,Pcx ftp://ftp.pcbsd.org/pub/handbook/9.0/handbook_en_ver9.0.html,Journaling_file_system ftp://ftp.pcbsd.org/pub/handbook/9.0/handbook_en_ver9.0.html,Nextstep ftp://ftp.pcbsd.org/pub/handbook/9.0/handbook_en_ver9.0.html,Freebsd_jail ftp://ftp.pcbsd.org/pub/handbook/9.0/handbook_en_ver9.0.html,Mac_address ftp://ftp.pcbsd.org/pub/handbook/9.0/handbook_en_ver9.0.html,IPv6_address Basically, in this format, I have a URL followed by each word that was mentioned in the original file. Applying LPA Now that we have the data in the required format, we can apply LPA. I use the following code to do that. The largest community is has 8500 nodes. Let us look at it’s contents: ... ... ... ... ... ... ... ... http://commons.wikimedia.org/wiki/File:Vestv%C... ... http://commons.wikimedia.org/wiki/File:Aston_M http://www.free-photos.biz/photographs/industr http://commons.wikimedia.org/wiki/File:Mercede http://commons.wikimedia.org/wiki/File:Odiorne http://commons.wikimedia.org/wiki/File:Hannove http://commons.wikimedia.org/wiki/File:Autosta http://www.free-photos.biz/photographs/food/ve http://commons.wikimedia.org/wiki/File:Smeaton http://commons.wikimedia.org/wiki/File:Mitsubi As we can see, the largest community is about free photos. Let us look at another: Philip_Neame Monsoon#Asia Arab_Jew Cave_of_Elijah Shi’a_Islam Culture_of_the_Song_Dynasty Safavid_dynasty John_Romer_(Egyptologist) Bamyan_Province Henry_Phillips_(horticulturist) Christ_Church,_Nazareth The second largest community is about dynasties or provinces, and people related to them. It has about 6000 nodes. There are some false positives as well, which means more iterations in LPA can probably give good results. Third largest community is: Hansi Vadodara_district Cox-Forbes_theory Puppis_(constellation) University_of_Alexandria historical_scholarship Villuercas Center_of_the_universe genre Velayuthampalayam and it is mostly about science. It has 4000 nodes, and by the looks of it, it can probably use more number of iterations as well. Conclusion In this post we went through what communities are, and how do we detect them. We used the algorithm Label Propagation algorithm to determine that. It works by initially labelling nodes, then iterating and propagating labels and determining the final label at convergence. We then used a dataset that contains a URL and references to wikipedia text for several mentions of words in that url, and determined communities from them. Most popular community is about stock photos, then about dynasties and then science. I hope you liked my post.

Determining Communities In Graphs Using Label Propagation Algorithm 

Music & ML: Clustering on Spotify Audio Features

ERROR: invalid target_lang (ay)

Muzika i ML: grupisanje na Spotify audio funkcijama

Música i ML: agrupació a les funcions d'àudio de Spotify

Μουσική & ML: Συγκεντρώνοντας στο Spotify Audio Features

Música y ML: Clustering en características de audio de Spotify

Musique & ML: Clustering sur les fonctionnalités audio de Spotify

ERROR: invalid target_lang (iw)

音楽と機械学習: Spotify オーディオ機能のクラスタリング

음악 및 ML: Spotify 오디오 기능 클러스터링

Musique & ML: Clustering ya Spotify Audio

Musika at ML: Pag-cluster sa Spotify Audio Features

ERROR: invalid target_lang (ts)

Table of Links Abstract and 1. Introduction Abstract and 1. Introduction 2 Methodology 2 Methodology 3 Hardware 3 Hardware 4 Software 4 Software 5 Network 5 Network 6 Consensus 6 Consensus 7 Cryptocurrency Economics 7 Cryptocurrency Economics 8 Client API 8 Client API 9 Governance 9 Governance 10 Geography 10 Geography 11 Case Studies 11 Case Studies 12 Discussion and References 12 Discussion and References A. Decentralization and Policymaking A. Decentralization and Policymaking B. Software Testing B. Software Testing C. Brief Evaluations per Layer C. Brief Evaluations per Layer D. Measuring decentralization D. Measuring decentralization E. Fault Tolerance and Decentralization E. Fault Tolerance and Decentralization 5 Network Blockchain nodes communicate over a peer-to-peer (P2P) network, the decentralization properties of which have been of great interest to researchers [134,4,82,76]. Systems often implement a message diffusion mechanism [78] via a gossip protocol that avoids full graph connectivity [54]. Users typically use the Internet to access the P2P overlay, though some efforts try to introduce an independent infrastructure [47]. Here we explore networking aspects which present single points of failure, in terms of topology and bootstrapping. A notable research question that arises organically from our analysis, and touches upon both following subsections, is creating a P2P network that is both permissionless and Byzantine resilient, e.g., as explored in [49,124]. Topology. The first networking aspect of interest is the network’s topology. Evaluating a real-world network’s clustering properties is a well-known problem, traditionally done by generating random graphs and comparing the expected with the observed values [63,135,1]. In blockchain systems, every node maintains a list of peer connections. Crucially, message provenance is not typically provided, so no party can know the network origin of an incoming message. Therefore, the resource of interest are “bridges” (single nodes or small cluster of nodes) between the network graph’s components and the relevant parties are the bridges’ owners or operators. Here, a component is a single node or a cluster of tightly interconnected nodes. Topology Topology A distributed network, in the tradition of Baran [15], is key in maintaining safety and liveness. Under the CAP theorem [27], any networked system can satisfy at most two of the following properties: i) a consistent data copy; ii) data availability; iii) network partition tolerance. Ledger systems are no exception. Each node needs to maintain at least one connection to an honest party to receive all messages and avoid eclipse attacks [87]. If some parties cannot communicate with the rest of the network, either they halt and any transactions they submit are dropped (violating liveness) or they produce separate ledger versions (violating safety).[9] By preventing communication between a node and the rest of the network, an attacker reduces the honest computational power and isolates that node, making a 51% or a double spending attack (against the isolated node) easier to deploy. Additionally, an adversary can violate liveness by blocking the node’s transactions from reaching the rest of the network. Third, an adversary that controls all of a node’s connections can link transactions to the specific node, correlate the user’s addresses, and associate them with real-world information, such as an IP address, thus compromising privacy[10]. This also holds on a macroscopic level, i.e., a node that acts as a central communication hub between two clusters can obtain information and even deanonymize some participants (violating privacy). Node Bootstrapping and Peer Discovery. Joining a ledger’s network and synchronizing with it is the so-called “bootstrapping” process. All real-world ledgers rely on an initial (trusted) setup, the first (“genesis”) block.[11] Obtaining the correct genesis block is done in an out-of-band, typically secure mechanism, since it is often well-known and easy to validate from various sources. Node Bootstrapping and Peer Discovery. Node Bootstrapping and Peer Discovery. Initially, the node needs to connect to some peers and receive all available chains. Therefore, the resource here is bootstrapping nodes and the relevant parties are these nodes’ operators. Then, using the ledger protocol’s chain resolution mechanism, it decides which chain to adopt. There are two points of interest here. First, it should be guaranteed that the node connects to at least one honest peer, to retrieve all available information and avoid getting eclipsed. As explained above, if nodes gets eclipsed, the system’s liveness, safety and privacy properties may get compromised. However, connecting to honest peers is not straightforward, given that the node has no knowledge about the network participants, a standard problem in P2P networks [57]. Blockchain systems predominantly use either hardcoded peer lists or DNS seeding [120], although both techniques are censorship-prone. Notably, [120] showed that more secure alternatives, such as ZMap [61], cannot be feasibly used in existing blockchain systems. Second, even if the node connects to some honest peers, catching up by using only genesis (“bootstrapping from genesis”) is not always feasible. Especially in PoS systems, an attacker can effortlessly assemble an arbitrarily-long, seemingly correct chain (violating safety) [81,31,118]. To counter such attacks, some ledgers employ checkpoints [107,52,50], which are often issued centrally and are either hardcoded or received from the peers. Other solutions do exist, e.g., analyzing block density and relying on key erasures [12] or using VDFs [24,53], but enforcing and/or relaxing such assumptions still poses an interesting research problem. Authors:
(1) Christina Ovezik, University of Edinburgh (c.ovezik@ed.ac.uk);
(2) Dimitris Karakostas, University of Edinburgh (dkarakos@ed.ac.uk);
(3) Aggelos Kiayias, University of Edinburgh and IOG (akiayias@ed.ac.uk). Authors: Authors: (1) Christina Ovezik, University of Edinburgh (c.ovezik@ed.ac.uk); (2) Dimitris Karakostas, University of Edinburgh (dkarakos@ed.ac.uk); (3) Aggelos Kiayias, University of Edinburgh and IOG (akiayias@ed.ac.uk). This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv available on arxiv [9] In longest-chain protocols, like Bitcoin, miners keep producing blocks in isolation, ending up with different ledger versions. In BFT-style protocols, like Algorand, the ledger cannot be updated if a large number of block producers become unreachable and thus do not adopt new transactions. [10] Note that protecting privacy in the network layer can be a particularly challenging problem, even in the setting where no single adversary controls all of a node’s connections. [11] Some proposals rely on computational assumptions instead of a trusted setup [80], but their real-world performance and applicability is still untested.

Eclipse Attacks, Network Clusters & DNS Seeding

MongoDB is a document DB that comes with great flexibility and scaling. It is built using a scale out architecture where more nodes can be added to meet increasing demand. Having said that, setting it on multiple regions comes with a challenge to keep it available and consistent when regions go down. As MongoDB requires majority votes to pick Primary, a bad setup can leave it in read only mode. When I first worked on a project requiring a MongoDB deployment across multiple active-active regions, I thought it would be straightforward. But when I started going over the available options, I realized that knowledge of MongoDB’s election process and quorum requirements is important before choosing the best approach. In this article, I’m sharing my discoveries and the methods I investigated. The Quorum Problem in Multi-Region MongoDB Deployments MongoDB’s replica sets keep things running smoothly by using an election process that requires more than half of the voting members to pick a primary. If a primary cannot collect a majority of votes, it steps down, and no new primary can be elected, leaving the set read-only (all members become secondaries). This mechanism prevents split-brain scenarios with multiple primaries. For example, in a 4-member set split 2–2 by a network partition, neither side reaches three votes (the majority of 4), resulting in no primary until the partition heals. Even-numbered clusters are not ideal because a perfect split can block write availability. The election protocol will re-run elections if a tie occurs, but repeated ties delay failover. The best practice is to use an odd number of voting nodes to eliminate this risk. Consider a Deployment with: 3 nodes in East (voting members)


3 nodes in West (voting members)


1 additional node in North (tie-breaker node) 3 nodes in East (voting members) 3 nodes in East (voting members) 3 nodes in West (voting members) 3 nodes in West (voting members) 1 additional node in North (tie-breaker node) 1 additional node in North (tie-breaker node) If an entire region goes down, the remaining 3 nodes alone do not have a majority (they need 4 votes out of 7). A 7th node in a neutral region (North) act as a tie-breaker to maintain quorum. This node can either be an arbiter (vote-only) or a data-bearing node. Let’s examine the trade-offs. Approach 1: Setup with an Arbiter When setting up a MongoDB replica set, it is important to know how write concerns work. Especially when using an arbiter. In a three node configuration, there is one primary and one secondary, along with an arbiter. In this setup, the majority is equal to the number of data-bearing voting members. If you set the write concern to majority, it can lead to issues. If a data-bearing member goes down, writes might time out or never get acknowledged. To avoid this, it's generally better to use another data-bearing voting member instead of an arbiter. Arbiters don’t store any data, so they don’t have an oplog. This means they can’t track the latest operations and compare them like other nodes. During elections, this can sometimes cause arbiter voting for a secondary with lower priority, which might not be ideal. In larger setups, like three nodes in one, three in another, and a secondary in a third location. This kind of setup helps with redundancy and failover but also introduces considerations like latency and data consistency. 3 nodes in West (voting)
3 nodes in East (voting)
1 arbiter in North (voting, no data) 3 nodes in West (voting) 3 nodes in East (voting) 1 arbiter in North (voting, no data) Why It Works If West fails, the 3 East nodes and arbiter can elect a new primary.
If East fails, the 3 West nodes and arbiter can elect a new primary.
Requires no extra storage since the arbiter doesn’t hold data. If West fails, the 3 East nodes and arbiter can elect a new primary. If East fails, the 3 West nodes and arbiter can elect a new primary. Requires no extra storage since the arbiter doesn’t hold data. Downsides Arbiters don’t acknowledge writes, meaning that if a data-bearing secondary is also down, writes requiring majority can fail.


MongoDB discourages arbiters in production, as they add risk without redundancy. Arbiters don’t acknowledge writes, meaning that if a data-bearing secondary is also down, writes requiring majority can fail. Arbiters don’t acknowledge writes, meaning that if a data-bearing secondary is also down, writes requiring majority can fail. MongoDB discourages arbiters in production, as they add risk without redundancy. MongoDB discourages arbiters in production, as they add risk without redundancy. Best for: When cost constraints prevent adding a full data-bearing node, but a third vote is needed for quorum. Best for: Approach 2: 3-3-1 Setup with a Data-Bearing Tie-Breaker Node 3 nodes in West (voting)
3 nodes in East (voting)
1 data-bearing secondary in North 3 nodes in West (voting) 3 nodes in East (voting) 1 data-bearing secondary in North Why It Works If an entire region fails, the tie-breaker in third region can help elect majority.
The extra node stores data, reducing the risk of data loss.
Unlike an arbiter, this node can be queried.
MongoDB recommends this setup over using an arbiter. If an entire region fails, the tie-breaker in third region can help elect majority. The extra node stores data, reducing the risk of data loss. Unlike an arbiter, this node can be queried. MongoDB recommends this setup over using an arbiter. Downsides Higher storage and replication cost than an arbiter. Higher storage and replication cost than an arbiter. Higher storage and replication cost than an arbiter. Best for: High availability where an extra backup and read scaling are required. Best for: While arbiters can help with elections, they don’t contribute to data durability. Having odd number of data-bearing voting members is safer for maintaining a healthy and reliable replica set. Alternative Approaches Hidden Node Instead of a Tie-Breaker A hidden node is a data-bearing secondary that doesn’t vote or get queried but provides a full backup. It acts as a recovery option and can be manually promoted if needed but it does not participate in automatic elections. Sharded Cluster with Region-Aware Primaries For true active-active writes, each region can be a primary for different subsets of data using sharded clusters with zone sharding. This eliminates cross-region replication lag. This approach requires application-level changes and increased complexity. Clarifying Active-Active in MongoDB MongoDB does not support multi-primary replication in a replica set. Only one primary exists at a time, and secondaries replicate from it. If a region wants its own primary, it must be configured via sharded clusters. What to Remember: MongoDB replica sets require a majority for elections.
Only one primary exists at a time.
Arbiters should only be used as a last resort.
A two-region deployment must include a third vote in a separate location.
For true active-active, use sharding. MongoDB replica sets require a majority for elections. Only one primary exists at a time. Arbiters should only be used as a last resort. A two-region deployment must include a third vote in a separate location. For true active-active, use sharding. Final Thoughts If you can afford it, use a data-bearing node instead of an arbiter to maintain high availability and prevent write inconsistencies. Understanding above options helps in designing a resilient multi-region MongoDB deployment. References: https://www.mongodb.com/docs/manual/replication/ https://www.mongodb.com/docs/manual/replication/ https://www.mongodb.com/docs/manual/replication/ https://www.mongodb.com/docs/manual/sharding/ https://www.mongodb.com/docs/manual/sharding/ https://www.mongodb.com/docs/manual/sharding/ https://www.mongodb.com/docs/manual/core/replica-set-elections/ https://www.mongodb.com/docs/manual/core/replica-set-elections/ https://www.mongodb.com/docs/manual/core/replica-set-elections/ https://www.mongodb.com/docs/manual/core/replica-set-secondary/ https://www.mongodb.com/docs/manual/core/replica-set-secondary/ https://www.mongodb.com/docs/manual/core/replica-set-secondary/ https://www.mongodb.com/docs/manual/reference/write-concern/#calculating-majority-for-write-concern https://www.mongodb.com/docs/manual/reference/write-concern/#calculating-majority-for-write-concern https://www.mongodb.com/docs/manual/reference/write-concern/#calculating-majority-for-write-concern

How to Stop Your MongoDB Cluster from Going Read-Only at the Worst Possible Time

The ideal size of your Kubernetes clusters is a day 0 question and demands a definite answer. You find one giant cluster on one end of the spectrum and many small-sized ones on the other, with every combination in between. This decision will impact your organization for years to come. Worse, if you decide to change your topology, you're in for a time-wasting and expensive ride. I want to list each approach's pros and cons in this post. Then, I'll settle the discussion once and for all and argue why selecting the giant cluster option is better. The One Giant Cluster Approach Deciding on a single giant cluster has great benefits. Better Resource Utilization Kubernetes was designed to handle large-scale deployments, initially focusing on managing thousands of nodes to support extensive and complex containerized applications. This scalability was a key feature from its inception, enabling it to orchestrate resources across vast, distributed systems efficiently. Thus, a Kubernetes cluster is a scheduler at its core: it knows how to run workloads on nodes according to constraints. Without constraints, it will happily balance workloads across its available nodes. If you split the cluster into multiple clusters, you lose this benefit. You can have situations where one cluster is idle while another cluster is close to resource starvation and must cancel workloads and kill pods. Lower Operational Overhead Good Kubernetes practices mandate that you back up your etcd data, monitor your cluster metrics, log your cluster events, provide security-related tools, etc. Size aside, it stands to reason that it's more time-effective to operate fewer clusters. For example, regarding metrics, you'd set up a single Prometheus instance, potentially clustered to handle additional traffic, and be done with it. Automation can mitigate the repetitive aspect of installing and maintaining an instance for each cluster, but you'll still end up with as many instances as you have clusters (or more). Prometheus is just one example because many cluster admins have a long list of tools they run in every cluster. Straightforward Networking and Service Communication Service-to-service communication inside a single cluster is straightforward. Point to <service-name>.<namespace>.svc.cluster.local and be done with it. Even better, you only need the service name part inside the same namespace. <service-name>.<namespace>.svc.cluster.local You'll need a tool to help you with inter-cluster communication, from the simplicity of External DNS with LoadBalancer to the complexity of a full-fledged solution like Istio—both ends of the spectrum mandate time and operation costs. Simplified Governance Since every object is part of the same cluster inside a single cluster, one can enforce a centralized set of policies with a standardized approach. For example, you can create a namespace per team and environment, restricting access to only that team's members. Once you start having multiple clusters, even if you take the same approach, you'll duplicate the policy rules across clusters, with potential differences that will drift further with time. Cost Efficiency A single cluster means a single control plane, simplifying management and reducing overhead. While a control plane is essential for orchestration, its value comes from enabling efficient workload execution rather than directly running business applications. Additionally, many of the points above tie into cost optimization. With a single cluster, you only need to configure monitoring (e.g., Prometheus), logging, and security tools once, reducing duplication. In-place automation streamlines operations, helping manage costs without adding unnecessary infrastructure expenses. e.g. Downsides of a One Giant Cluster Approach Unfortunately, the giant cluster option is not only unicorns and rainbows; there are definite downsides. Larger Blast Radius The larger the cluster, the more teams will use it. Unfortunately, this means that if something bad happens to the cluster, it will wreak havoc on the work of more teams. It occurs regardless of whether the outage results from a malicious actor, a wrong configuration, or resource starvation. When an actual malicious actor does indeed breach access, a larger cluster exposes more workloads, and the actor can compromise more of them. Even without malicious actors, every maintenance operation and upgrade on a cluster can affect its users; the bigger the cluster, the larger the potential impact. When planning an upgrade for a single cluster, you need to conduct an impact analysis that encompasses all users and teams in the organization. Complex Multi-Tenancy Management Even within a single organization, multiple teams will use the Kubernetes cluster. Even if every team member behaves professionally, we must set strict policies to avoid issues. If the cluster resembles a building, you'd still put locks on your apartment even if you have friendly neighbors. Likewise, the cluster administrator must enforce strict rules to make sharing a cluster acceptable. At the very least, we need strict namespace isolation to avoid unnecessary access and resource quotas and enforce fairness across teams. The problems caused by using a single cluster by teams across a single organization multiply a hundredfold if the cluster is multi-tenant and shared across several organizations. Scalability Limits Regardless of how exceptional Kubernetes is, it's still a physically based system with physical limits. For example, some objects have a clear limit, but even if you never reach them, getting close to them will require excellent system administration skills with some fine-tuning. fine-tuning Even then, the more load you put on a Kubernetes API server, the more sluggish your system will be. If you're lucky, it will degrade linearly, but chances are it will hit some system limit and degrade all at once. Cluster-Wide Objects Kubernetes objects are namespace-scoped, but a couple of them are cluster-scoped. A cluster-scoped object can only have a single instance across the whole cluster. For example, a Custom Resource Definition is cluster-scoped. It means that if a team wants to use v1 of a CRD, then every team on the same cluster is stuck with v1 if they wish to use this CRD. Worse, if any team wants to upgrade to v2, they must coordinate across all teams using the CRD to synchronize the upgrade. What's the Ideal Size, Then? I could describe the pros and cons of very granular clusters, but they mirror the opposite of what we have just seen. For example, very granular clusters allow each team to work on their version of a CRD without stepping on another team's toes. For this reason, I'll avoid repeating myself. Most, if not all, articles evaluating the pros and cons of each end of the spectrum advise a meet-in-the-middle approach: "a couple" of clusters to mitigate the worst aspects of each extreme approach. It's all well and good, but none of them, at least none I've read, tell precisely how many "a couple" is. Is it a cluster per environment, i.e., production, staging, or development? Is it a cluster per team? i.e. ‎I'll take a risk and advertise for two clusters: one for production and the other for everything else. How would you manage the cons mentioned above? Read on. vCluster vCluster is an Open Source product that allows the creation of so-called virtual clusters. vCluster is part of the CNCF landscape, specifically a certified Kubernetes distribution. Being a certified distro means a virtual cluster offers every Kubernetes API you can expect, and you can deploy any application to it just like any other Kubernetes cluster. vCluster certified Kubernetes distribution vCluster operates by creating the virtual cluster in a dedicated namespace. You can specify the latter, or vCluster will infer it from the virtual cluster's name. By default, it creates a control plane using the vanilla k8s distribution, but you can choose another one, such as k3s. Likewise, by default, it stores its configuration in an SQLite database, which works particularly well for temporary and pre-production clusters, such as those you create for a pull request. Alternatively, you can rely on a regular etcd or even external databases such as MySQL or Postgres as a data store for more permanent usage and better resilience and scalability of the virtual cluster. Once you've created a virtual cluster via the CLI or the Helm chart, you can connect to it. The client-side CLI creates a dedicated reusable kubeconfig context. From within a virtual cluster, users see no other virtual clusters. If you need to access the host cluster resources from the virtual cluster or vice versa, vCluster uses a so-called syncer that syncs objects back and forth according to a configuration file. This way, you can set up an Ingress Controller on the host cluster and define your Ingress objects in the virtual cluster(s). How vCluster Mitigates the Downsides of a Giant Cluster Let's review each downside of a giant cluster and how vCluster handles it. Larger blast radius: When using a virtual cluster, the blast radius is automatically contained inside its boundaries. If you want to be conservative, aim for a small granularity approach, such as a cluster per team and environment. Larger blast radius: When using a virtual cluster, the blast radius is automatically contained inside its boundaries. If you want to be conservative, aim for a small granularity approach, such as a cluster per team and environment. Complex multi-tenancy management: Gone are multi-tenancy problems since your tenants don't see each other and are isolated inside their respective virtual clusters. Complex multi-tenancy management: Gone are multi-tenancy problems since your tenants don't see each other and are isolated inside their respective virtual clusters. Scalability limits: While the limits are still there, the chances of reaching them decrease with the number of virtual clusters. If your giant cluster had 100k services, they are now spread throughout all virtual clusters. Scalability limits: While the limits are still there, the chances of reaching them decrease with the number of virtual clusters. If your giant cluster had 100k services, they are now spread throughout all virtual clusters. Upgrades and maintenance risks: Upgrade and maintenance tasks are limited to the scope of a single virtual cluster. You can do them in turn, and they will only affect the virtual clusters you target. Upgrades and maintenance risks: Upgrade and maintenance tasks are limited to the scope of a single virtual cluster. You can do them in turn, and they will only affect the virtual clusters you target. Cluster-wide objects: Finally, with virtual clusters, every team can install their version of a CRD, and its virtual cluster binds the CRD. It allows each team to be entirely independent of each other regarding the version of a CRD they use. Cluster-wide objects: Finally, with virtual clusters, every team can install their version of a CRD, and its virtual cluster binds the CRD. It allows each team to be entirely independent of each other regarding the version of a CRD they use. But I Need Different Clusters! While a single giant cluster provides compelling advantages, there are contexts in which a multi-cluster approach is justified. The most common reason is geographic distribution—specific applications require clusters in multiple regions to meet compliance requirements, reduce latency, or provide disaster recovery. For example, companies operating under GDPR or financial regulations may need strict data residency enforcement, which requires region-specific clusters. Similarly, organizations with stringent security postures may enforce complete isolation between environments or business units, making separate clusters a hard requirement. However, even in these cases, vCluster remains relevant. It allows for minimizing the number of physical clusters while still enabling workload separation at a virtual level. Instead of creating a sprawling landscape of Kubernetes clusters, teams can deploy regional virtual clusters within a single host cluster, balancing isolation and operation complexity. Conclusion Kubernetes cluster topology decisions are critical and long-lasting. While many advocate for a middle-ground approach between a single cluster and many small ones, they rarely specify the exact setup. Instead of guessing how many clusters to create, consolidating everything into a single, well-managed giant cluster makes more sense. The benefits—better resource utilization, lower operational overhead, simplified networking, centralized governance, and cost efficiency—outweigh the downsides. That said, the traditional downsides of a giant cluster, such as a larger blast radius, multi-tenancy complexities, scalability limits, upgrade challenges, and cluster-wide object constraints, are valid concerns. This is where vCluster changes the game. By using virtual clusters, you retain all the advantages of a single giant cluster while mitigating its worst drawbacks. vCluster isolates workloads, reduces operational risk, scales dynamically, simplifies upgrades, and removes conflicts over cluster-wide objects. Enhanced with vCluster, one cluster for production and one giant cluster for everything else, is the best approach for long-term scalability, efficiency, and ease of operations.

The One Giant Cluster Approach: One Giant Kubernetes Cluster for Everything

Azure Kubernetes Service (AKS) simplifies the deployment, management, and scaling of containerized applications using Kubernetes. To interact with an AKS cluster, you need to establish a connection using kubectl, the Kubernetes command-line tool. This guide provides a step-by-step process to explain How To Connect Azure Kubernetes Cluster Using Kubectl Azure Kubernetes Service (AKS) kubectl Why Kubectl needed? Why Kubectl needed? Connecting to an AKS cluster is an essential step for managing workloads, monitoring performance, and deploying applications. This process is especially critical for: Monitoring Cluster Health: Using kubectl commands to retrieve performance metrics and check node status.
Application Deployment: Deploying and managing containerized applications in the AKS cluster.
Cluster Administration: Performing administrative tasks like scaling, updating, or debugging resources within the cluster. Monitoring Cluster Health: Using kubectl commands to retrieve performance metrics and check node status. Monitoring Cluster Health: Application Deployment: Deploying and managing containerized applications in the AKS cluster. Application Deployment: Cluster Administration: Performing administrative tasks like scaling, updating, or debugging resources within the cluster. Cluster Administration: Whether you’re a developer or administrator, establishing this connection ensures you can effectively manage your Kubernetes environment. To connect to an Azure Kubernetes Service (AKS) cluster using kubectl, you will need to perform the following steps: Prerequisites (install both Azure CLI and Kubectl) Prerequisites (install both Azure CLI and Kubectl) STEP: 1 Install the Azure CLI STEP: 1 Install the Azure CLI If you haven’t installed the Azure CLI on your local machine, you can download and install it from the official Microsoft documentation here. official Microsoft documentation here STEP: 2 Install Kubectl STEP: 2 Install Kubectl Install the Kubernetes command-line (click here) tool “Kubectl” , if you haven’t installed it already. Install the Kubernetes command-line (click here) Steps to connect Azure account: Steps to connect Azure account: STEP: 3 Authenticate with Azure STEP: 3 Authenticate with Azure Open command prompt, run az login command to authenticate your CLI with your Azure account. Once you run this command, you will be prompted to enter your Azure account credentials. az login az login Steps to connect Azure AKS Cluster: Steps to connect Azure AKS Cluster: Go to Azure Portal -> Kubernetes Services -> Select the required Cluster -> Overview -> Connect -> to find the entire command for the specific cluster itself or follow the below commands one by one by replacing with subscription Id, cluster name and resource group name. STEP: 4 Set the subscription STEP: 4 Set the subscription To set subscription, run az account set --subscriptiond 95fe7-8d2c-4297-ad8b-a8eb08322955 az account set --subscriptiond 95fe7-8d2c-4297-ad8b-a8eb08322955 STEP: 5 Generate kubeconfig file STEP: 5 Generate kubeconfig file Open command prompt, run the below command and run az aks get-credentials command to connect to your AKS cluster. The get-credentials command downloads credentials and configures the Kubernetes CLI to use them. #Syntax: #Syntax: az aks get-credentials --resource-group <resource-group-name> --name <cluster-name> az aks get-credentials --resource-group <resource-group-name> --name <cluster-name> Replace <resource-group-name> with the name of the resource group that contains your AKS cluster, and <cluster-name> with the name of your AKS cluster. az aks get-credentials --resource-group rg-dgtl-pprd-we-01 --name aks-dgtl-pprd-we-01 az aks get-credentials --resource-group rg-dgtl-pprd-we-01 --name aks-dgtl-pprd-we-01 Above command will create the kubeconfig file in the user root directory. To get kubeconfig file in the specific location Connect Azure Kubernetes Cluster Using Kubectl STEP: 6 Verify the connection STEP: 6 Verify the connection To verify that Kubectl is connected to your AKS cluster, run the Kubectl get nodes command. This command should display a list of the nodes in your AKS cluster. If the command returns the list of nodes, then you have successfully connected to your AKS cluster using Kubectl. kubectl get nodes kubectl get nodes Points to Remember Points to Remember Go to Azure Portal -> Kubernetes Services -> Select the required Cluster -> Overview -> Connect -> to find the entire command for the specific cluster itself or follow the below commands one by one by replacing with subscription Id, cluster name and resource group name.
First we need to login to the Azure account by configuring Azure subscription and login details. And then we need to connect to Kubernetes Cluster only then we can able to run Kubectl commands. Go to Azure Portal -> Kubernetes Services -> Select the required Cluster -> Overview -> Connect -> to find the entire command for the specific cluster itself or follow the below commands one by one by replacing with subscription Id, cluster name and resource group name. Go to Azure Portal -> Kubernetes Services -> Select the required Cluster -> Overview -> Connect Go to Azure Portal -> Kubernetes Services -> Select the required Cluster -> Overview -> Connect First we need to login to the Azure account by configuring Azure subscription and login details. And then we need to connect to Kubernetes Cluster only then we can able to run Kubectl commands. Conclusion Conclusion Connecting to an AKS cluster using kubectl is a fundamental skill for managing Kubernetes workloads in Azure. By following this guide, you can authenticate, configure, and verify your connection to the AKS cluster seamlessly. This enables you to monitor cluster performance, deploy applications, and manage resources effectively. As Kubernetes continues to be a vital platform for container orchestration, mastering tools like kubectl and Azure CLI is essential for efficient cluster management.

Here's How To Connect an Azure Kubernetes Cluster Using Kubectl

Abstract and 1 Introduction 2 Related Work 3 Image Compression Module 4 System Architecture 5 Experimental Results 6 Conclusions A Compression Results References 2 Related Work Popular remote display protocols offer high-resolution sessions, multimedia stream remoting, multi-monitor support, dynamic object compression, USB redirection, drive mapping and more. Microsoft Remote Desktop Protocol (RDP), VMwares PC-over-IP (PCoIP) and Citrixs HDX are the most commonly used, but there are other protocols from companies such as Ericom and Hewlett-Packard. Teradici sells hardware solutions based on PCoIP technology [10] and the Thin Client Internet Computing protocol is also applicable for remote displays [11]. But these protocols still have drawbacks; great performance requires a lot of bandwidth and energy, which can clog the network [35]. Plus, if the system is running with low CPU usage, protocols will hog bandwidth and weaken end user performance. So, an efficient system needs to cope with unstable network conditions, to operate in real time, and to satisfy the users’ demands. Almost all of these protocols have the disadvantages that they do not consider the network environment and are inconvenient to use with real-time applications such as video playback. Previous work has included the development of an adaptive encoding application to solve display performance problems encountered when using RDP [12]. The authors of another work have described problems that arise in the case of video conferencing and have developed a real-time camera control system to address the problems [13]. To reduce the size of images transferred over the network to improve video streaming quality, compression technology has been one of the most important contributions. Some encoding and decoding formats such as motion MPEG [14], [16] and H.264 AVC [15], [17] have been developed in hybrid remote display protocol. The challenges for compressing images have been summarized previously [18]. Various compression techniques have been developed for image compression [37], [38]. In this work, we focus on the k-means clustering for encoding the image data. This method is probably the most popular technique of representativebased clustering. Cluster analysis [44] is a major data analysis method that is widely used for many practical applications in emerging areas. Clustering is the process of partitioning a given set of objects into disjoint clusters. This is done in such a way that objects in the same cluster are similar whereas objects belonging to different clusters differ considerably with respect to their attributes [48]. The k-means algorithm [45], [46] is effective in producing clusters for many practical applications. But the original k-means algorithm is very computationally complex, especially for large data sets. Moreover, this algorithm yields different clusters depending on the random choice of initial centroids. For example, previous work has included the description of a 2D image compression scheme using k-means in a real-time multiprocessor system [39]. Another work presented a novel k-means algorithm for compressing images, including exploration of the strategy of applying statistical parameters for choosing the initial seeds [40]. Despite the efficiency of the latter scheme, it still has drawbacks like a priori fixation of the number of clusters and random selection of initial seeds. Various researchers have made several attempts to improve the performance of k-means clustering [41], [42], [43]. In addition, hierarchical clustering algorithms also usually apply in image processing. But as the number of records increase the performance of hierarchical algorithm goes decreasing and time for execution increased. K-mean algorithm also increases its time of execution but as compared to hierarchical algorithm its performance is better. As a general conclusion, k-means algorithm is good for large dataset and hierarchical is good for small datasets [6]. Therefore k-means is most suitable for image processing in the video streaming scheme. Beside, developing a model that efficiently provides QoS control is presently a challenging problem. Many researches have addressed the problem of providing QoS guarantees to various network applications and clients. Most of them have focused on servers without considering network delays [19], [20], on individual network routers [29], [30], [31], or on clients with assumptions of QoS supports in networks [32]. Some recent work focused on end-to-end QoS guarantees in network cores [33], [34], with the aim of guaranteeing QoS as measured from server-side network edges to client-side network edges. To solve this problem, previous approaches need to be revised to allow their application to a cloud environment in a manner that responds to various user behaviors and network conditions. In the present work, k-means clustering is used to reducing the size of one frame from the video. We then develop a scheme for the analysis of historical data as a means to consider network status and make a decision regarding the optimal compression ratio for QoS control. Eventually, we evaluate our system model using actual measurement results resulting from the processing of images from a realistic image dataset. This paper is available on arxiv under CC BY 4.0 DEED license. (1) Huu-Quoc Nguyen, Department of Computer Engineering, Kyung Hee University, 1 Seocheon, Giheung, Yongin, Gyeonggi, South Korea and quoc@khu.ac.kr;
(2) Tien-Dung Nguyen, Department of Computer Engineering, Kyung Hee University, 1 Seocheon, Giheung, Yongin, Gyeonggi, South Korea and ntiendung@khu.ac.kr;
(3) Van-Nam Pham, Department of Computer Engineering, Kyung Hee University, 1 Seocheon, Giheung, Yongin, Gyeonggi, South Korea and nampv@khu.ac.kr;
(4) Xuan-Qui Pham, Department of Computer Engineering, Kyung Hee University, 1 Seocheon, Giheung, Yongin, Gyeonggi, South Korea and pxuanqui@khu.ac.kr;
(5) Quang-Thai Ngo, Department of Computer Engineering, Kyung Hee University, 1 Seocheon, Giheung, Yongin, Gyeonggi, South Korea and nqthai@khu.ac.kr;
(6) Eui-Nam Huh, Department of Computer Engineering, Kyung Hee University, 1 Seocheon, Giheung, Yongin, Gyeonggi, South Korea and johnhuh@khu.ac.kr. Abstract and 1 Introduction Abstract and 1 Introduction 2 Related Work 2 Related Work 3 Image Compression Module 3 Image Compression Module 4 System Architecture 4 System Architecture 5 Experimental Results 5 Experimental Results 6 Conclusions 6 Conclusions A Compression Results A Compression Results References References 2 Related Work Popular remote display protocols offer high-resolution sessions, multimedia stream remoting, multi-monitor support, dynamic object compression, USB redirection, drive mapping and more. Microsoft Remote Desktop Protocol (RDP), VMwares PC-over-IP (PCoIP) and Citrixs HDX are the most commonly used, but there are other protocols from companies such as Ericom and Hewlett-Packard. Teradici sells hardware solutions based on PCoIP technology [10] and the Thin Client Internet Computing protocol is also applicable for remote displays [11]. But these protocols still have drawbacks; great performance requires a lot of bandwidth and energy, which can clog the network [35]. Plus, if the system is running with low CPU usage, protocols will hog bandwidth and weaken end user performance. So, an efficient system needs to cope with unstable network conditions, to operate in real time, and to satisfy the users’ demands. Almost all of these protocols have the disadvantages that they do not consider the network environment and are inconvenient to use with real-time applications such as video playback. Previous work has included the development of an adaptive encoding application to solve display performance problems encountered when using RDP [12]. The authors of another work have described problems that arise in the case of video conferencing and have developed a real-time camera control system to address the problems [13]. To reduce the size of images transferred over the network to improve video streaming quality, compression technology has been one of the most important contributions. Some encoding and decoding formats such as motion MPEG [14], [16] and H.264 AVC [15], [17] have been developed in hybrid remote display protocol. The challenges for compressing images have been summarized previously [18]. Various compression techniques have been developed for image compression [37], [38]. In this work, we focus on the k-means clustering for encoding the image data. This method is probably the most popular technique of representativebased clustering. Cluster analysis [44] is a major data analysis method that is widely used for many practical applications in emerging areas. Clustering is the process of partitioning a given set of objects into disjoint clusters. This is done in such a way that objects in the same cluster are similar whereas objects belonging to different clusters differ considerably with respect to their attributes [48]. The k-means algorithm [45], [46] is effective in producing clusters for many practical applications. But the original k-means algorithm is very computationally complex, especially for large data sets. Moreover, this algorithm yields different clusters depending on the random choice of initial centroids. For example, previous work has included the description of a 2D image compression scheme using k-means in a real-time multiprocessor system [39]. Another work presented a novel k-means algorithm for compressing images, including exploration of the strategy of applying statistical parameters for choosing the initial seeds [40]. Despite the efficiency of the latter scheme, it still has drawbacks like a priori fixation of the number of clusters and random selection of initial seeds. Various researchers have made several attempts to improve the performance of k-means clustering [41], [42], [43]. In addition, hierarchical clustering algorithms also usually apply in image processing. But as the number of records increase the performance of hierarchical algorithm goes decreasing and time for execution increased. K-mean algorithm also increases its time of execution but as compared to hierarchical algorithm its performance is better. As a general conclusion, k-means algorithm is good for large dataset and hierarchical is good for small datasets [6]. Therefore k-means is most suitable for image processing in the video streaming scheme. Beside, developing a model that efficiently provides QoS control is presently a challenging problem. Many researches have addressed the problem of providing QoS guarantees to various network applications and clients. Most of them have focused on servers without considering network delays [19], [20], on individual network routers [29], [30], [31], or on clients with assumptions of QoS supports in networks [32]. Some recent work focused on end-to-end QoS guarantees in network cores [33], [34], with the aim of guaranteeing QoS as measured from server-side network edges to client-side network edges. To solve this problem, previous approaches need to be revised to allow their application to a cloud environment in a manner that responds to various user behaviors and network conditions. In the present work, k-means clustering is used to reducing the size of one frame from the video. We then develop a scheme for the analysis of historical data as a means to consider network status and make a decision regarding the optimal compression ratio for QoS control. Eventually, we evaluate our system model using actual measurement results resulting from the processing of images from a realistic image dataset. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv (1) Huu-Quoc Nguyen, Department of Computer Engineering, Kyung Hee University, 1 Seocheon, Giheung, Yongin, Gyeonggi, South Korea and quoc@khu.ac.kr; (2) Tien-Dung Nguyen, Department of Computer Engineering, Kyung Hee University, 1 Seocheon, Giheung, Yongin, Gyeonggi, South Korea and ntiendung@khu.ac.kr; (3) Van-Nam Pham, Department of Computer Engineering, Kyung Hee University, 1 Seocheon, Giheung, Yongin, Gyeonggi, South Korea and nampv@khu.ac.kr; (4) Xuan-Qui Pham, Department of Computer Engineering, Kyung Hee University, 1 Seocheon, Giheung, Yongin, Gyeonggi, South Korea and pxuanqui@khu.ac.kr; (5) Quang-Thai Ngo, Department of Computer Engineering, Kyung Hee University, 1 Seocheon, Giheung, Yongin, Gyeonggi, South Korea and nqthai@khu.ac.kr; (6) Eui-Nam Huh, Department of Computer Engineering, Kyung Hee University, 1 Seocheon, Giheung, Yongin, Gyeonggi, South Korea and johnhuh@khu.ac.kr. (1) Huu-Quoc Nguyen, Department of Computer Engineering, Kyung Hee University, 1 Seocheon, Giheung, Yongin, Gyeonggi, South Korea and quoc@khu.ac.kr; (2) Tien-Dung Nguyen, Department of Computer Engineering, Kyung Hee University, 1 Seocheon, Giheung, Yongin, Gyeonggi, South Korea and ntiendung@khu.ac.kr; (3) Van-Nam Pham, Department of Computer Engineering, Kyung Hee University, 1 Seocheon, Giheung, Yongin, Gyeonggi, South Korea and nampv@khu.ac.kr; (4) Xuan-Qui Pham, Department of Computer Engineering, Kyung Hee University, 1 Seocheon, Giheung, Yongin, Gyeonggi, South Korea and pxuanqui@khu.ac.kr; (5) Quang-Thai Ngo, Department of Computer Engineering, Kyung Hee University, 1 Seocheon, Giheung, Yongin, Gyeonggi, South Korea and nqthai@khu.ac.kr; (6) Eui-Nam Huh, Department of Computer Engineering, Kyung Hee University, 1 Seocheon, Giheung, Yongin, Gyeonggi, South Korea and johnhuh@khu.ac.kr.

Focusing on the K-Means Clustering for Encoding the Image Data

Fast-Coresets: A Nearly-Linear Time Algorithm for Efficient Clustering

Authors:
(1) Andrew Draganov, Aarhus University and All authors contributed equally to this research;
(2) David Saulpic, Université Paris Cité & CNRS;
(3) Chris Schwiegelshohn, Aarhus University. Table of Links Abstract and 1 Introduction 2 Preliminaries and Related Work 2.1 On Sampling Strategies 2.2 Other Coreset Strategies 2.3 Coresets for Database Applications 2.4 Quadtree Embeddings 3 Fast-Coresets 4 Reducing the Impact of the Spread 4.1 Computing a crude upper-bound 4.2 From Approximate Solution to Reduced Spread 5 Fast Compression in Practice 5.1 Goal and Scope of the Empirical Analysis 5.2 Experimental Setup 5.3 Evaluating Sampling Strategies 5.4 Streaming Setting and 5.5 Takeaways 6 Conclusion 7 Acknowledgements 8 Proofs, Pseudo-Code, and Extensions and 8.1 Proof of Corollary 3.2 8.2 Reduction of k-means to k-median 8.3 Estimation of the Optimal Cost in a Tree 8.4 Extensions to Algorithm 1 References 5.2 Experimental Setup Algorithms We compare Fast-Coresets (Algorithm 1) against 4 different benchmark sampling strategies that span the space between optimal time and optimal accuracy, as well as state-of-the-art competitors BICO [38] and Streamkm++[1]. - Standard uniform sampling. Each point is sampled with equal probability and weights are set to n/m, where m is the size of the sample. - Lightweight coresets [6]. Obtain a coreset by sampling sensitivity values with respect to the 1-means solution, i.e. sensitivities are given by sˆ(p) = 1/|P|+cost(p, µ)/ cost(P, µ), where µ is the dataset mean. - Standard sensitivity sampling [47]. We refer to Section 2. - BICO [38]. Utilizes BIRCH [58] to produce a k-means coreset in a stream. - Streamkm++ [1]. Uses k-means++ to create large coresets for the streaming k-means task. We do not compare against group sampling [25] as it uses sensitivity sampling as a subroutine and is merely a preprocessing algorithm designed to facilitate the theoretical analysis. By the authors’ own admission, it should not be implemented. We use j going forward to describe the number of centers in the candidate j-means solution. Thus, lightweight coresets have j = 1 while Fast-Coresets have j = k. We take a moment here to motivate the welterweight coreset algorithm. Consider that lightweight coresets use the 1-means solution to obtain the sensitivities that dictate the sampling distribution whereas sensitivity sampling uses the full k-means solution. In effect, as we change the value of j, the cluster sizes |Cp| in our approximate solution change. Referring back to equation 1, one can see that setting j between 1 and k acts as an interpolation between uniform and sensitivity sampling. We default to j = log k unless stated otherwise. We use the term ‘accelerated sampling methods’ when referring to uniform, lightweight and welterweight coresets as a group. Datasets. We complement our suite of real datasets with the following artificial datasets. We default to n = 50 000 and d = 50 unless stated otherwise. - c-outlier. Place n − c points in a single location and c points a large distance away. This has the property that all clusters have size n/k when γ = 0 and, as γ grows, the cluster sizes diverge at an exponential rate. We note that this is a well-cluster able instance with respect to cost stability conditions, see [3, 22, 46, 54]. The artificial datasets are constructed to emphasize strengths and weaknesses of the various sampling schemas. For example, the c-outlier problem contains very little information and, as such, should be simple for any sampling strategy that builds a reasonable representation of its input. The geometric dataset then increases the difficulty by having more regions of interest that must be sampled. The Gaussian mixture dataset is harder still, as it incorporates uneven inter-cluster distances and inconsistent cluster sizes. Lastly, the benchmark dataset is devised to be a worst-case example for sensitivity sampling. Data Parameters In all real and artificial datasets, we add random uniform noise η with 0 ≤ ηi ≤ 0.001 in each dimension in order to make all points unique. Unless specifically varying these parameters, we default all algorithms in 5.2 to k = 100 for the Adult, MNIST, Star, and artificial datasets and k = 500 for the Song, Cover Type, Taxi, and Census datasets. Our default coreset size is then m = 40k. We refer to the coreset size scalar (the “40” in the previous sentence) as the m-scalar. We only run the dimension-reduction step on the MNIST dataset, as the remaining datasets already have sufficiently low dimensionality. We run our experiments on an Intel Core i9 10940X 3.3GHz 14-Core processor. This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Andrew Draganov, Aarhus University and All authors contributed equally to this research; (2) David Saulpic, Université Paris Cité & CNRS; (3) Chris Schwiegelshohn, Aarhus University. Authors: Authors: (1) Andrew Draganov, Aarhus University and All authors contributed equally to this research; (2) David Saulpic, Université Paris Cité & CNRS; (3) Chris Schwiegelshohn, Aarhus University. Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2 Preliminaries and Related Work 2.1 On Sampling Strategies 2.1 On Sampling Strategies 2.2 Other Coreset Strategies 2.2 Other Coreset Strategies 2.3 Coresets for Database Applications 2.3 Coresets for Database Applications 2.4 Quadtree Embeddings 2.4 Quadtree Embeddings 3 Fast-Coresets 3 Fast-Coresets 4 Reducing the Impact of the Spread 4 Reducing the Impact of the Spread 4.1 Computing a crude upper-bound 4.1 Computing a crude upper-bound 4.2 From Approximate Solution to Reduced Spread 4.2 From Approximate Solution to Reduced Spread 5 Fast Compression in Practice 5 Fast Compression in Practice 5.1 Goal and Scope of the Empirical Analysis 5.1 Goal and Scope of the Empirical Analysis 5.2 Experimental Setup 5.2 Experimental Setup 5.3 Evaluating Sampling Strategies 5.3 Evaluating Sampling Strategies 5.4 Streaming Setting and 5.5 Takeaways 5.4 Streaming Setting and 5.5 Takeaways 6 Conclusion 6 Conclusion 7 Acknowledgements 7 Acknowledgements 8 Proofs, Pseudo-Code, and Extensions and 8.1 Proof of Corollary 3.2 8 Proofs, Pseudo-Code, and Extensions and 8.1 Proof of Corollary 3.2 8.2 Reduction of k-means to k-median 8.2 Reduction of k-means to k-median 8.3 Estimation of the Optimal Cost in a Tree 8.3 Estimation of the Optimal Cost in a Tree 8.4 Extensions to Algorithm 1 8.4 Extensions to Algorithm 1 References References 5.2 Experimental Setup Algorithms We compare Fast-Coresets (Algorithm 1) against 4 different benchmark sampling strategies that span the space between optimal time and optimal accuracy, as well as state-of-the-art competitors BICO [38] and Streamkm++[1]. Algorithms - Standard uniform sampling. Each point is sampled with equal probability and weights are set to n/m, where m is the size of the sample. - Standard uniform sampling. - Lightweight coresets [6]. Obtain a coreset by sampling sensitivity values with respect to the 1-means solution, i.e. sensitivities are given by sˆ(p) = 1/|P|+cost(p, µ)/ cost(P, µ), where µ is the dataset mean. - Lightweight coresets [6]. - Standard sensitivity sampling [47]. We refer to Section 2. - Standard sensitivity sampling [47]. - BICO [38] . Utilizes BIRCH [58] to produce a k-means coreset in a stream. - BICO [38] - Streamkm++ [1]. Uses k-means++ to create large coresets for the streaming k-means task. - Streamkm++ [1]. We do not compare against group sampling [25] as it uses sensitivity sampling as a subroutine and is merely a preprocessing algorithm designed to facilitate the theoretical analysis. By the authors’ own admission, it should not be implemented. We use j going forward to describe the number of centers in the candidate j-means solution. Thus, lightweight coresets have j = 1 while Fast-Coresets have j = k. We take a moment here to motivate the welterweight coreset algorithm. Consider that lightweight coresets use the 1-means solution to obtain the sensitivities that dictate the sampling distribution whereas sensitivity sampling uses the full k-means solution. In effect, as we change the value of j, the cluster sizes |Cp| in our approximate solution change. Referring back to equation 1, one can see that setting j between 1 and k acts as an interpolation between uniform and sensitivity sampling. We default to j = log k unless stated otherwise. We use the term ‘accelerated sampling methods’ when referring to uniform, lightweight and welterweight coresets as a group. Datasets . We complement our suite of real datasets with the following artificial datasets. We default to n = 50 000 and d = 50 unless stated otherwise. Datasets - c-outlier. Place n − c points in a single location and c points a large distance away. - c-outlier. This has the property that all clusters have size n/k when γ = 0 and, as γ grows, the cluster sizes diverge at an exponential rate. We note that this is a well-cluster able instance with respect to cost stability conditions, see [3, 22, 46, 54]. The artificial datasets are constructed to emphasize strengths and weaknesses of the various sampling schemas. For example, the c-outlier problem contains very little information and, as such, should be simple for any sampling strategy that builds a reasonable representation of its input. The geometric dataset then increases the difficulty by having more regions of interest that must be sampled. The Gaussian mixture dataset is harder still, as it incorporates uneven inter-cluster distances and inconsistent cluster sizes. Lastly, the benchmark dataset is devised to be a worst-case example for sensitivity sampling. Data Parameters In all real and artificial datasets, we add random uniform noise η with 0 ≤ ηi ≤ 0.001 in each dimension in order to make all points unique. Unless specifically varying these parameters, we default all algorithms in 5.2 to k = 100 for the Adult, MNIST, Star, and artificial datasets and k = 500 for the Song, Cover Type, Taxi, and Census datasets. Our default coreset size is then m = 40k. We refer to the coreset size scalar (the “40” in the previous sentence) as the m-scalar. We only run the dimension-reduction step on the MNIST dataset, as the remaining datasets already have sufficiently low dimensionality. We run our experiments on an Intel Core i9 10940X 3.3GHz 14-Core processor. Data Parameters This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

How Different Sampling Strategies Affect Clustering Accuracy

Authors:
(1) Andrew Draganov, Aarhus University and All authors contributed equally to this research;
(2) David Saulpic, Université Paris Cité & CNRS;
(3) Chris Schwiegelshohn, Aarhus University. Table of Links Abstract and 1 Introduction 2 Preliminaries and Related Work 2.1 On Sampling Strategies 2.2 Other Coreset Strategies 2.3 Coresets for Database Applications 2.4 Quadtree Embeddings 3 Fast-Coresets 4 Reducing the Impact of the Spread 4.1 Computing a crude upper-bound 4.2 From Approximate Solution to Reduced Spread 5 Fast Compression in Practice 5.1 Goal and Scope of the Empirical Analysis 5.2 Experimental Setup 5.3 Evaluating Sampling Strategies 5.4 Streaming Setting and 5.5 Takeaways 6 Conclusion 7 Acknowledgements 8 Proofs, Pseudo-Code, and Extensions and 8.1 Proof of Corollary 3.2 8.2 Reduction of k-means to k-median 8.3 Estimation of the Optimal Cost in a Tree 8.4 Extensions to Algorithm 1 References 4 Reducing the Impact of the Spread We now show how one can replace the linear time-dependency on log ∆ with a logarithmic one (i.e., log ∆ → log(log ∆)). Overview of the Approach Without loss of generality, we assume that the smallest pairwise distance is at least 1, and ∆ is a (known) upper-bound on the diameter of the input. To remove the log ∆ dependency, we proceed by producing a substitute dataset P ′ that has a lower-bound on its minimum distance and an upper-bound on its maximum distance. We then show that, with overwhelming likelihood, reasonable solutions on P ′ have the same cost as solutions on P up to an additive error. In order to produce the substitute dataset P ′ , we first find a crude upper-bound on the cost of the optimal solution to the clustering task on P. We then create a grid such that, for every cluster in the optimal solution, the cluster is entirely contained in a grid cell with overwhelming likelihood – in some sense, points in different grid cell do not interact. We can then freely move these grid cells around without affecting the cost of the solution, as long as they stay far enough away so that points in different cells still do not interact. Thus, we can constrain the diameter of P ′ to be small with respect to the quality of the approximate solution. We show later how to constrain the minimum distance to be comparable to the quality of this approximate solution as well. We will focus this section on the simpler k-median problem but show how to reduce k-means to this case in Section 8.2. 4.1 Computing a crude upper-bound As described, we start by computing an approximate solution U such that OPT ≤ U ≤ poly(n)· OPT. For this, the first step is to embed the input into a quadtree. This embedding has two key properties. First, distances are preserved up to a multiplicative factor O(d log ∆), and therefore the k-median cost is preserved up to this factor as well. Second, the metric is a hierarchically separated tree: it can be represented with a tree, where points of P are the leafs. The distance between two points is then given by the depth of their lowest common ancestor – if it is at depth ℓ, their distance is √ d∆2−ℓ . Our first lemma shows that finding the first level of the tree for which the input lies in k + 1 disjoint subtrees provides us with the desired approximation. We prove this in Section 8.3. A direct consequence is that the first level of the tree for which at least k + 1 cells are non empty allows us to compute an O(n)-approximation for k-median on the tree metric. Since the tree metric approximates the original Euclidean metric up to O(d log ∆), this is therefore an O(nd log ∆)-approximation to k-median in the Euclidean space. To turn this observation into an algorithm, one needs to count the number of non-empty cells at a given level ℓ: for each point, we identify the cell that contains it using modulo operations. Furthermore, we count the number of distinct non-empty cells using a standard dictionary data structure. This is done in time O˜(nd), and pseudo-code is given Algorithm 2. Using a binary search on the O(log ∆) many levels then gives the following result: Given this crude approximate solution, it remains to create a substitute dataset P ′ that satisfies two requirements: 1 First, P ′ must have spread linear in the quality of the approximate solution. If this holds, Algorithm 1 on P ′ will take O˜(nd log log ∆) time. 2 Second, any reasonable solution on P ′ should be roughly equivalent to a corresponding solution on P. This would mean that running Algorithm 1 on P ′ gives us a valid coreset for P. 4.2 From Approximate Solution to Reduced Spread Proposition 4.4. Let P ′ be the dataset produced by Algorithm 3. It holds that: Combining the algorithm of Lemma 4.2, which gives a bound on U, with Lemma 4.5 brings us to the following theorem: To summarize, we have shown that one can, in O˜(nd) time, find a modified dataset P ′ that preserves the cost of corresponding k-means and k-median solutions from P. Importantly, this P ′ has spread that is logarithmic in the spread of P. As a result, one can apply Algorithm 1 onto P ′ in O˜(nd) time without compromising the compression quality with respect to P. Lastly, this compression on P ′ can be re-formatted onto P in O˜(nd) time. This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Andrew Draganov, Aarhus University and All authors contributed equally to this research; (2) David Saulpic, Université Paris Cité & CNRS; (3) Chris Schwiegelshohn, Aarhus University. Authors: Authors: (1) Andrew Draganov, Aarhus University and All authors contributed equally to this research; (2) David Saulpic, Université Paris Cité & CNRS; (3) Chris Schwiegelshohn, Aarhus University. Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2 Preliminaries and Related Work 2.1 On Sampling Strategies 2.1 On Sampling Strategies 2.2 Other Coreset Strategies 2.2 Other Coreset Strategies 2.3 Coresets for Database Applications 2.3 Coresets for Database Applications 2.4 Quadtree Embeddings 2.4 Quadtree Embeddings 3 Fast-Coresets 3 Fast-Coresets 4 Reducing the Impact of the Spread 4 Reducing the Impact of the Spread 4.1 Computing a crude upper-bound 4.1 Computing a crude upper-bound 4.2 From Approximate Solution to Reduced Spread 4.2 From Approximate Solution to Reduced Spread 5 Fast Compression in Practice 5 Fast Compression in Practice 5.1 Goal and Scope of the Empirical Analysis 5.1 Goal and Scope of the Empirical Analysis 5.2 Experimental Setup 5.2 Experimental Setup 5.3 Evaluating Sampling Strategies 5.3 Evaluating Sampling Strategies 5.4 Streaming Setting and 5.5 Takeaways 5.4 Streaming Setting and 5.5 Takeaways 6 Conclusion 6 Conclusion 7 Acknowledgements 7 Acknowledgements 8 Proofs, Pseudo-Code, and Extensions and 8.1 Proof of Corollary 3.2 8 Proofs, Pseudo-Code, and Extensions and 8.1 Proof of Corollary 3.2 8.2 Reduction of k-means to k-median 8.2 Reduction of k-means to k-median 8.3 Estimation of the Optimal Cost in a Tree 8.3 Estimation of the Optimal Cost in a Tree 8.4 Extensions to Algorithm 1 8.4 Extensions to Algorithm 1 References References 4 Reducing the Impact of the Spread We now show how one can replace the linear time-dependency on log ∆ with a logarithmic one (i.e., log ∆ → log(log ∆)). Overview of the Approach Without loss of generality, we assume that the smallest pairwise distance is at least 1, and ∆ is a (known) upper-bound on the diameter of the input. To remove the log ∆ dependency, we proceed by producing a substitute dataset P ′ that has a lower-bound on its minimum distance and an upper-bound on its maximum distance. We then show that, with overwhelming likelihood, reasonable solutions on P ′ have the same cost as solutions on P up to an additive error. Overview of the Approach In order to produce the substitute dataset P ′ , we first find a crude upper-bound on the cost of the optimal solution to the clustering task on P. We then create a grid such that, for every cluster in the optimal solution, the cluster is entirely contained in a grid cell with overwhelming likelihood – in some sense, points in different grid cell do not interact. We can then freely move these grid cells around without affecting the cost of the solution, as long as they stay far enough away so that points in different cells still do not interact. Thus, we can constrain the diameter of P ′ to be small with respect to the quality of the approximate solution. We show later how to constrain the minimum distance to be comparable to the quality of this approximate solution as well. We will focus this section on the simpler k-median problem but show how to reduce k-means to this case in Section 8.2. 4.1 Computing a crude upper-bound As described, we start by computing an approximate solution U such that OPT ≤ U ≤ poly(n)· OPT. For this, the first step is to embed the input into a quadtree. This embedding has two key properties. First, distances are preserved up to a multiplicative factor O(d log ∆), and therefore the k-median cost is preserved up to this factor as well. Second, the metric is a hierarchically separated tree: it can be represented with a tree, where points of P are the leafs. The distance between two points is then given by the depth of their lowest common ancestor – if it is at depth ℓ, their distance is √ d∆2−ℓ . Our first lemma shows that finding the first level of the tree for which the input lies in k + 1 disjoint subtrees provides us with the desired approximation. We prove this in Section 8.3. A direct consequence is that the first level of the tree for which at least k + 1 cells are non empty allows us to compute an O(n)-approximation for k-median on the tree metric. Since the tree metric approximates the original Euclidean metric up to O(d log ∆), this is therefore an O(nd log ∆)-approximation to k-median in the Euclidean space. To turn this observation into an algorithm, one needs to count the number of non-empty cells at a given level ℓ: for each point, we identify the cell that contains it using modulo operations. Furthermore, we count the number of distinct non-empty cells using a standard dictionary data structure. This is done in time O˜(nd), and pseudo-code is given Algorithm 2. Using a binary search on the O(log ∆) many levels then gives the following result: Given this crude approximate solution, it remains to create a substitute dataset P ′ that satisfies two requirements: 1 First, P ′ must have spread linear in the quality of the approximate solution. If this holds, Algorithm 1 on P ′ will take O˜(nd log log ∆) time. 2 Second, any reasonable solution on P ′ should be roughly equivalent to a corresponding solution on P. This would mean that running Algorithm 1 on P ′ gives us a valid coreset for P. 4.2 From Approximate Solution to Reduced Spread Proposition 4.4. Let P ′ be the dataset produced by Algorithm 3. It holds that: Proposition 4.4. Let P ′ be the dataset produced by Algorithm 3. It holds that: Combining the algorithm of Lemma 4.2, which gives a bound on U, with Lemma 4.5 brings us to the following theorem: To summarize, we have shown that one can, in O˜(nd) time, find a modified dataset P ′ that preserves the cost of corresponding k-means and k-median solutions from P. Importantly, this P ′ has spread that is logarithmic in the spread of P. As a result, one can apply Algorithm 1 onto P ′ in O˜(nd) time without compromising the compression quality with respect to P. Lastly, this compression on P ′ can be re-formatted onto P in O˜(nd) time. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

Stories, News & Mentions about CLUSTER

Cluster’s Stories on HackerNoon