Leia este artigo em português.
OpenShift Machine Config Server (MCS) Connection Refused on Port 22623: How to Fix and Add New Nodes Successfully. If you’ve ever attempted to add a new node to an OpenShift (OCP) cluster and encountered issues with the Machine Config Server (MCS) rejecting connections on port 22623, you’re not alone. This security measure, introduced in OpenShift 4.5+, was designed to protect sensitive configurations but can create significant operational hurdles, preventing worker and master nodes from retrieving their required settings.
In this article, we’ll explore why this issue occurs, its impact on cluster scalability, and — most importantly — how to automate the fix with a custom script.
The Problem: MCS Connection Refused
When adding a new node, it needs to pull its configuration from the Machine Config Server. However, you may see an error like this:
# curl -kv https://api.test.mydomain.example.com:22623/config/worker
* Trying 192.168.1.1...
* TCP_NODELAY set
* connect to 192.168.1.1 port 22623 failed: Connection refused
* Failed to connect to api.test.mydomain.example.com port 22623: Connection refused
* Closing connection 0
curl: (7) Failed to connect to api.test.mydomain.example.com port 22623: Connection refusedStrangely, you can access the same URL from a bastion host but not from the OpenShift nodes themselves.
Why This Happens
Starting with OCP 4.5, OpenShift automatically blocks access to port 22623 from within the cluster using iptables rules. This is done for security reasons to prevent pods with elevated privileges from accessing sensitive node configurations.
To check if your nodes are affected, run:
iptables -L | grep 22623You’ll likely see something like:
REJECT tcp -- anywhere anywhere tcp dpt:22623 flags:FIN,SYN,RST,ACK/SYN reject-with icmp-port-unreachableThis explicitly blocks traffic on port 22623, preventing nodes from retrieving their configs.
Why OpenShift Blocks This Port
- Security Measure: Prevents unauthorized access to machine configs from privileged containers.
- Cluster Hardening: Reduces attack surface for internal threats.
- Side Effect: Breaks node scaling and bootstrap processes.
Impact: Why This Is a Big Deal
If a new node cannot access the Machine Config Server, it:
- Fails to retrieve its ignition config.
- Stalls during provisioning.
- Does not join the cluster and remains in a non-ready state.
For organizations that frequently scale their clusters or perform automated node onboarding, this behavior is a major blocker.
Solution: Removing the Firewall Rules
To allow new nodes to retrieve their configurations, we need to remove the iptables rules blocking port 22623.
Manual Fix
You can remove the rule manually on a single node:
iptables -D FORWARD -p tcp --dport 22623 -j REJECT
iptables -D OUTPUT -p tcp --dport 22623 -j REJECTHowever, this is not scalable for large clusters.
Automated Fix: Using a Custom Script
To automate this across all nodes, I created a Bash script that:
- Identifies all cluster nodes.
- Detects whether they use
iptablesornftables. - Removes the blocking firewall rules on each node.
- Ensures all nodes can reach the MCS without manual intervention.
Repository Github
At the moment, I cannot directly execute scripts from a GitHub repository. However, you can follow these steps to clone the repository and execute the script on your local machine:
Step 1: Clone the Repository
git clone https://github.com/Gabrielandre02/openshift-iptables-22623
cd openshift-iptables-22623/EN-US/Step 2: Grant Execution Permission
chmod +x iptables-openshift-22623.shStep 3: Run the Script
./iptables-openshift-22623.shImportant Note
Before running the script, make sure you are logged into your OpenShift cluster API using the oc CLI:oc login https://your-openshift-api:6443 -u <your-username> -p <your-password>
Once logged in, the script will automatically remove firewall rules blocking port 22623, allowing your new nodes to retrieve their configurations and join the cluster successfully.
Verify the Fix
If no results appear, the fix was successful.
Next, test access to the Machine Config Server:
curl -kv https://api.test.mydomain.example.com:22623/config/workerNow, you can proceed with the process and add new nodes to your OpenShift cluster.
💡 Who am I?
I’m Gabriel Carmo, a CNCF Kubestronaut holding the CKA, CKAD, CKS, KCNA, and KCSA certifications, as well as Red Hat Certified OpenShift Administrator.
I work in DevOps Engineering, building and evolving multi-cloud Kubernetes platforms. My work focuses on reducing engineering teams’ cognitive load through platform engineering, automation, GitOps, and self-service experiences.
In my daily work, I connect infrastructure as code, cloud networking, governance, security, observability, and SRE practices to build more standardized, resilient, and operable platforms.