OpenShift Machine Config Server (MCS) Connection Refused on Port 22623: How to Fix and Add New…

OpenShift Machine Config Server (MCS) Connection Refused on Port 22623: How to Fix and Add New Nodes Successfully

If you’ve ever attempted to add a new node to an OpenShift (OCP) cluster and encountered issues with the Machine Config Server (MCS) rejecting connections on port 22623, you’re not alone. This security measure, introduced in OpenShift 4.5+, was designed to protect sensitive configurations but can create significant operational hurdles, preventing worker and master nodes from retrieving their required settings.

In this article, we’ll explore why this issue occurs, its impact on cluster scalability, and — most importantly — how to automate the fix with a custom script.

The Problem: MCS Connection Refused

When adding a new node, it needs to pull its configuration from the Machine Config Server. However, you may see an error like this:

# curl -kv https://api.test.mydomain.example.com:22623/config/worker 
*   Trying 192.168.1.1... 
* TCP_NODELAY set 
* connect to 192.168.1.1 port 22623 failed: Connection refused 
* Failed to connect to api.test.mydomain.example.com port 22623: Connection refused 
* Closing connection 0 
curl: (7) Failed to connect to api.test.mydomain.example.com port 22623: Connection refused

Strangely, you can access the same URL from a bastion host but not from the OpenShift nodes themselves.

Why This Happens

Starting with OCP 4.5, OpenShift automatically blocks access to port 22623 from within the cluster using iptables rules. This is done for security reasons to prevent pods with elevated privileges from accessing sensitive node configurations.

To check if your nodes are affected, run:

iptables -L | grep 22623

You’ll likely see something like:

REJECT     tcp  --  anywhere             anywhere             tcp dpt:22623 flags:FIN,SYN,RST,ACK/SYN reject-with icmp-port-unreachable

This explicitly blocks traffic on port 22623, preventing nodes from retrieving their configs.

Why OpenShift Blocks This Port

Security Measure: Prevents unauthorized access to machine configs from privileged containers.
Cluster Hardening: Reduces attack surface for internal threats.
Side Effect: Breaks node scaling and bootstrap processes.

Impact: Why This Is a Big Deal

If a new node cannot access the Machine Config Server, it:

Fails to retrieve its ignition config.
Stalls during provisioning.
Does not join the cluster and remains in a non-ready state.

For organizations that frequently scale their clusters or perform automated node onboarding, this behavior is a major blocker.

Solution: Removing the Firewall Rules

To allow new nodes to retrieve their configurations, we need to remove the iptables rules blocking port 22623.

Manual Fix

You can remove the rule manually on a single node:

iptables -D FORWARD -p tcp --dport 22623 -j REJECT 
iptables -D OUTPUT -p tcp --dport 22623 -j REJECT

However, this is not scalable for large clusters.

Automated Fix: Using a Custom Script

To automate this across all nodes, I created a Bash script that:

Identifies all cluster nodes.
Detects whether they use iptables or nftables.
Removes the blocking firewall rules on each node.
Ensures all nodes can reach the MCS without manual intervention.

Repository Github

At the moment, I cannot directly execute scripts from a GitHub repository. However, you can follow these steps to clone the repository and execute the script on your local machine:

Step 1: Clone the Repository

git clone https://github.com/Gabrielandre02/openshift-iptables-22623 
cd openshift-iptables-22623/EN-US/

Step 2: Grant Execution Permission

chmod +x iptables-openshift-22623.sh

Step 3: Run the Script

./iptables-openshift-22623.sh

Important Note

Before running the script, make sure you are logged into your OpenShift cluster API using the oc CLI:

oc login https://your-openshift-api:6443 -u <your-username> -p <your-password>

Once logged in, the script will automatically remove firewall rules blocking port 22623, allowing your new nodes to retrieve their configurations and join the cluster successfully.

Verify the Fix

If no results appear, the fix was successful.

Next, test access to the Machine Config Server:

curl -kv https://api.test.mydomain.example.com:22623/config/worker

Now, you can proceed with the process and add new nodes to your OpenShift cluster.

💡 Who Am I?

I am Gabriel Carmo, passionate about technology (especially Open Source). I have experience in Cloud, Kubernetes, OpenShift, Zabbix, Dynatrace, and more! Always exploring new technologies and sharing knowledge. 🚀

📬 Let's Connect?
🔗 LinkedIn
🐙 GitHub
🦊 GitLab
🏅 Credly
📧 Contact: contato@gabrielandre.com.br