Pacemaker Cluster (HA) Basic and Advanced QA

Basic Questions and Answers

• Q: What is Pacemaker, and what problem does it solve?

  • A: Pacemaker is a cluster resource manager for Linux. It manages the availability of services (resources) across a cluster of machines. If one node in the cluster fails, Pacemaker automatically moves the services to another healthy node, ensuring high availability. It solves the problem of single points of failure.

• Q: What are the key components of a Pacemaker cluster?

  • A:
    • Corosync: Provides the messaging and membership layer. It's responsible for cluster communication and detecting node failures.
    • Pacemaker: The core resource manager. It makes decisions about where to run resources based on policies and constraints.
    • Resource Agents (RAs): Scripts or programs that control the start, stop, and monitoring of services (resources). They interface between Pacemaker and the actual services.

• Q: What is a resource in the context of Pacemaker?

  • A: A resource is a service or application that Pacemaker manages. It could be a web server, a database, a virtual IP address, or any other service that needs to be highly available.

• Q: What are constraints in Pacemaker, and why are they important?

  • A: Constraints define rules about where resources can run. They are crucial for controlling the behavior of the cluster. Common constraint types include:
    • Location Constraints: Specify preferred or forbidden nodes for a resource.
    • Ordering Constraints: Define the order in which resources should be started or stopped.
    • Colocation Constraints: Specify which resources should run on the same node or on different nodes.

Advanced Questions and Answers

• Q: How does Pacemaker handle fencing (STONITH)?

  • A: STONITH (Shoot The Other Node In The Head) is a critical component for preventing data corruption in a cluster. When a node fails, Pacemaker needs to ensure that it's truly offline before promoting resources to another node. STONITH devices (e.g., IPMI, power switches) are used to physically power off or reboot the failed node, guaranteeing that it can't interfere.

• Q: Explain the concept of quorum in a Pacemaker cluster.

  • A: Quorum is the minimum number of nodes that must be online for the cluster to function correctly. It's essential for preventing split-brain scenarios, where the cluster is divided into two or more parts, each thinking it's the primary. Quorum is usually half the number of nodes plus one.

• Q: How do you configure resource dependencies and ordering in Pacemaker?

  • A: Resource dependencies and ordering are managed using constraints:
    • Ordering constraints: Control the startup and shutdown order of resources. For example, a database must be started before the applications that use it.
    • Colocation constraints: Ensure that related resources run on the same node (e.g., web server and database) or on different nodes.

• Q: What are some strategies for testing a Pacemaker cluster failover?

  • A: Testing is crucial to ensure that the cluster behaves as expected. Common methods include:
    • Simulating node failures: Physically powering off a node or using tools to simulate a failure.
    • Resource failures: Simulating failures of individual resources (e.g., stopping a service).
    • Network partitioning: Simulating network issues to test how the cluster handles communication problems.

• Q: How do you troubleshoot a Pacemaker cluster?

  • A: Troubleshooting requires examining logs and using Pacemaker's command-line tools:
    • crm status: Shows the current status of the cluster and resources.
    • crm_mon: Monitors the cluster in real-time.
    • Examining Corosync and Pacemaker logs for error messages.

• Q: What are some best practices for configuring and managing a Pacemaker cluster?

  • A:
    • Use a minimum of three nodes for production clusters to avoid split-brain issues.
    • Properly configure STONITH to prevent data corruption.
    • Thoroughly test failover scenarios.
    • Use constraints effectively to manage resource dependencies and placement.
    • Monitor the cluster closely and set up alerts for critical events.
    • Plan for regular maintenance and updates.

====================================================================

Pacemaker Cluster - Basic & Advanced Interview Questions and Answers

Pacemaker is a high-availability cluster manager used in Linux systems for failover and redundancy. Below are some basic and advanced questions with answers for Linux administrators.

---

🛠 Basic Pacemaker Cluster Questions

1️⃣ What is Pacemaker in Linux?

Answer:
Pacemaker is an open-source high-availability (HA) cluster manager that manages failover, resource allocation, and node monitoring to ensure system uptime. It works with Corosync to provide a complete HA solution.

---

2️⃣ What are the key components of Pacemaker?

Answer:
Pacemaker consists of three main components:

• Cluster Information Base (CIB): Stores cluster configuration and status.
• Policy Engine (PE): Determines where and how resources run.
• Controller (CRMd): Executes cluster changes and ensures nodes are in sync.

---

3️⃣ What is Corosync, and why is it used with Pacemaker?

Answer:
Corosync is a cluster communication system responsible for:

• Messaging between nodes.
• Quorum management.
• Node membership detection.

Pacemaker relies on Corosync for node coordination and cluster messaging.

---

4️⃣ How do you install Pacemaker on RHEL 8?

Answer:
Run the following commands:

sudo dnf install pacemaker corosync pcs -y

Enable and start the necessary services:

sudo systemctl enable --now pcsd
sudo systemctl enable --now corosync
sudo systemctl enable --now pacemaker

---

5️⃣ How do you configure a basic Pacemaker cluster?

Answer:
1️⃣ Set the cluster nodes:

sudo pcs cluster auth node1 node2 --username hacluster --password mypassword

2️⃣ Create and start the cluster:

sudo pcs cluster setup --name mycluster node1 node2
sudo pcs cluster start --all

3️⃣ Enable the cluster to start at boot:

sudo pcs cluster enable --all

---

6️⃣ How do you check the status of a Pacemaker cluster?

Answer:

pcs status

or

crm_mon -1

---

7️⃣ What is STONITH in Pacemaker?

Answer:
STONITH (Shoot The Other Node in The Head) is a fencing mechanism that prevents split-brain scenarios by forcibly rebooting or shutting down an unresponsive node.

Enable STONITH in Pacemaker:

sudo pcs property set stonith-enabled=true

---

🛠 Advanced Pacemaker Cluster Questions

8️⃣ How does Pacemaker decide which node should run a resource?

Answer:
Pacemaker uses constraints and scores to determine node selection:

• Location Constraints – Assigns preferred nodes.
• Order Constraints – Ensures resources start in a specific order.
• Colocation Constraints – Ensures resources run together.

Example:

pcs constraint location myresource prefers node1=100

---

9️⃣ How do you manually failover a resource in Pacemaker?

Answer:
Move a resource to another node:

pcs resource move myresource node2

To return it to its original node:

pcs resource clear myresource

---

🔟 How do you add a new resource to the cluster?

Answer:
For example, to add an Apache service as a cluster resource:

pcs resource create webserver ocf:heartbeat:apache configfile=/etc/httpd/conf/httpd.conf op monitor interval=30s

---

1️⃣1️⃣ How do you configure a resource to restart after failure?

Answer:
Set resource-stickiness and failover-policy:

pcs resource defaults resource-stickiness=100
pcs resource defaults migration-threshold=3

This allows a resource to fail 3 times before moving to another node.

---

1️⃣2️⃣ What is quorum in Pacemaker, and how is it managed?

Answer:
Quorum ensures a cluster has the majority of active nodes before taking action. If quorum is lost, Pacemaker prevents split-brain issues.

Check quorum status:

pcs quorum status

Disable quorum enforcement (for 2-node clusters):

pcs property set no-quorum-policy=ignore

---

1️⃣3️⃣ How do you remove a node from a Pacemaker cluster?

Answer:
Stop the node:

pcs cluster stop node2

Remove from the cluster:

pcs cluster node remove node2

---

1️⃣4️⃣ How do you recover a failed Pacemaker cluster node?

Answer:
1️⃣ Reboot the node:

sudo reboot

2️⃣ Start Pacemaker services:

sudo systemctl start pacemaker corosync

3️⃣ Re-add the node (if needed):

pcs cluster node add node2

---

1️⃣5️⃣ How do you troubleshoot Pacemaker issues?

Answer:
📌 Check cluster logs:

journalctl -xe

📌 Check Corosync logs:

cat /var/log/corosync/corosync.log

📌 Check Pacemaker logs:

cat /var/log/pacemaker.log

📌 Check resource status:

pcs status

---

🛠 Summary of Key Pacemaker Commands

Task	Command
Install Pacemaker	dnf install pacemaker corosync pcs -y
Start Cluster	pcs cluster start --all
Check Cluster Status	pcs status
Enable Cluster on Boot	pcs cluster enable --all
Add a Resource	pcs resource create <resource>
Move Resource to Another Node	pcs resource move <resource> <node>
Remove a Node	pcs cluster node remove <node>
View Logs	journalctl -xe

---

🚀 Conclusion

• Pacemaker + Corosync provide high availability for Linux applications.
• Constraints, fencing (STONITH), and quorum are crucial for preventing failures.
• Mastering pcs commands helps in troubleshooting and managing clusters efficiently.

Would you like help with Pacemaker automation scripts? 🚀