Nagios getting error WARNING: my_system() seteuid(0): Operation not permitted

I have a Nagios 4.3.4 server running that monitors all of our infrastructures.  I’m always adding plugins and checks for all of our servers.  I added a new server that was running Centos 7.3 and Kerio Connect.  This server has RAID5, and the plugin I’m using specifically needs to sudo to read the RAID data.

I’ve got Nagios setup with the following settings

define service {
 use host-service
 host_name keriomail
 service_description RAID
 check_command check_nrpe!check_perc5i
 max_check_attempts       3
 check_interval           10
 retry_interval           15
 contact_groups admins
}

Checking the Nagios Server

It’s a remote NRPE call that when Nagios is trying to connect and get the results, it’s returning an error.  If I manually run the check_nrpe command I get:

/usr/lib64/nagios/plugins/check_nrpe -H keriomail.example.com -c check_perc5i

Command line on the Nagios server the result is:

NRPE: Unable to read output

Checking the NRPE Client

Verify in my configuration that the check_perc5i is appropriately set up.

[root@keriomail ~]# grep perc5i /etc/nrpe.d/local.cfg 
command[check_perc5i]=/usr/bin/sudo /usr/lib64/nagios/plugins/check_perc5i

If I run as root, I get the desired result.

[root@keriomail ~]# /usr/bin/sudo /usr/lib64/nagios/plugins/check_perc5i
OKAY - VirtualDrives=1, Degraded=0, Offline=0, PhysicalDevices=7, Disks=6, CriticalDisks=0, FailedDisks=0, MemoryCorrectableErrors=0, MemoryUncorrectableErrors=0

Yes, RAID check outputs the correct result.

Next, verify that we’ve got sudoers setup correctly that allows user Nagios to execute the script as root.

[root@keriomail ~]# grep nagios /etc/sudoers.d/nagios 
Defaults:nagios !requiretty
nagios      ALL=(ALL)   NOPASSWD:   /usr/lib64/nagios/plugins/

Then check to make sure nrpe daemon is running

[root@keriomail ~]# ps aux | grep nrpe
nrpe     16590  0.0  0.0  44884  1524 ?        Ss   16:12   0:00 /usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -d

Yes, nrpe is running.

Those all look right, so what am I missing?

Debugging NRPE Setup

Now that we verified our setup let’s do some debugging now to track down where the issue is occurring.  I’m going to enable debugging on the nrpe server.  In the nrpe.cfg file I enabled debug

[root@keriomail ~]# grep debug /etc/nagios/nrpe.cfg 
# This option determines whether or not debugging messages are logged to the
# Values: 0=debugging off, 1=debugging on
debug=1

Then don’t forget to reload your configuration

[root@keriomail ~]# systemctl restart nrpe

Now we can tail the logs and see what is happening when we run the check_nrpe call from our Nagios server.

Here are the error messages I see my the messages log:

[root@ keriomail ~]# tail -f /var/log/messages | grep nrpe

Jul  8 11:34:21 keriomail nrpe[5045]: CONN_CHECK_PEER: checking if host is allowed: 208.81.236.29 port 4832
Jul  8 11:34:21 keriomail nrpe[5045]: is_an_allowed_host (AF_INET): is host >208.81.236.29< an allowed host >208.81.236.29<
Jul  8 11:34:21 keriomail nrpe[5045]: is_an_allowed_host (AF_INET): host is in allowed host list!
Jul  8 11:34:21 keriomail nrpe[5046]: WARNING: my_system() seteuid(0): Operation not permitted

We’re getting an error in the log, that the seteuid is not permitted.  Which looks like when attempting to run the command with sudo isn’t functioning properly.  So let’s take a closer look at our sudo permissions.

Resolving Unable to Read Output

Let’s test some changes, I remembered I copied my settings from a CentOS 6.9 server, so maybe there is a difference between them.

nagios    1803  0.0  0.0  41560  2892 ?        Ss   Jun24   0:14 /usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -d

And we run to the same check for nrpe on CentOS 7.3 server that we’re having problems.

nrpe 16590 0.0 0.0 44884 1524 ? Ss 16:12 0:00 /usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -d

I can see that on CentoOS 6.x nrpe is running as user nagios and on CentOS 7.x it’s running as user nrpe.  So that means our sudoers settings are incorrect, that is probably what’s generating the error.

So I’m going to make one more change, and change the user from nagios to nrpe in my settings

[root@keriomail ~]# grep nrpe /etc/sudoers.d/nagios 
Defaults:nrpe !requiretty
nrpe ALL=(ALL) NOPASSWD: /usr/lib64/nagios/plugins/

After making that change, I run the check_nrpe command one more time.

[root@nagiosserver ~]# /usr/lib64/nagios/plugins/check_nrpe -H keriomail.example.com -c check_perc5i
OKAY - VirtualDrives=1, Degraded=0, Offline=0, PhysicalDevices=7, Disks=6, CriticalDisks=0, FailedDisks=0, MemoryCorrectableErrors=0, MemoryUncorrectableErrors=0

That resolved my problem, username miss match.

So make sure you’re sudoers is using the correct username nrpe when going from CentOS 6.x to 7.x.