Problem Note
68358:
The ETL driver fails to connect with CAS and errors occur when the sas.ops-agent, sas.ops-agentsrv, and sas.ops service IDs are not registered
As part of its metric collection and publishing, the SAS Viya operations infrastructure uses three internal service IDs that are registered with the identities service. These IDs are internal service IDs and are not normally displayed in SAS
®
Environment Manager.
The sas.ops-agentsrv internal service ID is used by the SAS Viya operations agent server (sas-viya-ops-agent-default) and other commands (for example,
sas-ops
validate
)
.
The sas.ops-agent internal service ID is used by the SAS Viya operations agent (sas-viya-ops-agent-default).
The sas.ops-agent internal service ID is used by components such as the sas-peek collector when executed by the SAS Viya operations agent server (sas-viya-ops-agentsrv-default).
The sas.ops internal service ID is used by components such as the sas-peek collector when it is executed manually as the
sas
user ID.
In cases where these internal service IDs no longer can obtain valid OAuth tokens, various components of the SAS Viya operations infrastructure return errors.
When this issue occurs, the etl_driver.sas program that runs every five minutes might fail to connect to SAS
®
Cloud Analytic Services (CAS). In addition, the rolloff.sas program, which runs nightly at 2:00 a.m., might fail to connect to CAS with this error. Also, the audit.sas program, which is called by the ev-genaudit task every 2.5 hours, might fail to connect to CAS. When this problem occurs, a message similar to the following is displayed in the etl_driver_
yyyymmdd
_
run-ID
_
host-name
_
process_ID
.pgmlog file. This file resides in the
/opt/sas/viya/config/var/log/evmsvrops/evdm
directory on the [Operations] host that is defined in
sas_viya_playbook/inventory.ini
.
2021-09-02T15:32:02.765000-04:00|PGM|pax-3|sas|195073| The format [shifttxt] exists.
2021-09-02T15:32:02.825000-04:00|PGM|pax-3|sas|195073| Kerberos authentication failure in function gss_init_sec_context: Unspecified GSS failure. Minor code may provide more information.
2021-09-02T15:32:02.825000-04:00|PGM|pax-3|sas|195073| Additional information:
No Kerberos credentials available
(default cache: KCM.
2021-09-02T15:32:02.825000-04:00|PGM|pax-3|sas|195073| Kerberos initialization failed. Your credential cache is either expired or missing.
2021-09-02T15:32:02.827000-04:00|PGM|pax-3|sas|195073| ERROR: Kerberos initialization failed. Your credential cache is either expired or missing.
2021-09-02T15:32:02.827000-04:00|PGM|pax-3|sas|195073| ERROR: Unable to connect to Cloud Analytic Services pax-1.tum.sashq-d.openstack.sas.com on port 5570. Verify connection parameters
2021-09-02T15:32:02.827000-04:00|PGM|pax-3|sas|195073| and retry.
2021-09-02T15:32:02.834000-04:00|PGM|pax-3|sas|195073| NOTE: Data mart lock released; data mart no longer locked.
2021-09-02T15:32:02.834000-04:00|PGM|pax-3|sas|195073| ERROR: Terminating session because CAS connection information missing or incorrect (rc=[196]).
2021-09-02T15:32:02.834000-04:00|PGM|pax-3|sas|195073| ERROR: Execution terminated by an %ABORT statement.
2021-09-02T15:32:02.834000-04:00|PGM|pax-3|sas|195073|
2021-09-02T15:32:02.834000-04:00|PGM|pax-3|sas|195073| ERROR: Errors printed on page 3.
When you do not have the credentials to connect with CAS, the authentication mechanism falls back to Kerberos and a Kerberos error is displayed. This fact does not mean that the etl_driver.sas, rolloff.sas, or audit.sas programs actually use Kerberos to connect to CAS. Instead, because these SAS programs connect to CAS by using the sas.ops-agentsrv internal service ID, this error can occur if the sas.ops-agentsrv internal service ID becomes deregistered.
When you execute the command
/opt/sas/viya/home/bin/sas-ops
validate
-level
3
-verbose
on any host as the
sas
user ID, the following error might be returned:
Validation test(s) completed with 1 warnings(s) and 4 errors(s):
ERROR The system CASLIB [SystemData] could not be found
ERROR Unable to retrieve list of CAS servers: Get "
https://rint04-0023.race.sas.com/casManagement/servers
": oauth2: cannot fetch token: 401
Response: {"error":"unauthorized","error_description":"
Bad credentials
"}
ERROR Unable to obtain OAuth token for sas.ops-agentsrv: oauth2: cannot fetch token: 401
Response: {"error":"unauthorized","error_description":"Bad credentials"}
Because the
sas
-ops
validate
command uses the sas.ops-agentsrv internal service ID, this error can occur if the sas.ops-agentsrv internal service ID becomes deregistered.
When you manually execute the
/opt/sas/viya/home/bin/sas-peek
cas
-level
3
command as the
sas
user ID on the CAS controller or on the backup controller, the following errors might be returned:
bash-4.4$ EMI_LOG_LEVEL=debug /opt/sas/viya/home/bin/sas-peek cas -level 3
2021-09-02 15:18:49.397 DEBUG [metrics.go:37] [sas-peek] - OAuth client credentials found for sas.ops
2021-09-02 15:18:50.792 WARN [metrics.go:120] [sas-peek] - Get
https://pax-1.tum.sashq-d.openstack.sas.com:8777/cas
: oauth2: cannot fetch token: 401
Response: {"error":"unauthorized","error_description":"
Bad credentials
"}
2021-09-02 15:18:50.792 WARN [metrics.go:120] [sas-peek] - Get
https://pax-1.tum.sashq-d.openstack.sas.com:8777/cas/nodes/metrics
: oauth2: cannot fetch token: 401
Response: {"error":"unauthorized","error_description":"Bad credentials"}
2021-09-02 15:18:50.792 WARN [metrics.go:120] [sas-peek] - Get
https://pax-1.tum.sashq-d.openstack.sas.com:8777/system
: oauth2: cannot fetch token: 401
Response: {"error":"unauthorized","error_description":"Bad credentials"}
2021-09-02 15:18:50.792 WARN [metrics.go:120] [sas-peek] - Get
https://pax-1.tum.sashq-d.openstack.sas.com:8777/cas/nodes/memoryMetrics
: oauth2: cannot fetch token: 401
Response: {"error":"unauthorized","error_description":"Bad credentials"}
2021-09-02 15:18:50.792 WARN [metrics.go:120] [sas-peek] - Get
https://pax-1.tum.sashq-d.openstack.sas.com:8777/cas/nodes/cpuTime
: oauth2: cannot fetch token: 401
Response: {"error":"unauthorized","error_description":"Bad credentials"}
{"version":1,"collectorName":"sas-peek-cas","collectorVersion":"1.5.23+803659e","properties":{"consulNodeName":"pax-1.tum.sashq-d.openstack.sas.com","hostname":"pax-1.tum.sashq-d.openstack.sas.com","os":"linux_amd64"},"measurements":null,"timeStamp":"2021-09-02T15:18:49.223884-04:00"}
Even though the output in the EMI_LOG_LEVEL=DEBUG line returns the message
OAuth
client
credentials
found
for
sas.ops
, that message is misleading. The
Bad
credentials
error that is shown above means that the OAuth token that is found is not valid for the sas.ops internal service ID. This error can occur when the sas.ops internal service ID becomes deregistered.
The sas-viya-ops-agent-default agent schedules the
CASMetrics
task. This task calls the command
/opt/sas/viya/home/bin/sas-peek
cas
-level
3
. On CAS controller or the backup controller machines, the SAS Viya operations agent server (sas-viya-ops-agentsrv-default) might log errors in
/opt/sas/viya/config/var/log/ops-agent/default/sas-ops-agent_
yyyy
-mm-dd_hh-mm-ss
.log
under either of the following circumstances:
You modify the
/opt/sas/viya/home/bin/sas-ops-agentclt
script by using
-loquacity
9
to start the sas-viya-ops-agent-default agent for extra logging.
You turn on extra logging by submitting the command
/opt/sas/viya/home/bin/ops-agent-cmd set -name ops-agent -- -debug true
, which is not the default action.
The errors that occur are as follows:
2021-07-30 16:21:17.360 INFO [xmexec.go:115] [sas-ops-agent] - Task59-CASMetrics status Running
2021-07-30 16:21:17.360 INFO [xmexbuft.go:144] [sas-ops-agent] - Task59-CASMetrics starting in go-routine 9390
2021-07-30 16:21:17.509 INFO [xmexbuft.go:422] [sas-ops-agent] - Task59-CASMetrics command return code was 0
2021-07-30 16:21:17.750 INFO [xmexbuft.go:543] [sas-ops-agent] - Task59-CASMetrics publisher return code was 1
2021-07-30 16:21:17.750 DEBUG [xmpubinfo.go:86] [sas-ops-agent] - Task59-CASMetrics payload file name /opt/sas/viya/config/var/lib/evmcltsvcs/spool/ops-agent/payloads/CASMetrics.T59.P27949.json
2021-07-30 16:21:17.752 INFO [xmexbuft.go:572] [sas-ops-agent] - Task59-CASMetrics payload buffer saved 262 bytes
2021-07-30 16:21:17.752 INFO [xmexbuft.go:612] [sas-ops-agent] - Task59-CASMetrics program /opt/sas/viya/home/bin/sas-peek produced 262 bytes
2021-07-30 16:21:17.752 INFO [xmexbuft.go:614] [sas-ops-agent] - Task59-CASMetrics used 0.1991s system time and 0.6280s user time
2021-07-30 16:21:17.752 INFO [xmexbuft.go:616] [sas-ops-agent] - Task59-CASMetrics ended exit status 0
publisher exit status 1
2021-07-30 16:21:17.752 INFO [xmexbuft.go:636] [sas-ops-agent] - Task59-CASMetrics Publisher program /opt/sas/viya/home/bin/sas-event-pub error
2021-07-30 16:21:17.543 DEBUG [event_pub.go:192] [sas-event-pub] - loadPublishers
2021-07-30 16:21:17.543 DEBUG [event_pub.go:207] [sas-event-pub] - publisherDisabled [p1:console]
2021-07-30 16:21:17.543 DEBUG [event_pub.go:237] [sas-event-pub] - publisherDisabled [p1:http]
2021-07-30 16:21:17.543 DEBUG [event_pub.go:241] [sas-event-pub] - Configured to publish events to sas.metric
2021-07-30 16:21:17.601 DEBUG [event_pub.go:244] [sas-event-pub] - addPublisher [p1:amqp]
2021-07-30 16:21:17.742 DEBUG [event_pub.go:133] [sas-event-pub] - publishFailed [p1:
unknown payload type
: []]
2021-07-30 16:21:17.742 DEBUG [event_pub.go:139] [sas-event-pub] - sas-event-pub complete [successes:0
errors:1
]
When the command
sas-peek
cas
-level
3
is scheduled to run by the sas-viya-ops-agent-default agent, it uses the sas.ops-agent internal service ID. As a result, if the sas.ops-agent internal service ID is deregistered, the errors that are shown above occur.
If you execute the following commands in an attempt to resolve all of the issues that are described earlier, no errors are returned:
/opt/sas/viya/home/bin/ops-util register -id sas.ops
/opt/sas/viya/home/bin/ops-util register -id sas.ops-agent
/opt/sas/viya/home/bin/ops-util register -id sas.ops-agentsrv
However, when you submit the
ops-util
command with the
token
option, you still might see the following errors:
bash-4.4$ /opt/sas/viya/home/bin/ops-util token -id sas.ops-agent
oauth2: cannot fetch token: 401
Response: {"error":"unauthorized","error_description":"
Bad credentials
"}
bash-4.4$ /opt/sas/viya/home/bin/ops-util token -id sas.ops-agentsrv
oauth2: cannot fetch token: 401
Response: {"error":"unauthorized","error_description":"Bad credentials"}
bash-4.4$ /opt/sas/viya/home/bin/ops-util token -id sas.ops
oauth2: cannot fetch token: 401
Response: {"error":"unauthorized","error_description":"Bad credentials"}
In this message,
Bad
credentials
means that an OAuth token cannot be obtained for the given service ID.
To resolve these issues, submit the following commands as the
sas
user ID, but add the
-force
option:
/opt/sas/viya/home/bin/ops-util register -force -id sas.ops-agentsrv
/opt/sas/viya/home/bin/ops-util register -force -id sas.ops-agent
/opt/sas/viya/home/bin/ops-util register -force -id sas.ops
To test whether an OAuth token can be obtained for the given ID, submit the following commands:
/opt/sas/viya/home/bin/ops-util token -id sas.ops-agentsrv
/opt/sas/viya/home/bin/ops-util token -id sas.ops-agent
/opt/sas/viya/home/bin/ops-util token -id sas.ops
Note:
You can add the
-debug
option, as shown below, for more verbose logging:
/opt/sas/viya/home/bin/ops-util -debug token -id sas.ops-agentsrv
/opt/sas/viya/home/bin/ops-util -debug token -id sas.ops-agent
/opt/sas/viya/home/bin/ops-util -debug token -id sas.ops
To avoid the need to reregister the service IDs in the future, submit the following commands as the
sas
user ID to modify the tasks that call
ops-util
and add the
-force
option. Making these modifications ensures that given IDs are registered with the identities service:
/opt/sas/viya/home/bin/ops-config -base config/ops-agentsrv/sas.ops.task/registerOpsAgentSvrServiceTask load -values commandArgs="register -force -id sas.ops-agentsrv"
/opt/sas/viya/home/bin/ops-config -base config/ops-agent/sas.ops.task/registerOpsAgentServiceTask load -values commandArgs="register -force -id sas.ops-agent"
/opt/sas/viya/home/bin/ops-config -base config/ops-agent/sas.ops.task/registerOpsServiceTask load -values commandArgs="register -force -id sas.ops"
Click the
Hot Fix
tab in this note for a link to instructions about accessing and applying the software update.
Operating System and Release Information
|
SAS System
|
SAS Viya
|
Linux for x64
|
3.5
|
3.5
|
Viya
|
Viya
|
*
For software releases that are not yet generally available, the Fixed
Release is the software release in which the problem is planned to be
fixed.
Viya on Windows: An update for this issue is available for SAS Viya 3.5. For instructions on how to access and apply software updates, see the Updating Your SAS Viya software section in the SAS Viya 3.5 for Windows Deployment Guide at
http://documentation.sas.com/?softwareId=administration&softwareVersion=3.5&softwareContextId=softwareUpdatesWin
Viya on Linux: An update for this issue is available for SAS Viya 3.5. For instructions on how to access and apply software updates, see the Updating Your SAS Viya software section in the SAS Viya 3.5 for Linux Deployment Guide at
http://documentation.sas.com/?softwareId=administration&softwareVersion=3.5&softwareContextId=softwareUpdates
In cases where these internal service IDs no longer can obtain valid OAuth tokens, various components of the SAS® Viya® operations infrastructure return errors.
This content is presented in an iframe, which your browser does not support.
To view the RateIT tab, click
here.
