SALT-MASTER PATCHED state.sls causes UnicodeDecodeError: 'utf8' codec can't decode byte 0xfc (this i

link管理
链接快照平台
输入网页链接，自动生成快照
标签化管理网页链接
Description of Issue

A comment causes an exception but is not shown at all.
I would be extremely grateful if Salt would have an extremely robust way to display text, at least in the exception. E.g. delete all non-ASCII characters.
Character ü is not-ASCII but must not be a problem for Salt.
Setup

The comment is not reported.
I cannot tell which state in the sls causes the comment
Steps to Reproduce Issue

salt MINION state.sls syncP2P
[ERROR   ] An un-handled exception was caught by salt's global exception handler:
UnicodeDecodeError: 'utf8' codec can't decode byte 0xfc in position 642: invalid start byte
Traceback (most recent call last):
  File "/bin/salt", line 10, in <module>
    salt_main()
  File "/usr/lib/python2.7/site-packages/salt/scripts.py", line 495, in salt_main
    client.run()
  File "/usr/lib/python2.7/site-packages/salt/cli/salt.py", line 197, in run
    self._output_ret(ret_, out, retcode=retcode)
  File "/usr/lib/python2.7/site-packages/salt/cli/salt.py", line 376, in _output_ret
    _retcode=retcode)
  File "/usr/lib/python2.7/site-packages/salt/output/__init__.py", line 96, in display_output
    display_data = try_printout(data, out, opts, **kwargs)
  File "/usr/lib/python2.7/site-packages/salt/output/__init__.py", line 43, in try_printout
    printout = get_printout(out, opts)(data, **kwargs)
  File "/usr/lib/python2.7/site-packages/salt/output/highstate.py", line 157, in output
    for host, hostdata in six.iteritems(data)
  File "/usr/lib/python2.7/site-packages/salt/output/highstate.py", line 335, in _format_host
    ret['comment'] = salt.utils.stringutils.to_unicode(ret['comment'])
  File "/usr/lib/python2.7/site-packages/salt/utils/stringutils.py", line 160, in to_unicode
    raise exc  # pylint: disable=raising-bad-type
UnicodeDecodeError: 'utf8' codec can't decode byte 0xfc in position 642: invalid start byte
Versions Report
Salt Version:
           Salt: 2018.3.4
Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: 1.2.5
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.5 (default, Jun 20 2019, 20:27:34)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.3.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4
System Versions:
           dist: centos 7.6.1810 Core
         locale: UTF-8
        machine: x86_64
        release: 3.10.0-957.5.1.el7.x86_64
         system: Linux
        version: CentOS Linux 7.6.1810 Core
Minion 2016.11.3
          I now use a workaround by patching the salt master
diff /usr/lib/python2.7/site-packages/salt/output/highstate.py /usr/lib/python2.7/site-packages/salt/output/highstate.py.markus.bak
335,336c335
<                 text = ret['comment']
<                 ret['comment'] = ''.join([i if ord(i) < 128 else '_' for i in text])
>                 ret['comment'] = salt.utils.stringutils.to_unicode(ret['comment'])
On the eventbus I found
        "service_|-magnicopy_|-magnicopy_|-running": {
            "__id__": "magnicopy",
            "__run_num__": 5,
            "changes": {},
            "comment": "An exception occurred in this state: Traceback (most recent call last):\n  File \"C:\\
salt\\bin\\lib\\site-packages\\salt\\state.py\", line 1746, in call\n    **cdata['kwargs'])\n  File \"C:\\salt
\\bin\\lib\\site-packages\\salt\\loader.py\", line 1703, in wrapper\n    return f(*args, **kwargs)\n  File \"C
:\\salt\\bin\\lib\\site-packages\\salt\\states\\service.py\", line 367, in running\n    func_ret = __salt__['s
ervice.start'](name)\n  File \"C:\\salt\\bin\\lib\\site-packages\\salt\\modules\\win_service.py\", line 405, i
n start\n    'Failed To Start {0}: {1}'.format(name, exc[2]))\nCommandExecutionError: Failed To Start dhlcopy:
 Es wird bereits eine Instanz des Dienstes ausgef<FC>hrt.\n",
            "duration": 15.0,
            "name": "dhlcopy",
            "result": false,
            "start_time": "11:43:23.700000"
From the eventbus I extract the comment
An exception occurred in this state: Traceback (most recent call last):\n  File \"C:\\salt\\bin\\lib\\site-packages\\salt\\state.py\", line 1746, in call\n    **cdata['kwargs'])\n  File \"C:\\salt\\bin\\lib\\site-packages\\salt\\loader.py\", line 1703, in wrapper\n    return f(*args, **kwargs)\n  File \"C:\\salt\\bin\\lib\\site-packages\\salt\\states\\service.py\", line 367, in running\n    func_ret = __salt__['service.start'](name)\n  File \"C:\\salt\\bin\\lib\\site-packages\\salt\\modules\\win_service.py\", line 405, in start\n    'Failed To Start {0}: {1}'.format(name, exc[2]))\nCommandExecutionError: Failed To Start magnicopy: Es wird bereits eine Instanz des Dienstes ausgef<FC>hrt.\n
The character at position 690 in the comment should be ü (U Umlaut)

I assume the difference of 690 and 642  can be explained by counting non visible space
(From the Salt Error) Hex 0xfc is decimal 252

In ISOLATIN-1 character at position 252 is ü
This leads to the state
magnicopy:
  service.running:
    - enable: True
    - fire_event: True
CommandExecutionError: Failed To Start magnicopy: Es wird bereits eine Instanz des Dienstes ausgefhrt.
means "the service is already running".
Please fix or verify in recent version:
A sls state file should not break if a comment contains "foreign languages"
service.running should handle "the service is already running" in German.
      changed the title
state.sls causes UnicodeDecodeError: 'utf8' codec can't decode byte 0xfc
SALT-MASTER PATCHED state.sls causes UnicodeDecodeError: 'utf8' codec can't decode byte 0xfc
    Sep 27, 2019
      changed the title
SALT-MASTER PATCHED state.sls causes UnicodeDecodeError: 'utf8' codec can't decode byte 0xfc
SALT-MASTER PATCHED state.sls causes UnicodeDecodeError: 'utf8' codec can't decode byte 0xfc (this is ü)
    Sep 27, 2019
          I found this exception also in /var/log/salt/master

The problem starts with Jinja in an reactor .
The same function to_unicode() as above is involved:
File "/usr/lib/python2.7/site-packages/salt/utils/stringutils.py", line 160, in to_unicode
The problem seems to me that Jinja or Salt cannot (always) handle string that contain the letter ü.
I must assume that the Root cause is again the operating system response
Es wird bereits eine Instanz des Dienstes ausgeführt
Using data | json() may make my situation worse: I even do not know what is the problematic string is because SaltRenderError does not output the offending string to /var/log/salt/master
I  would like to propose that the SaltRenderer is able to
handle unexpected Windows codepages,
more than 7-bit ASCII characters and
ultimately outputs the offending string in a "best effort layout" for debugging.
  runner.syncP2Pminimal_runner.main:
    - arg:
      - {{ data | json() }}      # Escape potential YAML-inconformant response fragments in Json, e.g. colons.    <======================
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/utils/templates.py", line 393, in render_jinja_tmpl
    output = template.render(**decoded_context)
  File "/usr/lib/python2.7/site-packages/jinja2/environment.py", line 969, in render
[...]
2019-10-19 05:23:27,185 [salt.utils.reactor:101 ][ERROR   ][23268] Failed to render "/srv/salt/reactor/syncP2Pminimal_reactor.sls":
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/utils/reactor.py", line 92, in render_reaction
    data=data)
  File "/usr/lib/python2.7/site-packages/salt/state.py", line 384, in render_template
    **kwargs)
  File "/usr/lib/python2.7/site-packages/salt/template.py", line 93, in compile_template
    ret = render(input_data, saltenv, sls, **render_kwargs)
  File "/usr/lib/python2.7/site-packages/salt/renderers/jinja.py", line 70, in render
    **kws)
  File "/usr/lib/python2.7/site-packages/salt/utils/templates.py", line 170, in render_tmpl
    output = render_str(tmplstr, context, tmplpath)
  File "/usr/lib/python2.7/site-packages/salt/utils/templates.py", line 442, in render_jinja_tmpl
    trace=tracestr)
SaltRenderError: Jinja error: 'utf8' codec can't decode byte 0xfc in position 856: invalid start byte
SaltRenderError: Jinja error: 'utf8' codec can't decode byte 0xfc in position 856: invalid start byte
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/utils/templates.py", line 393, in render_jinja_tmpl
    output = template.render(**decoded_context)
  File "/usr/lib/python2.7/site-packages/jinja2/environment.py", line 969, in render
    return self.environment.handle_exception(exc_info, True)
  File "/usr/lib/python2.7/site-packages/jinja2/environment.py", line 742, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "<template>", line 4, in top-level template code
  File "/usr/lib/python2.7/site-packages/salt/utils/jinja.py", line 824, in format_json
    return Markup(salt.utils.stringutils.to_unicode(json_txt))
  File "/usr/lib/python2.7/site-packages/salt/utils/stringutils.py", line 160, in to_unicode
    raise exc  # pylint: disable=raising-bad-type
UnicodeDecodeError: 'utf8' codec can't decode byte 0xfc in position 856: invalid start byte
issue cmd.run 'chcp & chcp 65001 & dir'  twice or more:
Active code page: 437

Active code page: 65001

Volume in drive C is Über
Even if the codepage is reported as 437 initially, setting it to UTF-8 has an influence on the next run.
I would guess that one has to set the codepage "deeper down" in Python.
I start to suspect that Salt relies on codepage 65001, a.k.a. UTF-8.
@twangboy does Salt set the Windows codepage?
          Entry in /var/log/salt/master
2019-10-28 09:05:02,375 [salt.utils.templates:181 ][ERROR   ][4548] Rendering exception occurred
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/utils/templates.py", line 170, in render_tmpl
    output = render_str(tmplstr, context, tmplpath)
  File "/usr/lib/python2.7/site-packages/salt/utils/templates.py", line 442, in render_jinja_tmpl
    trace=tracestr)
SaltRenderError: Jinja error: 'utf8' codec can't decode byte 0xfc in position 833: invalid start byte
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/utils/templates.py", line 393, in render_jinja_tmpl
    output = template.render(**decoded_context)
  File "/usr/lib/python2.7/site-packages/jinja2/environment.py", line 969, in render
    return self.environment.handle_exception(exc_info, True)
  File "/usr/lib/python2.7/site-packages/jinja2/environment.py", line 742, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "<template>", line 4, in top-level template code
  File "/usr/lib/python2.7/site-packages/salt/utils/jinja.py", line 824, in format_json
    return Markup(salt.utils.stringutils.to_unicode(json_txt))
  File "/usr/lib/python2.7/site-packages/salt/utils/stringutils.py", line 160, in to_unicode
    raise exc  # pylint: disable=raising-bad-type
UnicodeDecodeError: 'utf8' codec can't decode byte 0xfc in position 833: invalid start byte
; line 4
  runner.syncP2P_runner.main:
    - arg:
      - {{ data | json() }}      # Escape potential YAML-inconformant response fragments in Json, e.g. colons.    <======================
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/utils/templates.py", line 393, in render_jinja_tmpl
    output = template.render(**decoded_context)
  File "/usr/lib/python2.7/site-packages/jinja2/environment.py", line 969, in render
[...]
cmd.run 'dir'
Datentr\x84ger in Laufwerk C: ist Windows
Datenträger in Laufwerk C: ist Windows
UnicodeDecodeError: 'ascii' codec can't decode byte 0x84 in position 8: ordinal not in range(128)
cmd.run 'whoami'
nt-autorit\x84t\\system
nt-autorität\\system
Traceback (most recent call last):
  File "/bin/salt", line 10, in <module>
    salt_main()
  File "/usr/lib/python2.7/site-packages/salt/scripts.py", line 495, in salt_main
    client.run()
  File "/usr/lib/python2.7/site-packages/salt/cli/salt.py", line 206, in run
    self._print_returns_summary(ret)
  File "/usr/lib/python2.7/site-packages/salt/cli/salt.py", line 316, in _print_returns_summary
    and minion_ret.startswith("Minion did not return")
UnicodeDecodeError: 'ascii' codec can't decode byte 0x84 in position 10: ordinal not in range(128)
enable_developer_mode:
  cmd.run:
    - name: 'dism /online /Add-Capability /CapabilityName:Tools.DeveloperMode.Core~~~~0.0.1.0'
UnicodeDecodeError: 'ascii' codec can't decode byte 0x81 in position 333: ordinal not in range(128)
Traceback (most recent call last):
  File "/bin/salt", line 10, in <module>
    salt_main()
  File "/usr/lib/python2.7/site-packages/salt/scripts.py", line 495, in salt_main
    client.run()
  File "/usr/lib/python2.7/site-packages/salt/cli/salt.py", line 197, in run
    self._output_ret(ret_, out, retcode=retcode)
  File "/usr/lib/python2.7/site-packages/salt/cli/salt.py", line 376, in _output_ret
    _retcode=retcode)
  File "/usr/lib/python2.7/site-packages/salt/output/__init__.py", line 96, in display_output
    display_data = try_printout(data, out, opts, **kwargs)
  File "/usr/lib/python2.7/site-packages/salt/output/__init__.py", line 43, in try_printout
    printout = get_printout(out, opts)(data, **kwargs)
  File "/usr/lib/python2.7/site-packages/salt/output/highstate.py", line 157, in output
    for host, hostdata in six.iteritems(data)
  File "/usr/lib/python2.7/site-packages/salt/output/highstate.py", line 236, in _format_host
    schanged, ctext = _format_changes(ret['changes'], orchestration)
  File "/usr/lib/python2.7/site-packages/salt/output/highstate.py", line 527, in _format_changes
    ctext = _nested_changes(changes)
  File "/usr/lib/python2.7/site-packages/salt/output/highstate.py", line 500, in _nested_changes
    nested_indent=14)
Where is the ASCII codec?
     Verzeichnis von C:\windows\system32\config\systemprofile
    24.06.2020  11:28            19.090 markus2.txt
                   1 Datei(en),         19.090 Bytes
                   0 Verzeichnis(se), 329.521.315.840 Bytes frei
[ERROR   ] An un-handled exception was caught by salt's global exception handler:
UnicodeDecodeError: 'ascii' codec can't decode byte 0x84 in position 8: ordinal not in range(128)
Traceback (most recent call last):
  File "/bin/salt", line 10, in <module>
    salt_main()
  File "/usr/lib/python2.7/site-packages/salt/scripts.py", line 516, in salt_main
    client.run()
  File "/usr/lib/python2.7/site-packages/salt/cli/salt.py", line 207, in run
    self._print_returns_summary(ret)
  File "/usr/lib/python2.7/site-packages/salt/cli/salt.py", line 317, in _print_returns_summary
    and minion_ret.startswith("Minion did not return")
UnicodeDecodeError: 'ascii' codec can't decode byte 0x84 in position 8: ordinal not in range(128)
Traceback (most recent call last):
  File "/bin/salt", line 10, in <module>
    salt_main()
  File "/usr/lib/python2.7/site-packages/salt/scripts.py", line 516, in salt_main
    client.run()
  File "/usr/lib/python2.7/site-packages/salt/cli/salt.py", line 207, in run
    self._print_returns_summary(ret)
  File "/usr/lib/python2.7/site-packages/salt/cli/salt.py", line 317, in _print_returns_summary
    and minion_ret.startswith("Minion did not return")
UnicodeDecodeError: 'ascii' codec can't decode byte 0x84 in position 8: ordinal not in range(128)
PATCH
root# diff  /root/.emacs.d/backups/\!usr\!lib\!python2.7\!site-packages\!salt\!utils\!stringutils.py.~1~ /usr/lib/python2.7/site-packages/salt/utils/stringutils.py
121c121
< def to_unicode(s, encoding=None, errors='strict', normalize=False):
> def to_unicode(s, encoding='latin1', errors='strict', normalize=False):
Version
root# salt-master -V

/usr/lib/python2.7/site-packages/salt/scripts.py:102: DeprecationWarning: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date.  Salt will drop support for Python 2.7 in the Sodium release or later.

Salt Version:

Salt: 2019.2.5
Dependency Versions:

cffi: 1.6.0

cherrypy: Not Installed

dateutil: Not Installed

docker-py: Not Installed

gitdb: Not Installed

gitpython: Not Installed

ioflo: Not Installed

Jinja2: 2.7.2

libgit2: Not Installed

libnacl: Not Installed

M2Crypto: Not Installed

Mako: Not Installed

msgpack-pure: Not Installed

msgpack-python: 0.5.6

mysql-python: 1.2.5

pycparser: 2.14

pycrypto: 2.6.1

pycryptodome: Not Installed

pygit2: Not Installed

Python: 2.7.5 (default, Apr  2 2020, 13:16:51)

python-gnupg: Not Installed

PyYAML: 3.11

PyZMQ: 15.3.0

RAET: Not Installed

smmap: Not Installed

timelib: Not Installed

Tornado: 4.2.1

ZMQ: 4.1.4
System Versions:

dist: centos 7.8.2003 Core

locale: UTF-8

machine: x86_64

release: 3.10.0-957.5.1.el7.x86_64

system: Linux

version: CentOS Linux 7.8.2003 Core
    severity-medium
  3rd level, incorrect or bad functionality, confusing and lacks a work around
    State-Module