The presentation from Python meetup by JettyCloud about solving a problem found in a library that uses hashlib, followed by an overview of the CPython hashlib module with implementation details.
2. $ whoami
Software engineer
Python Software Foundation (PSF) contributing member
Python meetups member & speaker
Co-organized Python meetups/drinkups
Played CTF with SiBears
2
6. Authentication with remote machine "SERVER" for user "user" will be using NTLM v1 authentication (with extended
security)
Now switching over to SMB2 protocol communication
SMB2 dialect negotiation successful
Performing NTLMv2 authentication (on SMB2) with server challenge "b'da16c5c3e38ca2df'"
Performing NTLMv1 authentication (on SMB2) with server challenge "b'da16c5c3e38ca2df'"
Server supports SMB signing
SMB signing deactivated. SMB messages will NOT be signed.
Authentication (on SMB2) failed. Please check username and password.
Traceback (most recent call last):
File "/home/user/proj/smb_module.py", line 43, in <module>
downloader.list_dir()
File "/home/user/proj/
smb_module.py", line 36, in list_dir
all_builds = self._connection.listPath(self.service_name, path)
File "/home/user/proj/venv/lib/python3.9/site-packages/smb/SMBConnection.py", line 201, in listPath
self._listPath(service_name, path, cb, eb, search = search, pattern = pattern, timeout = timeout)
File "/home/user/proj/venv/lib/python3.9/site-packages/smb/base.py", line 601, in _listPath_SMB2
raise NotReadyError('SMB connection not authenticated')
smb.base.NotReadyError: SMB connection not authenticated
Problem
6
17. Run in verbose mode, to analyse imported modules
$ python -v smb_module.py # on host where OK
...
import 'nmb.utils' # <_frozen_importlib_external.SourceFileLoader object at 0x7f3199c4b370>
import 'nmb.base' # <_frozen_importlib_external.SourceFileLoader object at 0x7f3199c42ca0>
# /usr/local/lib/python3.9/site-packages/smb/__pycache__/ntlm.cpython-39.pyc matches
/usr/local/lib/python3.9/site-packages/smb/ntlm.py
# code object from '/usr/local/lib/python3.9/site-packages/smb/__pycache__/ntlm.cpython-39.pyc'
# /usr/local/lib/python3.9/site-packages/smb/utils/__pycache__/rc4.cpython-39.pyc matches
/usr/local/lib/python3.9/site-packages/smb/utils/rc4.py
# code object from '/usr/local/lib/python3.9/site-packages/smb/utils/__pycache__/rc4.cpython-39.pyc'
import 'smb.utils.rc4' # <_frozen_importlib_external.SourceFileLoader object at 0x7f319bdd2a00>
# /usr/local/lib/python3.9/site-packages/smb/utils/__pycache__/pyDes.cpython-39.pyc matches
/usr/local/lib/python3.9/site-packages/smb/utils/pyDes.py
# code object from '/usr/local/lib/python3.9/site-packages/smb/utils/__pycache__/pyDes.cpython-39.pyc'
import 'smb.utils.pyDes' # <_frozen_importlib_external.SourceFileLoader object at 0x7f319bdd2970>
import 'smb.ntlm' # <_frozen_importlib_external.SourceFileLoader object at 0x7f3199c4b220>
...
17
18. Run in verbose mode, to analyse imported modules
$ python -v smb_module.py # on host where FAILS
...
import 'nmb.utils' # <_frozen_importlib_external.SourceFileLoader object at 0x7f3b9ea1a700>
import 'nmb.base' # <_frozen_importlib_external.SourceFileLoader object at 0x7f3b9ea0ca90>
# /home/user/proj/venv/lib/python3.9/site-packages/smb/__pycache__/ntlm.cpython-39.pyc matches
/home/user/proj/venv/lib/python3.9/site-packages/smb/ntlm.py
# code object from '/home/user/proj/venv/lib/python3.9/site-packages/smb/__pycache__/ntlm.cpython-39.pyc'
# /home/user/proj/venv/lib/python3.9/site-packages/smb/utils/__pycache__/rc4.cpython-39.pyc matches
/home/user/proj/venv/lib/python3.9/site-packages/smb/utils/rc4.py
# code object from '/home/user/proj/venv/lib/python3.9/site-packages/smb/utils/__pycache__/rc4.cpython-39.pyc'
import 'smb.utils.rc4' # <_frozen_importlib_external.SourceFileLoader object at 0x7f3b9ea03640>
# /home/user/proj/venv/lib/python3.9/site-packages/smb/utils/__pycache__/pyDes.cpython-39.pyc matches
/home/user/proj/venv/lib/python3.9/site-packages/smb/utils/pyDes.py
# code object from '/home/user/proj/venv/lib/python3.9/site-packages/smb/utils/__pycache__/pyDes.cpython-39.pyc'
import 'smb.utils.pyDes' # <_frozen_importlib_external.SourceFileLoader object at 0x7f3b9ea03580>
# /home/user/proj/venv/lib/python3.9/site-packages/smb/utils/__pycache__/md4.cpython-39.pyc matches
/home/user/proj/venv/lib/python3.9/site-packages/smb/utils/md4.py
# code object from '/home/user/proj/venv/lib/python3.9/site-packages/smb/utils/__pycache__/md4.cpython-39.pyc'
# /home/user/proj/venv/lib/python3.9/site-packages/smb/utils/__pycache__/U32.cpython-39.pyc matches
/home/user/proj/venv/lib/python3.9/site-packages/smb/utils/U32.py
# code object from '/home/user/proj/venv/lib/python3.9/site-packages/smb/utils/__pycache__/U32.cpython-39.pyc'
import 'smb.utils.U32' # <_frozen_importlib_external.SourceFileLoader object at 0x7f3b9e572a00>
import 'smb.utils.md4' # <_frozen_importlib_external.SourceFileLoader object at 0x7f3b9e572550>
import 'smb.ntlm' # <_frozen_importlib_external.SourceFileLoader object at 0x7f3b9ea03e80>
...
18
19. Where is it imported from?
Source: [2]
19
try:
import hashlib
hashlib.new('md4')
def MD4(): return hashlib.new('md4')
except ( ImportError, ValueError ):
from .utils.md4 import MD4
20. Try to use hash type – OK
(venv) $ python # on host where OK
Python 3.9.15 (main, Oct 12 2022, 19:14:37)
[GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import hashlib
>>> hashlib.new( 'md4')
<md4 _hashlib.HASH object @ 0x7ffbedfbbc70>
20
21. Try to use hash type – FAILS
21
(venv) $ python # on host where FAILS
Python 3.9.15 (main, Oct 12 2022, 19:14:37)
[GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import hashlib
>>> hashlib.new( 'md4')
Traceback (most recent call last):
File "/usr/lib/python3.9/hashlib.py", line 160, in __hash_new
return _hashlib.new(name, data, **kwargs)
ValueError: [digital envelope routines] unsupported
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.9/hashlib.py", line 166, in __hash_new
return __get_builtin_constructor(name)(data)
File "/usr/lib/python3.9/hashlib.py", line 123, in __get_builtin_constructor
raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type md4
22. pysmb-provided MD4 implementation for Python3 is broken
https:/
/github.com/miketeo/pysmb/issues/196
pysmb 1.2.8 fixes this issue
Bug
22
Source: [3]
26. Try to use hash type after enable legacy providers
(venv) $ python # on host where FAILED
Python 3.9.15 (main, Oct 12 2022, 19:14:37)
[GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import hashlib
>>> hashlib.new( 'md4')
<md4 _hashlib.HASH object @ 0x7fe885fade10>
26
27. Authentication with remote machine "SERVER" for user "user" will be using NTLM v1 authentication (with extended
security)
Now switching over to SMB2 protocol communication
SMB2 dialect negotiation successful
Performing NTLMv2 authentication (on SMB2) with server challenge "b'137550e8bf56b78d'"
Performing NTLMv1 authentication (on SMB2) with server challenge "b'137550e8bf56b78d'"
Server supports SMB signing
SMB signing deactivated. SMB messages will NOT be signed.
Authentication (on SMB2) successful!
...
Problem solved
27
28. How hashlib is related to OpenSSL
28
hashlib.py _hashlib.so libcrypto.so
python3.9
$ objdump -T python3.9 | egrep "blake|md4|md5|sha1|sha3|sha256|sha512"
003a90df g DF .text 0000008c Base PyInit__sha512
003a5998 g DF .text 00000058 Base PyInit__sha1
003a5e9d g DF .text 000001da Base PyInit__sha3
00304aca g DF .text 00000058 Base PyInit__md5
00388b47 g DF .text 000002fb Base PyInit__blake2
003a5b7f g DF .text 0000008c Base PyInit__sha256
$ objdump -T libcrypto.so
...
... g DF .text ...0c OPENSSL_1_1_1 EVP_sha3_384
... g DF .text ...0c OPENSSL_1_1_0 EVP_md4
... g DF .text ...0c OPENSSL_1_1_0 EVP_md5
... g DF .text ...0c OPENSSL_1_1_0
EVP_blake2s256
... g DF .text ...0c OPENSSL_1_1_0 EVP_sha1
... g DF .text ...0c OPENSSL_1_1_0 EVP_md5_sha1
...
$ objdump -T _hashlib.cpython-39-x86_64-linux-gnu.so
...
00000000 DF *UND* 00000000 OPENSSL_1_1_0
EVP_blake2s256
00000000 DF *UND* 00000000 OPENSSL_1_1_0 EVP_sha384
00000000 DF *UND* 00000000 OPENSSL_1_1_1 EVP_sha3_512
00000000 DF *UND* 00000000 OPENSSL_1_1_1 EVP_shake128
00000000 DF *UND* 00000000 OPENSSL_1_1_0 EVP_md5
00000000 DF *UND* 00000000 OPENSSL_1_1_0 EVP_sha1
...
OpenSSL consists of two libraries: libcrypto and libssl
29. Changes between 1.1.1 and 3.0.0 [7 sep 2021]
...
* The implementation of the EVP digests MD2, MD4, MDC2, WHIRLPOOL and
RIPEMD-160 have been moved to the legacy provider.
…
* The low-level MD2, MD4, MD5, MDC2, RIPEMD160 and Whirlpool digest
functions have been deprecated.
…
MD4 is deprecated and disabled by default in OpenSSL 3.0.0
29
Source: [6]
41. try:
import _hashlib
new = __hash_new
__get_hash = __get_openssl_constructor
algorithms_available = algorithms_available.union(
_hashlib.openssl_md_meth_names)
except ImportError :
new = __py_new
__get_hash = __get_builtin_constructor
hashlib.py – new
Source: [12]
41
__get_builtin_constructor
42. def __hash_new(name, data= b'', **kwargs):
"""new(name, data=b'') - Return a new hashing object using the named algorithm;
optionally initialized with data (which must be a bytes-like object).
"""
if name in __block_openssl_constructor:
# Prefer our builtin blake2 implementation.
return __get_builtin_constructor(name)(data, **kwargs)
try:
return _hashlib.new(name, data, **kwargs)
except ValueError:
# If the _hashlib module (OpenSSL) doesn't support the named
# hash, try using our builtin implementations.
# This allows for SHA224/256 and SHA384/512 support even though
# the OpenSSL library prior to 0.9.8 doesn't provide them.
return __get_builtin_constructor(name)(data)
hashlib.py – __hash_new
Source: [13]
'blake2b', 'blake2s'
42
43. def __get_builtin_constructor(name):
cache = __builtin_constructor_cache
constructor = cache.get(name)
if constructor is not None:
return constructor
try:
if name in {'SHA1', 'sha1'}:
import _sha1
cache['SHA1'] = cache['sha1'] = _sha1.sha1
elif name in {'MD5', 'md5'}:
import _md5
cache['MD5'] = cache['md5'] = _md5.md5
...
elif name in {'blake2b', 'blake2s'}:
import _blake2
cache['blake2b'] = _blake2.blake2b
cache['blake2s'] = _blake2.blake2s
...
except ImportError:
pass # no extension module, this hash is unsupported.
constructor = cache.get(name)
if constructor is not None:
return constructor
raise ValueError('unsupported hash type ' + name)
hashlib.py – __get_builtin_constructor
Source: [14]
43
hashlib.py
python3.9
$ objdump -T python3.9 | egrep "blake|md4|md5|sha1|sha3|sha256|sha512"
003a90df g DF .text 0000008c Base PyInit__sha512
003a5998 g DF .text 00000058 Base PyInit__sha1
003a5e9d g DF .text 000001da Base PyInit__sha3
00304aca g DF .text 00000058 Base PyInit__md5
00388b47 g DF .text 000002fb Base PyInit__blake2
003a5b7f g DF .text 0000008c Base PyInit__sha256
44. try:
import _hashlib
new = __hash_new
__get_hash = __get_openssl_constructor
algorithms_available = algorithms_available.union(
_hashlib.openssl_md_meth_names)
except ImportError :
new = __py_new
__get_hash = __get_builtin_constructor
hashlib.py – __get_hash
Source: [15]
44
to prepare named constructors
45. def __get_openssl_constructor (name):
if name in __block_openssl_constructor:
# Prefer our builtin blake2 implementation.
return __get_builtin_constructor(name)
try:
# MD5, SHA1, and SHA2 are in all supported OpenSSL versions
# SHA3/shake are available in OpenSSL 1.1.1+
f = getattr(_hashlib, 'openssl_' + name)
# Allow the C module to raise ValueError. The function will be
# defined but the hash not actually available. Don't fall back to
# builtin if the current security policy blocks a digest, bpo#40695.
f(usedforsecurity =False)
# Use the C function directly (very fast)
return f
except (AttributeError , ValueError):
return __get_builtin_constructor(name)
hashlib.py – __get_openssl_constructor
Source: [16]
'blake2b', 'blake2s'
45
46. for __func_name in __always_supported:
# try them all, some may not work due to the OpenSSL
# version not supporting that algorithm.
try:
globals()[__func_name] = __get_hash(__func_name)
except ValueError:
import logging
logging.exception( 'code for hash %s was not found.' , __func_name)
hashlib.py – prepare all named constructors for algorithms_guaranteed
>>> hashlib. __dict__
{..., 'md5': <built-in function openssl_md5>, 'sha1': <built-in function openssl_sha1>,
'sha224': <built-in function openssl_sha224>, 'sha256': <built-in function openssl_sha256>,
'sha384': <built-in function openssl_sha384>, 'sha512': <built-in function openssl_sha512>,
'blake2b': <class '_blake2.blake2b'>, 'blake2s': <class '_blake2.blake2s'>, 'sha3_224': <built-in
function openssl_sha3_224>, 'sha3_256': <built-in function openssl_sha3_256>, 'sha3_384':
<built-in function openssl_sha3_384>, 'sha3_512': <built-in function openssl_sha3_512>,
'shake_128' : <built-in function openssl_shake_128>, 'shake_256' : <built-in function
openssl_shake_256>}
Source: [17]
__get_openssl_constructor
or
__get_builtin_constructor
46
50. Sources: [18], [19]
PEP 644 – Require OpenSSL 1.1.1 or newer
Version 3.0 will be supported until 2026-09-07 (LTS).
Version 1.1.1 will be supported until 2023-09-11 (LTS).
Version 1.0.2 is no longer supported. Extended support for 1.0.2 to gain access to security fixes
for that version is available.
Versions 1.1.0, 1.0.1, 1.0.0 and 0.9.8 are no longer supported.
50
OpenSSL 1.1.1 -> TLS 1.3 version with faster handshake and more secure.
OpenSSL 1.1.0c+ -> Fully fork and thread safe.
OpenSSL 1.1.0+ ships with SHA-3 and SHAKE. Python could rely on OpenSSL’s libcrypto.
Benefits:
51. Summary
51
Python has a set of guaranteed hash algorithms.
Some extra algorithms Python loads from OpenSSL’s libcrypto.
The available OpenSSL algorithms may differ depending on the versions.
Remember about “python -v” verbose mode for analysis.