Various Python utility functions
pip install bx_py_utils
Here only a simple list about existing utilities. Please take a look into the sources and tests for deeper informations.
anonymize()
- Anonymize the given string with special handling for eMail addresses and the possibility to truncate the output.anonymize_dict()
- Returns a new dict with anonymized values for keys containing one of the given keywords.
FnmatchExclude()
- Helper for auto docexclude_func
that exclude files via fnmatch pattern.assert_readme()
- Check and update README file with generate_modules_doc()assert_readme_block()
- Check and update README file: Asset that "text_block" is present between the markers.generate_modules_doc()
- Generate a list of function/class information via pdoc.get_code_location()
- Return start and end line number for an object via inspect.
ClientSideCertManager()
- Helper to manage client-side TLS certificate via AWS Secrets Manager by
SecretsManager()
- Access AWS Secrets Manager values
removeprefix()
- Backport ofremoveprefix
from PEP-616 (Python 3.9+)removesuffix()
- Backport ofremovesuffix
from PEP-616 (Python 3.9+)
dict_get()
- nested dictget()
dict_list2markdown()
- Convert a list of dictionaries into a markdown table.pluck()
- Extract values from a dict, if they are present
Doc-Write, see: https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/doc_write/README.md
OverrideEnviron()
- Context manager to change 'os.environ' temporarily.cgroup_memory_usage()
- Returns the memory usage of the cgroup the Python interpreter is running in.
print_exc_plus()
- Print traceback information with a listing of all the local variables in each frame.
EmptyFileError()
- Will be raised from get_and_assert_file_size() if a 0-bytes file was found.FileError()
- Base error class for all 'file_utils' exceptions.FileHasher()
- Context Manager for generate different hashes from file content while processing a file.FileSizeError()
- File size is not the same as the expected size.NamedTemporaryFile2()
- Generates a temp file with the given filename without any random name sequence.OverlongFilenameError()
- cut_filename() error: The file name can not be shortened, because sterm is to short.TempFileHasher()
- File like context manager that combines NamedTemporaryFile2 and FileHasher.cut_filename()
- Short the file name (and keep the last suffix). Raise OverlongFilenameError if it can't fit.get_and_assert_file_size()
- Check file size of given file object. Raise EmptyFileError for empty files or return sizesafe_filename()
- Makes an arbitrary input suitable to be used as a filename.
filename_matcher()
- Enhance fnmatch that accept a list of patterns.
introspection_query()
- Generate GraphQL introspection query with variable nested depth.
url_safe_encode()
- Encode bytes into a URL safe string.url_safe_hash()
- Generate a URL safe hash withmax_size
from given string/bytes.
ElementsNotFoundError()
- Happens if requested HTML elements cannot be foundInvalidHtml()
- XMLSyntaxError with better error messages: used in validate_html()get_html_elements()
- Returns the selected HTML elements as stringpretty_format_html()
- Pretty format given HTML document via BeautifulSoup (Needs 'beautifulsoup4' package)validate_html()
- Validate a HTML document via XMLParser (Needs 'lxml' package)
pformat()
- Format given object: Try JSON fist and fallback to pformat()
human_timedelta()
- Converts a time duration into a friendly text representation.
import_string()
- Import a dotted module path and return the attribute/class designated by the last name in the path.
chunk_iterable()
- Returns a generator that yields slices of iterable of the givenchunk_size
.
ChangeCurrentWorkDir()
- Context Manager change the "CWD" to an other directory.MockCurrentWorkDir()
- Context Manager to move the "CWD" to a temp directory.assert_is_dir()
- Check if given path is a directoryassert_is_file()
- Check if given path is a file
processify()
- Decorator to run a function as a process.
get_pyproject_config()
- Get a config section from "pyproject.toml". The path can be optional specify.
rison_dumps()
- Encode as RISON, a URL-safe encoding format.
FrameNotFound()
- Base class for lookup errors.last_frame_outside_path()
- Returns the stack frame that is the direct successor of given "file_path".
compare_sentences()
- Calculates the Levenshtein distance between text1 and text2. With filter functionality.ensure_lf()
- Replace line endings to unix-style.get_words()
- Extract words from a text. With filter functionality.is_uuid()
- Returns True if text is a valid UUID (https://www.rfc-editor.org/rfc/rfc9562#name-uuid-format).levenshtein_distance()
- Calculates the Levenshtein distance between two strings.startswith_prefixes()
- >>> startswith_prefixes('foobar', prefixes=('foo','bar'))truncate()
- Truncates the given string to the given lengthuuid_from_text()
- Generate a UUID instance from the given text in a determinism may via SHA224 hash.
assert_equal()
- Check if the two objects are the same. Display a nice diff, usingpformat()
assert_text_equal()
- Check if the two text strings are the same. Display an error message with a diff.pformat_ndiff()
- Generate andiff
from two objects, usingpformat()
pformat_unified_diff()
- Generate a unified diff from two objects, usingpformat()
text_ndiff()
- Generate andiff
between two text strings.text_unified_diff()
- Generate a unified diff between two text strings.
MassContextManager()
- A context manager / decorator that enter/exit a list of mocks.MassContextManagerExceptions()
- Common base class for all non-exit exceptions.
parse_dt()
- Helper for easy generate adatetime
instance via string.
DenyAnyRealRequestContextManager()
- Context manager that denies any request via docket/urllib3. Will raise DenyCallError.deny_any_real_request()
- Deny any request via docket/urllib3. Useful for tests, because they should mock all requests.
FileWatcher()
- Helper to record which new files have been created.
NoLogs()
- Context manager to Suppress all logger outputsRaiseLogUsage()
- A log handler, that raise an error on every log output.
SecretsManagerMock()
- Mock forbx_py_utils.aws.secret_manager.SecretsManager()
MockedBoto3Session()
- Mock forboto3.session.Session()
MockUUIDGenerator()
- Helper to mockuuid.uuid4()
with reproducible results (e.g. for snapshot tests)
A simple mock for Boto3's S3 modules.
PseudoS3Client()
- Simulates a boto3 S3 client object in tests
RedirectOut()
- Redirect stdout + stderr into a buffer (with optional strip the output)
assert_json_requests_mock()
- Check the requests mock history. In this case all requests must be JSON.assert_json_requests_mock_snapshot()
- Check requests mock history via snapshot. Accepts only JSON requests.assert_requests_mock()
- Check the requests mock history. Accept mixed "text" and "JSON".assert_requests_mock_snapshot()
- Check requests mock history via snapshot. Accept mixed "text" and "JSON".
Assert complex output via auto updated snapshot files with nice diff error messages.
SnapshotChanged()
- Assertion failed.assert_binary_snapshot()
- Assert binary data via snapshot fileassert_html_snapshot()
- Assert "html" string via snapshot file with validate and pretty formatassert_py_snapshot()
- Assert complex python objects vio PrettyPrinter() snapshot file.assert_snapshot()
- Assert given data serialized to JSON snapshot file.assert_text_snapshot()
- Assert "text" string via snapshot fileget_snapshot_file()
- Generate a file path use stack information to fill not provided path components.
MockTimeMonotonicGenerator()
- Helper to mocktime.monotonic()
in tests.
BaseDocTests()
- Helper to include all doctests in unittests, without change unittest setup. Just add a normal TestCase.assert_no_flat_tests_functions()
- Check if there exists normal test functions (That will not be executed by normal unittests)
FreezeXlsxTimes()
- Context manager / decorator intended to freeze timestamps of xlsx files creation by e.g.: openpyxl.generate_xlsx_md_snapshot()
- Generate a markdown snapshot of a XLSX: Display ZIP info + Sheets content as Markdown.xlsx2dict()
- Convert a XLSX file content into a dictionary: Every sheet is a key, and the value is a list of dictionaries.xlsx2markdown()
- Convert all Sheets of a XLSX into markdown tables.
FreezeZipFileDatetime()
- Context manager / decorator to freezes the modification time of files written to a zip file.zip_info()
- Generates similar information thanunzip -v
: Yields ZipFileInfo for each file in the zip file.zip_info_markdown()
- Generates a markdown representation of the zip file content. Similar tounzip -v
output.
cutout()
- Mark a point in a long text by line no + column with context lines around.
Quick hint about snapshot. If you have many snapshots in your project and you need to change many with a code change, then you can run the tests without a snapshot change leading to an error, by set RAISE_SNAPSHOT_ERRORS=0
in your environment.
e.g.:
RAISE_SNAPSHOT_ERRORS=0 python3 -m unittest
Renew all snapshot files with:
make update-test-snapshot-files
We split bx_py_utils
and moved all Django related utilities into the separated project:
So, bx_py_utils
is better usable in non-Django projects, because Django will not installed as decency of "bx_py_utils"
To start developing, just run make install
to create a .venv
and install all needed packages.
The minimal requirements are only python3-venv
and python3-pip
(uv
will be installed via pip
in .venv
)
e.g.:
~$ git clone https://github.com/boxine/bx_py_utils.git
~$ cd bx_py_utils
~/bx_py_utils$ make
help List all commands
install-base-req Install needed base packages via apt
install Install the project in a Python virtualenv
update-requirements Update requirements
lint Run code formatters and linter
fix-code-style Fix code formatting
tox-listenvs List all tox test environments
tox Run tests via tox with all environments
test Run tests
coverage Run tests with coverage
update-test-snapshot-files Update all snapshot files (by remove and recreate all snapshot files)
mypy Run mypy
pip-audit Run https://github.com/pypa/pip-audit
publish Release new version to PyPi
clean Remove created files from the test project
- Increase verion number in
bx_py_utils/__init__.py
- Create pull request
- After merge, call:
make publish
MIT. Patches welcome!
We’ve been rethinking the listening experience for kids and have created an ecosystem where haptic and listening experience are combined via smart technology - the Toniebox.
We are constantly looking for engineers to join our team in different areas. If you’d be interested in contributing to our platform, have a look at: https://tonies.com/jobs/