#249 All of Linux as a Python API

37:12
 
Share
 

Manage episode 301862812 series 1305988
By Michael Kennedy and Brian Okken. Discovered by Player FM and our community — copyright is owned by the publisher, not Player FM, and audio is streamed directly from their servers. Hit the Subscribe button to track updates in Player FM, or paste the feed URL into other podcast apps.

Watch the live stream:

Watch on YouTube
About the show

Sponsored by us:

Special guest: Erik Christiansen

Michael #1: Fickling

  • via Oli
  • A Python pickling decompiler and static analyzer
  • Pickled ML models are becoming the data exchange and workflow of ML
  • Analyses pickle files for security risks - It can also remove or insert [malicious] code into pickle files...
  • Created by a security firm, it can be a useful defensive or offensive tool.
  • Perhaps it is time to screen all pickles?
 >>> import ast >>> import pickle >>> from fickling.pickle import Pickled >>> print(ast.dump(Pickled.load(pickle.dumps([1, 2, 3, 4])).ast, indent=4)) Module( body=[ Assign( targets=[ Name(id='result', ctx=Store())], value=List( elts=[ Constant(value=1), Constant(value=2), Constant(value=3), Constant(value=4)], ctx=Load()))]) 
  • You can test for common patterns of malicious pickle files with the --check-safety option
  • You can also safely trace the execution of the Pickle virtual machine without exercising any malicious code with the --trace option.
  • Finally, you can inject arbitrary Python code that will be run on unpickling into an existing pickle file with the --inject option.
  • See Risky Biz's episode for more details.

Brian #2: Python Project-Local Virtualenv Management

  • Hynek Schlawack
  • Only works on UNIX-like systems. MacOS, for example.
  • Instructions
    • Install direnv. (ex: brew install direnv)
    • Put this into a .envrc file in your project root:
    • layout python python3.9
  • Now
    • when you cd into that directory or a subdirectory, your virtual environment is loaded.
    • when you cd out of it, the venv is unloaded
  • Notes:
    • Michael covered direnv on Episode 185. But it wasn’t until Hynek spelled it out for me how to use it with venv that I understood the simplicity and power.
    • Not really faster than creating a venv, but when flipping between several projects, it’s way faster than deactivating/activating.
    • You can also set env variables per directory (kinda the point of direnv)

Erik #3: Testcontainers

“Python port for testcontainers-java that allows using docker containers for functional and integration testing. Testcontainers-python provides capabilities to spin up docker containers (such as a database, Selenium web browser, or any other container) for testing. “ (pypi description).

  • Provides cloud native services, many databases and the like (e.g. Google Cloud Pub/Sub, Kafka..)
  • Originally a java project, still a way to go for us python programmers to implement all services
  • Provides an example for use in CI/CD by leveraging Docker in Docker
 import sqlalchemy from testcontainers.mysql import MySqlContainer with MySqlContainer('mysql:5.7.17') as mysql: engine = sqlalchemy.create_engine(mysql.get_connection_url()) version, = engine.execute("select version()").fetchone() print(version) # 5.7.17 

Michael #4: jc

  • via Garett
  • CLI tool and python library that converts the output of popular command-line tools and file-types to JSON or Dictionaries. This allows piping of output to tools like jq and simplifying automation scripts.
  • Run it as COMMAND ARGS | jc --COMMAND
  • Commands include: systemctl, passwd, ls, jobs, hosts, du, and cksum.

Brian #5: What is Python's Ellipsis Object?

  • Florian Dahlitz
  • Ellipsis or is a constant defined in Python
    • “Ellipsis: The same as the ellipsis literal “...”. Special value used mostly in conjunction with extended slicing syntax for user-defined container data types.”
  • Can be used in type hinting
    • Func returns two int tuple
 def return_tuple() -> tuple[int, int]: pass 
  • Func returns one or more integer:
 def return_tuple() -> tuple[int, ...]: pass 
  • Replacement for pass:
 def my_function(): ... 

Erik #6: PyTorch Forecasting PyTorch Forecasting aims to ease state-of-the-art timeseries forecasting with neural networks for both real-world cases and research alike. The goal is to provide a high-level API with maximum flexibility for professionals and reasonable defaults for beginners.

  • basically tries to achieve for time series what fast.ai has achieved for computer vision and natural language processing
  • The package is built on PyTorch Lightning to allow training on CPUs, single and multiple GPUs out-of-the-box.
  • Implements of Temporal Fusion Transformers
    • interpretable - can calculate feature importance
  • Hyperparameter tuning with optuna

Extras

Brian

Michael

Joke: 200 == 400

254 episodes