splash image

February 22, 2023

Getting started with Python Packaging

In preparation for using a lot more Python, I decided to refresh my Python knowedge and publish my first Python module at https://pypi.org/project/shortscale/.

Some readers may recognize shortscale from earlier explorations in JavaScript, Rust, and Go.

This post covers the following steps:

  1. Install Python on macOS
  2. Write the skeleton code, with just a one-line function.
  3. Build and publish the incomplete v0.1 module.
  4. Complete the logic v1.0.0.
  5. Benchmarks
  6. Python in the browser
  7. Jupyter notebooks

Install python v3.10 (the hard way)

Installing Python on macOS is easiest with the official installer or with homebrew.

I wanted a way to switch between Python versions, so I followed the instructions for pyenv.

NOTE: This does a full local build of CPython, and requires dependencies different from the macOS command line tools.

# 1. Install pyenv 
# from https://github.com/pyenv/pyenv#set-up-your-shell-environment-for-pyenv
git clone https://github.com/pyenv/pyenv.git $HOME/.pyenv
export PYENV_ROOT="$HOME/.pyenv"
export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init --path)"

# 2. Fix dependencies for macOS 
# from https://github.com/pyenv/pyenv/wiki#suggested-build-environment
brew install openssl readline sqlite3 xz zlib tcl-tk

# 3. After the brew install, fix LDFLAGS, CPPFLAGS and add tcl-tk/bin onto PATH 
export LDFLAGS="$LDFLAGS -L$HOME/homebrew/opt/openssl@3/lib -L$HOME/homebrew/opt/readline/lib -L$HOME/homebrew/opt/sqlite/lib -L$HOME/homebrew/opt/zlib/lib -L$HOME/homebrew/opt/tcl-tk/lib -L$HOME/homebrew/opt/openssl@3/lib -L$HOME/homebrew/opt/readline/lib -L$HOME/homebrew/opt/sqlite/lib -L$HOME/homebrew/opt/zlib/lib -L$HOME/homebrew/opt/tcl-tk/lib"
export CPPFLAGS="$CPPFLAGS -I$HOME/homebrew/opt/openssl@3/include -I$HOME/homebrew/opt/readline/include -I$HOME/homebrew/opt/sqlite/include -I$HOME/homebrew/opt/zlib/include -I$HOME/homebrew/opt/tcl-tk/include -I$HOME/homebrew/opt/openssl@3/include -I$HOME/homebrew/opt/readline/include -I$HOME/homebrew/opt/sqlite/include -I$HOME/homebrew/opt/zlib/include -I$HOME/homebrew/opt/tcl-tk/include"
export PATH=$HOME/homebrew/opt/tcl-tk/bin:$PATH

# 4. Use pyenv to build and install python v3.10 and make it the global default
pyenv install 3.10
pyenv global 3.10

# Point to the installed version in  .bash_profile (instead of depending on the pyenv shim)
export PATH=$HOME/.pyenv/versions/3.10.9/bin:$PATH

Virtual environments and pip

Python modules and their dependencies can be installed from pypi.org using pip install.

Configuring a virtual environment will isolate modules under a .venv directory, which is easy to clean up, rather than installing everything globally.

I created a venv under my home directory.

python3 -m venv ~/.venv

Instead of "activating" the venv, which changes the prompt, I prepended the .venv/bin directory onto my PATH.

export PATH=$HOME/.venv/bin:$PATH

Create a new module called shortscale

First I wrote a skeleton shortscale function which just returns a string with the input.

The rest of the code is boilerplate, to make the function callable on the command line. Passing base=0 to int() enables numeric literal input with different bases.

shortscale.py

"""English conversion from number to string"""
import sys

__version__ = "0.1.0"

def shortscale(num: int) -> str:
  return '{} ({} bits)'.format(num, num.bit_length())

def main():
  if len(sys.argv) < 2:
    print ('Usage: shortscale num')
    sys.exit(1)

  print(shortscale(int(sys.argv[1],0)))
  sys.exit(0)

if __name__ == '__main__':
  main()

The output looks like this:

$ python shortscale.py 0x42
66 (7 bits)

Next, I built and published this incomplete v0.1 shortscale module.

Unlike the npm JavaScript ecosystem, you can't just use pip to publish a module to the pypi repository. There are different build tools to choose from.

I chose setuptools because it appears to be the recommended tool, and shows what it's doing. This meant installing build and twine.

Python packages are described in a pyproject.toml. Note that project.scripts points to the CLI entrypoint at main().

pyproject.toml

[project]
name = "shortscale"
description = "English conversion from number to string"
authors = [{name = "Jürgen Leschner", email = "jldec@users.noreply.github.com"}]
readme = "README.md"
license = {file = "LICENSE"}
classifiers = ["License :: OSI Approved :: MIT License"]
dynamic = ["version"]

[project.urls]
Home = "https://github.com/jldec/shortscale-py"

[project.scripts]
shortscale = "shortscale:main"

[build-system]
requires = ["setuptools>=61.0"]
build-backend = "setuptools.build_meta"

[tool.setuptools.dynamic]
version = {attr = "shortscale.__version__"}

Build the module

The build tool creates 2 module bundles (source and runnable code) in the ./dist directory.

$ python -m build
...
Successfully built shortscale-0.1.0.tar.gz and shortscale-0.1.0-py3-none-any.whl

Publish to pypi.org

$ python -m twine upload dist/*
Uploading distributions to https://upload.pypi.org/legacy/
Uploading shortscale-0.1.0-py3-none-any.whl
Uploading shortscale-0.1.0.tar.gz
...
View at:
https://pypi.org/project/shortscale/0.1.0/

Install and run in a venv

The moment of truth. Install the module in a new venv, and invoke it.

$ mkdir test
$ cd test
$ python -m venv .venv
$ source .venv/bin/activate

(.venv) $ pip install shortscale
...
Successfully installed shortscale-0.1.0

(.venv) $ shortscale 0xffffffffffff
281474976710655 (48 bits)

$ deactivate

Complete the logic

Python still amazes me with its terseness and readability.

The first iteration had 3 functions, of which the longest had 30 lines with generous spacing.

One of those functions decomposes a number into powers of 1000.

def powers_of_1000(n: int):
    """
    Return list of (n, exponent) for each power of 1000.
    List is ordered highest exponent first.
    n = 0 - 999.
    exponent = 0,1,2,3...
    """
    p_list = []
    exponent = 0
    while n > 0:
        p_list.insert(0, (n % 1000, exponent))
        n = n // 1000
        exponent += 1

    return p_list

Playing aound in a Jupyter notebook, I was able to eliminate the extra function (and the list which it returns), simply by reversing the order of building the shortscale output.

Screenshot of a Jupyter notebook in VS Code exploring shortscale

Using a Jupyter environment in VS Code is a clear win. The result was simpler and faster.

Testing

There is nice support for Python testing and debugging in VS Code.

The function to run unit tests took just 3 lines.

Screenshot of VS Code Test integration for pytest

Benchmarks

I was pleased with the benchmarks as well. For this string manipulation micro-benchmark, CPython 3.11 is only 1.5x slower than V8 JavaScript!

Compiled languages like Go and Rust will outperform that, but again, not by a huge amount.

The results below are from my personal M1 arm64 running macOS.

Python

Python v3.11.2

$ python tests/bench_shortscale.py

 50000 calls,    5000000 bytes,     1264 ns/call
100000 calls,   10000000 bytes,     1216 ns/call
200000 calls,   20000000 bytes,     1216 ns/call

Python v3.10.9

$ python tests/bench_shortscale.py

 50000 calls,    5000000 bytes,     1811 ns/call
100000 calls,   10000000 bytes,     1808 ns/call
200000 calls,   20000000 bytes,     1809 ns/call

Javascript

$ node test/bench.js

20000 calls, 2000000 bytes, 796 ns/call
20000 calls, 2000000 bytes, 790 ns/call
20000 calls, 2000000 bytes, 797 ns/call

Go

$ go test -bench . -benchmem

BenchmarkShortscale-8   	 4227788	       252.0 ns/op	     248 B/op	       5 allocs/op

Rust

$ cargo bench

running 2 tests
test a_shortscale                        ... bench:         182 ns/iter (+/- 3)
test b_shortscale_string_writer_no_alloc ... bench:          63 ns/iter (+/- 2)

Let's run shortscale in the browser

Open your browser on https://pyodide.org/en/stable/console.html and paste the following python commands into the python REPL, line by line.

import micropip
await micropip.install("shortscale")
import shortscale

shortscale.shortscale(0xffff0000)

shortscale.bench_shortscale()

Screenshot of browser at https://pyodide.org/en/stable/console.html running shortscale

It looks like Python in WASM in the browser is only 2 to 3 times slower than native CPython. Amazing!

Jupyter notebooks on GitHub

GitHub shows the output of Jupyter notebook (.ipynb) files in your browser

https://github.com/jldec/shortscale-py/blob/main/shortscale.ipynb

Screenshot of https://github.com/jldec/shortscale-py/blob/main/shortscale.ipynb

Google Colaboratory

You can also open the notebook from GitHub in a Google Colaboratory environment

https://colab.research.google.com/github/jldec/shortscale-py/blob/main/shortscale.ipynb

Screenshot of Google Colaboratory running at https://colab.research.google.com/github/jldec/shortscale-py/blob/main/shortscale.ipynb

Keep on learning
🚀

debug

user: anonymous

{
  "path": "/blog/getting-started-with-python-packaging",
  "attrs": {
    "alias": "/getting-started-with-python",
    "title": "Getting started with Python Packaging",
    "splash": {
      "image": "/images/rockford-office.jpg"
    },
    "date": "2023-02-22",
    "layout": "BlogPostLayout",
    "excerpt": "From zero to publishing a Python module on pypi.org."
  },
  "md": "# Getting started with Python Packaging\n\nIn preparation for using a lot more Python, I decided to refresh my Python knowedge and publish my first Python module at https://pypi.org/project/shortscale/.\n\nSome readers may recognize [shortscale](forays-from-node-to-rust) from earlier explorations in [JavaScript](https://github.com/jldec/shortscale), [Rust](https://github.com/jldec/shortscale-rs), and [Go](https://github.com/jldec/shortscale-go).\n\nThis post covers the following steps:\n\n1. Install Python on macOS\n2. Write the skeleton code, with just a one-line function.\n3. Build and publish the incomplete [v0.1](https://github.com/jldec/shortscale-py/tree/bb9b026b9097ce9c601e632a9d1f74a7da6adf29) module.\n4. Complete the logic [v1.0.0](https://github.com/jldec/shortscale-py/commit/6ab4a4b541590a60ebe2944473094465ba8f14f5).\n5. Benchmarks\n6. Python in the browser\n7. Jupyter notebooks\n\n## Install python v3.10 (the hard way)\n\nInstalling Python on macOS is easiest with [the official installer](https://www.python.org/downloads/macos/) or [with homebrew](https://realPython.com/installing-python/#how-to-install-from-homebrew). \n\nI wanted a way to switch between Python versions, so I followed the instructions for [pyenv](https://github.com/pyenv/pyenv). \n\nNOTE: This does a full local build of CPython, and requires dependencies different from the macOS command line tools.\n\n```\n# 1. Install pyenv \n# from https://github.com/pyenv/pyenv#set-up-your-shell-environment-for-pyenv\ngit clone https://github.com/pyenv/pyenv.git $HOME/.pyenv\nexport PYENV_ROOT=\"$HOME/.pyenv\"\nexport PATH=\"$PYENV_ROOT/bin:$PATH\"\neval \"$(pyenv init --path)\"\n\n# 2. Fix dependencies for macOS \n# from https://github.com/pyenv/pyenv/wiki#suggested-build-environment\nbrew install openssl readline sqlite3 xz zlib tcl-tk\n\n# 3. After the brew install, fix LDFLAGS, CPPFLAGS and add tcl-tk/bin onto PATH \nexport LDFLAGS=\"$LDFLAGS -L$HOME/homebrew/opt/openssl@3/lib -L$HOME/homebrew/opt/readline/lib -L$HOME/homebrew/opt/sqlite/lib -L$HOME/homebrew/opt/zlib/lib -L$HOME/homebrew/opt/tcl-tk/lib -L$HOME/homebrew/opt/openssl@3/lib -L$HOME/homebrew/opt/readline/lib -L$HOME/homebrew/opt/sqlite/lib -L$HOME/homebrew/opt/zlib/lib -L$HOME/homebrew/opt/tcl-tk/lib\"\nexport CPPFLAGS=\"$CPPFLAGS -I$HOME/homebrew/opt/openssl@3/include -I$HOME/homebrew/opt/readline/include -I$HOME/homebrew/opt/sqlite/include -I$HOME/homebrew/opt/zlib/include -I$HOME/homebrew/opt/tcl-tk/include -I$HOME/homebrew/opt/openssl@3/include -I$HOME/homebrew/opt/readline/include -I$HOME/homebrew/opt/sqlite/include -I$HOME/homebrew/opt/zlib/include -I$HOME/homebrew/opt/tcl-tk/include\"\nexport PATH=$HOME/homebrew/opt/tcl-tk/bin:$PATH\n\n# 4. Use pyenv to build and install python v3.10 and make it the global default\npyenv install 3.10\npyenv global 3.10\n\n# Point to the installed version in  .bash_profile (instead of depending on the pyenv shim)\nexport PATH=$HOME/.pyenv/versions/3.10.9/bin:$PATH\n```\n\n## Virtual environments and pip\n\nPython [modules](https://docs.python.org/3/tutorial/modules.html) and their dependencies can be installed from [pypi.org](https://pypi.org/) using `pip install`.\n\nConfiguring a [virtual environment](https://docs.python.org/3/library/venv.html) will isolate modules under a `.venv` directory, which is easy to clean up, rather than installing everything globally.\n\nI created a venv under my home directory.\n\n```sh\npython3 -m venv ~/.venv\n```\n\nInstead of \"activating\" the venv, which changes the prompt, I prepended the .venv/bin directory onto my PATH.\n\n```\nexport PATH=$HOME/.venv/bin:$PATH\n```\n\n## Create a new module called shortscale\n\nFirst I wrote a skeleton `shortscale` function which just returns a string with the input.\n\nThe rest of the code is boilerplate, to make the function callable on the command line. Passing base=0 to [int()](https://docs.python.org/3/library/functions.html#int) enables numeric literal input with different bases. \n\n#### shortscale.py\n```py\n\"\"\"English conversion from number to string\"\"\"\nimport sys\n\n__version__ = \"0.1.0\"\n\ndef shortscale(num: int) -> str:\n  return '{} ({} bits)'.format(num, num.bit_length())\n\ndef main():\n  if len(sys.argv) < 2:\n    print ('Usage: shortscale num')\n    sys.exit(1)\n\n  print(shortscale(int(sys.argv[1],0)))\n  sys.exit(0)\n\nif __name__ == '__main__':\n  main()\n\n```\n\nThe output looks like this:\n\n```sh\n$ python shortscale.py 0x42\n66 (7 bits)\n```\n\nNext, I built and published this incomplete [v0.1](https://github.com/jldec/shortscale-py/tree/bb9b026b9097ce9c601e632a9d1f74a7da6adf29) shortscale module.\n\nUnlike the npm JavaScript ecosystem, you can't just use pip to publish a module to the pypi repository. There are [different build tools](https://packaging.python.org/en/latest/tutorials/packaging-projects/#creating-pyproject-toml) to choose from.\n\nI chose `setuptools` because it appears to be the recommended tool, and shows what it's doing. This meant installing [build](https://pypi.org/project/build/) and [twine](https://pypi.org/project/build/).   \n\nPython packages are described in a `pyproject.toml`. Note that project.scripts points to the CLI entrypoint at main().\n\n#### pyproject.toml\n```toml\n[project]\nname = \"shortscale\"\ndescription = \"English conversion from number to string\"\nauthors = [{name = \"Jürgen Leschner\", email = \"jldec@users.noreply.github.com\"}]\nreadme = \"README.md\"\nlicense = {file = \"LICENSE\"}\nclassifiers = [\"License :: OSI Approved :: MIT License\"]\ndynamic = [\"version\"]\n\n[project.urls]\nHome = \"https://github.com/jldec/shortscale-py\"\n\n[project.scripts]\nshortscale = \"shortscale:main\"\n\n[build-system]\nrequires = [\"setuptools>=61.0\"]\nbuild-backend = \"setuptools.build_meta\"\n\n[tool.setuptools.dynamic]\nversion = {attr = \"shortscale.__version__\"}\n```\n\n### Build the module\nThe build tool creates 2 module bundles (source and runnable code) in the ./dist directory.\n\n```\n$ python -m build\n...\nSuccessfully built shortscale-0.1.0.tar.gz and shortscale-0.1.0-py3-none-any.whl\n```\n\n### Publish to pypi.org\n```\n$ python -m twine upload dist/*\nUploading distributions to https://upload.pypi.org/legacy/\nUploading shortscale-0.1.0-py3-none-any.whl\nUploading shortscale-0.1.0.tar.gz\n...\nView at:\nhttps://pypi.org/project/shortscale/0.1.0/\n```\n\n## Install and run in a venv\n\nThe moment of truth. Install the module in a new venv, and invoke it.\n\n```sh\n$ mkdir test\n$ cd test\n$ python -m venv .venv\n$ source .venv/bin/activate\n\n(.venv) $ pip install shortscale\n...\nSuccessfully installed shortscale-0.1.0\n\n(.venv) $ shortscale 0xffffffffffff\n281474976710655 (48 bits)\n\n$ deactivate\n```\n\n## Complete the logic\n\nPython still amazes me with its terseness and readability. \n\nThe first iteration had [3 functions](https://github.com/jldec/shortscale-py/blob/main/shortscale.py), of which the longest had 30 lines with generous spacing. \n\nOne of those functions decomposes a number into powers of 1000.\n\n```py\ndef powers_of_1000(n: int):\n    \"\"\"\n    Return list of (n, exponent) for each power of 1000.\n    List is ordered highest exponent first.\n    n = 0 - 999.\n    exponent = 0,1,2,3...\n    \"\"\"\n    p_list = []\n    exponent = 0\n    while n > 0:\n        p_list.insert(0, (n % 1000, exponent))\n        n = n // 1000\n        exponent += 1\n\n    return p_list\n```\n\nPlaying aound in a Jupyter notebook, I was able to eliminate the extra function (and the list which it returns), simply by reversing the order of building the shortscale output.\n\n![Screenshot of a Jupyter notebook in VS Code exploring shortscale](/images/jupyter-notebook-vs-code.png)\n\nUsing a Jupyter environment in VS Code is a clear win. The [result](https://github.com/jldec/shortscale-py/pull/3) was simpler and faster.\n\n\n## Testing\nThere is nice support for Python testing and debugging in VS Code. \n\nThe [function](https://github.com/jldec/shortscale-py/blob/main/shortscale/tests/test_shortscale.py) to run unit tests took just 3 lines. \n\n![Screenshot of VS Code Test integration for pytest](/images/python-test-vscode.png)\n\n## Benchmarks\n\nI was pleased with the [benchmarks](https://github.com/jldec/shortscale-py/blob/main/tests/bench_shortscale.py) as well. For this string manipulation micro-benchmark, CPython 3.11 is only 1.5x slower than V8 JavaScript! \n\nCompiled languages like Go and Rust will outperform that, but again, not by a huge amount.   \n\nThe results below are from my personal M1 arm64 running macOS.\n\n### Python\n\n#### Python v3.11.2\n```\n$ python tests/bench_shortscale.py\n\n 50000 calls,    5000000 bytes,     1264 ns/call\n100000 calls,   10000000 bytes,     1216 ns/call\n200000 calls,   20000000 bytes,     1216 ns/call\n```\n\n#### Python v3.10.9\n```\n$ python tests/bench_shortscale.py\n\n 50000 calls,    5000000 bytes,     1811 ns/call\n100000 calls,   10000000 bytes,     1808 ns/call\n200000 calls,   20000000 bytes,     1809 ns/call\n```\n\n### Javascript\n```\n$ node test/bench.js\n\n20000 calls, 2000000 bytes, 796 ns/call\n20000 calls, 2000000 bytes, 790 ns/call\n20000 calls, 2000000 bytes, 797 ns/call\n```\n\n### Go\n```\n$ go test -bench . -benchmem\n\nBenchmarkShortscale-8   \t 4227788\t       252.0 ns/op\t     248 B/op\t       5 allocs/op\n```\n\n### Rust\n```\n$ cargo bench\n\nrunning 2 tests\ntest a_shortscale                        ... bench:         182 ns/iter (+/- 3)\ntest b_shortscale_string_writer_no_alloc ... bench:          63 ns/iter (+/- 2)\n```\n\n## Let's run shortscale in the browser\n\nOpen your browser on https://pyodide.org/en/stable/console.html and paste the following python commands into the python REPL, line by line.\n\n```python\nimport micropip\nawait micropip.install(\"shortscale\")\nimport shortscale\n\nshortscale.shortscale(0xffff0000)\n\nshortscale.bench_shortscale()\n```\n\n![Screenshot of browser at https://pyodide.org/en/stable/console.html running shortscale](/images/pyodide-shortscale.png)\n\nIt looks like Python in WASM in the browser is only 2 to 3 times slower than native CPython. Amazing!\n\n## Jupyter notebooks on GitHub\n\nGitHub shows the output of Jupyter notebook (.ipynb) files in your browser\n\nhttps://github.com/jldec/shortscale-py/blob/main/shortscale.ipynb\n\n![Screenshot of https://github.com/jldec/shortscale-py/blob/main/shortscale.ipynb](/images/shortscale-notebook-github.png)\n\n## Google Colaboratory\n\nYou can also open the [notebook](https://github.com/jldec/shortscale-py/blob/main/shortscale.ipynb) from GitHub in a [Google Colaboratory](https://colab.research.google.com/github/jldec/shortscale-py/blob/main/shortscale.ipynb) environment\n\nhttps://colab.research.google.com/github/jldec/shortscale-py/blob/main/shortscale.ipynb\n\n![Screenshot of Google Colaboratory running at https://colab.research.google.com/github/jldec/shortscale-py/blob/main/shortscale.ipynb](/images/shortscale-on-google-colaboratory.png)\n\n>> Keep on learning  \n>> 🚀\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
  "html": "<h1>Getting started with Python Packaging</h1>\n<p>In preparation for using a lot more Python, I decided to refresh my Python knowedge and publish my first Python module at <a href=\"https://pypi.org/project/shortscale/\">https://pypi.org/project/shortscale/</a>.</p>\n<p>Some readers may recognize <a href=\"forays-from-node-to-rust\">shortscale</a> from earlier explorations in <a href=\"https://github.com/jldec/shortscale\">JavaScript</a>, <a href=\"https://github.com/jldec/shortscale-rs\">Rust</a>, and <a href=\"https://github.com/jldec/shortscale-go\">Go</a>.</p>\n<p>This post covers the following steps:</p>\n<ol>\n<li>Install Python on macOS</li>\n<li>Write the skeleton code, with just a one-line function.</li>\n<li>Build and publish the incomplete <a href=\"https://github.com/jldec/shortscale-py/tree/bb9b026b9097ce9c601e632a9d1f74a7da6adf29\">v0.1</a> module.</li>\n<li>Complete the logic <a href=\"https://github.com/jldec/shortscale-py/commit/6ab4a4b541590a60ebe2944473094465ba8f14f5\">v1.0.0</a>.</li>\n<li>Benchmarks</li>\n<li>Python in the browser</li>\n<li>Jupyter notebooks</li>\n</ol>\n<h2>Install python v3.10 (the hard way)</h2>\n<p>Installing Python on macOS is easiest with <a href=\"https://www.python.org/downloads/macos/\">the official installer</a> or <a href=\"https://realPython.com/installing-python/#how-to-install-from-homebrew\">with homebrew</a>.</p>\n<p>I wanted a way to switch between Python versions, so I followed the instructions for <a href=\"https://github.com/pyenv/pyenv\">pyenv</a>.</p>\n<p>NOTE: This does a full local build of CPython, and requires dependencies different from the macOS command line tools.</p>\n<pre><code># 1. Install pyenv \n# from https://github.com/pyenv/pyenv#set-up-your-shell-environment-for-pyenv\ngit clone https://github.com/pyenv/pyenv.git $HOME/.pyenv\nexport PYENV_ROOT=&quot;$HOME/.pyenv&quot;\nexport PATH=&quot;$PYENV_ROOT/bin:$PATH&quot;\neval &quot;$(pyenv init --path)&quot;\n\n# 2. Fix dependencies for macOS \n# from https://github.com/pyenv/pyenv/wiki#suggested-build-environment\nbrew install openssl readline sqlite3 xz zlib tcl-tk\n\n# 3. After the brew install, fix LDFLAGS, CPPFLAGS and add tcl-tk/bin onto PATH \nexport LDFLAGS=&quot;$LDFLAGS -L$HOME/homebrew/opt/openssl@3/lib -L$HOME/homebrew/opt/readline/lib -L$HOME/homebrew/opt/sqlite/lib -L$HOME/homebrew/opt/zlib/lib -L$HOME/homebrew/opt/tcl-tk/lib -L$HOME/homebrew/opt/openssl@3/lib -L$HOME/homebrew/opt/readline/lib -L$HOME/homebrew/opt/sqlite/lib -L$HOME/homebrew/opt/zlib/lib -L$HOME/homebrew/opt/tcl-tk/lib&quot;\nexport CPPFLAGS=&quot;$CPPFLAGS -I$HOME/homebrew/opt/openssl@3/include -I$HOME/homebrew/opt/readline/include -I$HOME/homebrew/opt/sqlite/include -I$HOME/homebrew/opt/zlib/include -I$HOME/homebrew/opt/tcl-tk/include -I$HOME/homebrew/opt/openssl@3/include -I$HOME/homebrew/opt/readline/include -I$HOME/homebrew/opt/sqlite/include -I$HOME/homebrew/opt/zlib/include -I$HOME/homebrew/opt/tcl-tk/include&quot;\nexport PATH=$HOME/homebrew/opt/tcl-tk/bin:$PATH\n\n# 4. Use pyenv to build and install python v3.10 and make it the global default\npyenv install 3.10\npyenv global 3.10\n\n# Point to the installed version in  .bash_profile (instead of depending on the pyenv shim)\nexport PATH=$HOME/.pyenv/versions/3.10.9/bin:$PATH\n</code></pre>\n<h2>Virtual environments and pip</h2>\n<p>Python <a href=\"https://docs.python.org/3/tutorial/modules.html\">modules</a> and their dependencies can be installed from <a href=\"https://pypi.org/\">pypi.org</a> using <code>pip install</code>.</p>\n<p>Configuring a <a href=\"https://docs.python.org/3/library/venv.html\">virtual environment</a> will isolate modules under a <code>.venv</code> directory, which is easy to clean up, rather than installing everything globally.</p>\n<p>I created a venv under my home directory.</p>\n<pre><code class=\"language-sh\">python3 -m venv ~/.venv\n</code></pre>\n<p>Instead of &quot;activating&quot; the venv, which changes the prompt, I prepended the .venv/bin directory onto my PATH.</p>\n<pre><code>export PATH=$HOME/.venv/bin:$PATH\n</code></pre>\n<h2>Create a new module called shortscale</h2>\n<p>First I wrote a skeleton <code>shortscale</code> function which just returns a string with the input.</p>\n<p>The rest of the code is boilerplate, to make the function callable on the command line. Passing base=0 to <a href=\"https://docs.python.org/3/library/functions.html#int\">int()</a> enables numeric literal input with different bases.</p>\n<h4><a href=\"http://shortscale.py\">shortscale.py</a></h4>\n<pre><code class=\"language-py\">&quot;&quot;&quot;English conversion from number to string&quot;&quot;&quot;\nimport sys\n\n__version__ = &quot;0.1.0&quot;\n\ndef shortscale(num: int) -&gt; str:\n  return '{} ({} bits)'.format(num, num.bit_length())\n\ndef main():\n  if len(sys.argv) &lt; 2:\n    print ('Usage: shortscale num')\n    sys.exit(1)\n\n  print(shortscale(int(sys.argv[1],0)))\n  sys.exit(0)\n\nif __name__ == '__main__':\n  main()\n\n</code></pre>\n<p>The output looks like this:</p>\n<pre><code class=\"language-sh\">$ python shortscale.py 0x42\n66 (7 bits)\n</code></pre>\n<p>Next, I built and published this incomplete <a href=\"https://github.com/jldec/shortscale-py/tree/bb9b026b9097ce9c601e632a9d1f74a7da6adf29\">v0.1</a> shortscale module.</p>\n<p>Unlike the npm JavaScript ecosystem, you can't just use pip to publish a module to the pypi repository. There are <a href=\"https://packaging.python.org/en/latest/tutorials/packaging-projects/#creating-pyproject-toml\">different build tools</a> to choose from.</p>\n<p>I chose <code>setuptools</code> because it appears to be the recommended tool, and shows what it's doing. This meant installing <a href=\"https://pypi.org/project/build/\">build</a> and <a href=\"https://pypi.org/project/build/\">twine</a>.</p>\n<p>Python packages are described in a <code>pyproject.toml</code>. Note that project.scripts points to the CLI entrypoint at main().</p>\n<h4>pyproject.toml</h4>\n<pre><code class=\"language-toml\">[project]\nname = &quot;shortscale&quot;\ndescription = &quot;English conversion from number to string&quot;\nauthors = [{name = &quot;Jürgen Leschner&quot;, email = &quot;jldec@users.noreply.github.com&quot;}]\nreadme = &quot;README.md&quot;\nlicense = {file = &quot;LICENSE&quot;}\nclassifiers = [&quot;License :: OSI Approved :: MIT License&quot;]\ndynamic = [&quot;version&quot;]\n\n[project.urls]\nHome = &quot;https://github.com/jldec/shortscale-py&quot;\n\n[project.scripts]\nshortscale = &quot;shortscale:main&quot;\n\n[build-system]\nrequires = [&quot;setuptools&gt;=61.0&quot;]\nbuild-backend = &quot;setuptools.build_meta&quot;\n\n[tool.setuptools.dynamic]\nversion = {attr = &quot;shortscale.__version__&quot;}\n</code></pre>\n<h3>Build the module</h3>\n<p>The build tool creates 2 module bundles (source and runnable code) in the ./dist directory.</p>\n<pre><code>$ python -m build\n...\nSuccessfully built shortscale-0.1.0.tar.gz and shortscale-0.1.0-py3-none-any.whl\n</code></pre>\n<h3>Publish to <a href=\"http://pypi.org\">pypi.org</a></h3>\n<pre><code>$ python -m twine upload dist/*\nUploading distributions to https://upload.pypi.org/legacy/\nUploading shortscale-0.1.0-py3-none-any.whl\nUploading shortscale-0.1.0.tar.gz\n...\nView at:\nhttps://pypi.org/project/shortscale/0.1.0/\n</code></pre>\n<h2>Install and run in a venv</h2>\n<p>The moment of truth. Install the module in a new venv, and invoke it.</p>\n<pre><code class=\"language-sh\">$ mkdir test\n$ cd test\n$ python -m venv .venv\n$ source .venv/bin/activate\n\n(.venv) $ pip install shortscale\n...\nSuccessfully installed shortscale-0.1.0\n\n(.venv) $ shortscale 0xffffffffffff\n281474976710655 (48 bits)\n\n$ deactivate\n</code></pre>\n<h2>Complete the logic</h2>\n<p>Python still amazes me with its terseness and readability.</p>\n<p>The first iteration had <a href=\"https://github.com/jldec/shortscale-py/blob/main/shortscale.py\">3 functions</a>, of which the longest had 30 lines with generous spacing.</p>\n<p>One of those functions decomposes a number into powers of 1000.</p>\n<pre><code class=\"language-py\">def powers_of_1000(n: int):\n    &quot;&quot;&quot;\n    Return list of (n, exponent) for each power of 1000.\n    List is ordered highest exponent first.\n    n = 0 - 999.\n    exponent = 0,1,2,3...\n    &quot;&quot;&quot;\n    p_list = []\n    exponent = 0\n    while n &gt; 0:\n        p_list.insert(0, (n % 1000, exponent))\n        n = n // 1000\n        exponent += 1\n\n    return p_list\n</code></pre>\n<p>Playing aound in a Jupyter notebook, I was able to eliminate the extra function (and the list which it returns), simply by reversing the order of building the shortscale output.</p>\n<p><img src=\"/images/jupyter-notebook-vs-code.png\" alt=\"Screenshot of a Jupyter notebook in VS Code exploring shortscale\"></p>\n<p>Using a Jupyter environment in VS Code is a clear win. The <a href=\"https://github.com/jldec/shortscale-py/pull/3\">result</a> was simpler and faster.</p>\n<h2>Testing</h2>\n<p>There is nice support for Python testing and debugging in VS Code.</p>\n<p>The <a href=\"https://github.com/jldec/shortscale-py/blob/main/shortscale/tests/test_shortscale.py\">function</a> to run unit tests took just 3 lines.</p>\n<p><img src=\"/images/python-test-vscode.png\" alt=\"Screenshot of VS Code Test integration for pytest\"></p>\n<h2>Benchmarks</h2>\n<p>I was pleased with the <a href=\"https://github.com/jldec/shortscale-py/blob/main/tests/bench_shortscale.py\">benchmarks</a> as well. For this string manipulation micro-benchmark, CPython 3.11 is only 1.5x slower than V8 JavaScript!</p>\n<p>Compiled languages like Go and Rust will outperform that, but again, not by a huge amount.</p>\n<p>The results below are from my personal M1 arm64 running macOS.</p>\n<h3>Python</h3>\n<h4>Python v3.11.2</h4>\n<pre><code>$ python tests/bench_shortscale.py\n\n 50000 calls,    5000000 bytes,     1264 ns/call\n100000 calls,   10000000 bytes,     1216 ns/call\n200000 calls,   20000000 bytes,     1216 ns/call\n</code></pre>\n<h4>Python v3.10.9</h4>\n<pre><code>$ python tests/bench_shortscale.py\n\n 50000 calls,    5000000 bytes,     1811 ns/call\n100000 calls,   10000000 bytes,     1808 ns/call\n200000 calls,   20000000 bytes,     1809 ns/call\n</code></pre>\n<h3>Javascript</h3>\n<pre><code>$ node test/bench.js\n\n20000 calls, 2000000 bytes, 796 ns/call\n20000 calls, 2000000 bytes, 790 ns/call\n20000 calls, 2000000 bytes, 797 ns/call\n</code></pre>\n<h3>Go</h3>\n<pre><code>$ go test -bench . -benchmem\n\nBenchmarkShortscale-8   \t 4227788\t       252.0 ns/op\t     248 B/op\t       5 allocs/op\n</code></pre>\n<h3>Rust</h3>\n<pre><code>$ cargo bench\n\nrunning 2 tests\ntest a_shortscale                        ... bench:         182 ns/iter (+/- 3)\ntest b_shortscale_string_writer_no_alloc ... bench:          63 ns/iter (+/- 2)\n</code></pre>\n<h2>Let's run shortscale in the browser</h2>\n<p>Open your browser on <a href=\"https://pyodide.org/en/stable/console.html\">https://pyodide.org/en/stable/console.html</a> and paste the following python commands into the python REPL, line by line.</p>\n<pre><code class=\"language-python\">import micropip\nawait micropip.install(&quot;shortscale&quot;)\nimport shortscale\n\nshortscale.shortscale(0xffff0000)\n\nshortscale.bench_shortscale()\n</code></pre>\n<p><img src=\"/images/pyodide-shortscale.png\" alt=\"Screenshot of browser at https://pyodide.org/en/stable/console.html running shortscale\"></p>\n<p>It looks like Python in WASM in the browser is only 2 to 3 times slower than native CPython. Amazing!</p>\n<h2>Jupyter notebooks on GitHub</h2>\n<p>GitHub shows the output of Jupyter notebook (.ipynb) files in your browser</p>\n<p><a href=\"https://github.com/jldec/shortscale-py/blob/main/shortscale.ipynb\">https://github.com/jldec/shortscale-py/blob/main/shortscale.ipynb</a></p>\n<p><img src=\"/images/shortscale-notebook-github.png\" alt=\"Screenshot of https://github.com/jldec/shortscale-py/blob/main/shortscale.ipynb\"></p>\n<h2>Google Colaboratory</h2>\n<p>You can also open the <a href=\"https://github.com/jldec/shortscale-py/blob/main/shortscale.ipynb\">notebook</a> from GitHub in a <a href=\"https://colab.research.google.com/github/jldec/shortscale-py/blob/main/shortscale.ipynb\">Google Colaboratory</a> environment</p>\n<p><a href=\"https://colab.research.google.com/github/jldec/shortscale-py/blob/main/shortscale.ipynb\">https://colab.research.google.com/github/jldec/shortscale-py/blob/main/shortscale.ipynb</a></p>\n<p><img src=\"/images/shortscale-on-google-colaboratory.png\" alt=\"Screenshot of Google Colaboratory running at https://colab.research.google.com/github/jldec/shortscale-py/blob/main/shortscale.ipynb\"></p>\n<blockquote>\n<blockquote>\n<p>Keep on learning<br>\n🚀</p>\n</blockquote>\n</blockquote>\n"
}