In preparation for using a lot more Python, I decided to refresh my Python knowedge and publish my first Python module at https://pypi.org/project/shortscale/.
Some readers may recognize shortscale from earlier explorations in JavaScript, Rust, and Go.
This post covers the following steps:
Installing Python on macOS is easiest with the official installer or with homebrew.
I wanted a way to switch between Python versions, so I followed the instructions for pyenv.
NOTE: This does a full local build of CPython, and requires dependencies different from the macOS command line tools.
# 1. Install pyenv
# from https://github.com/pyenv/pyenv#set-up-your-shell-environment-for-pyenv
git clone https://github.com/pyenv/pyenv.git $HOME/.pyenv
export PYENV_ROOT="$HOME/.pyenv"
export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init --path)"
# 2. Fix dependencies for macOS
# from https://github.com/pyenv/pyenv/wiki#suggested-build-environment
brew install openssl readline sqlite3 xz zlib tcl-tk
# 3. After the brew install, fix LDFLAGS, CPPFLAGS and add tcl-tk/bin onto PATH
export LDFLAGS="$LDFLAGS -L$HOME/homebrew/opt/openssl@3/lib -L$HOME/homebrew/opt/readline/lib -L$HOME/homebrew/opt/sqlite/lib -L$HOME/homebrew/opt/zlib/lib -L$HOME/homebrew/opt/tcl-tk/lib -L$HOME/homebrew/opt/openssl@3/lib -L$HOME/homebrew/opt/readline/lib -L$HOME/homebrew/opt/sqlite/lib -L$HOME/homebrew/opt/zlib/lib -L$HOME/homebrew/opt/tcl-tk/lib"
export CPPFLAGS="$CPPFLAGS -I$HOME/homebrew/opt/openssl@3/include -I$HOME/homebrew/opt/readline/include -I$HOME/homebrew/opt/sqlite/include -I$HOME/homebrew/opt/zlib/include -I$HOME/homebrew/opt/tcl-tk/include -I$HOME/homebrew/opt/openssl@3/include -I$HOME/homebrew/opt/readline/include -I$HOME/homebrew/opt/sqlite/include -I$HOME/homebrew/opt/zlib/include -I$HOME/homebrew/opt/tcl-tk/include"
export PATH=$HOME/homebrew/opt/tcl-tk/bin:$PATH
# 4. Use pyenv to build and install python v3.10 and make it the global default
pyenv install 3.10
pyenv global 3.10
# Point to the installed version in .bash_profile (instead of depending on the pyenv shim)
export PATH=$HOME/.pyenv/versions/3.10.9/bin:$PATH
Python modules and their dependencies can be installed from pypi.org using pip install
.
Configuring a virtual environment will isolate modules under a .venv
directory, which is easy to clean up, rather than installing everything globally.
I created a venv under my home directory.
python3 -m venv ~/.venv
Instead of "activating" the venv, which changes the prompt, I prepended the .venv/bin directory onto my PATH.
export PATH=$HOME/.venv/bin:$PATH
First I wrote a skeleton shortscale
function which just returns a string with the input.
The rest of the code is boilerplate, to make the function callable on the command line. Passing base=0 to int() enables numeric literal input with different bases.
"""English conversion from number to string"""
import sys
__version__ = "0.1.0"
def shortscale(num: int) -> str:
return '{} ({} bits)'.format(num, num.bit_length())
def main():
if len(sys.argv) < 2:
print ('Usage: shortscale num')
sys.exit(1)
print(shortscale(int(sys.argv[1],0)))
sys.exit(0)
if __name__ == '__main__':
main()
The output looks like this:
$ python shortscale.py 0x42
66 (7 bits)
Next, I built and published this incomplete v0.1 shortscale module.
Unlike the npm JavaScript ecosystem, you can't just use pip to publish a module to the pypi repository. There are different build tools to choose from.
I chose setuptools
because it appears to be the recommended tool, and shows what it's doing. This meant installing build and twine.
Python packages are described in a pyproject.toml
. Note that project.scripts points to the CLI entrypoint at main().
[project]
name = "shortscale"
description = "English conversion from number to string"
authors = [{name = "Jürgen Leschner", email = "jldec@users.noreply.github.com"}]
readme = "README.md"
license = {file = "LICENSE"}
classifiers = ["License :: OSI Approved :: MIT License"]
dynamic = ["version"]
[project.urls]
Home = "https://github.com/jldec/shortscale-py"
[project.scripts]
shortscale = "shortscale:main"
[build-system]
requires = ["setuptools>=61.0"]
build-backend = "setuptools.build_meta"
[tool.setuptools.dynamic]
version = {attr = "shortscale.__version__"}
The build tool creates 2 module bundles (source and runnable code) in the ./dist directory.
$ python -m build
...
Successfully built shortscale-0.1.0.tar.gz and shortscale-0.1.0-py3-none-any.whl
$ python -m twine upload dist/*
Uploading distributions to https://upload.pypi.org/legacy/
Uploading shortscale-0.1.0-py3-none-any.whl
Uploading shortscale-0.1.0.tar.gz
...
View at:
https://pypi.org/project/shortscale/0.1.0/
The moment of truth. Install the module in a new venv, and invoke it.
$ mkdir test
$ cd test
$ python -m venv .venv
$ source .venv/bin/activate
(.venv) $ pip install shortscale
...
Successfully installed shortscale-0.1.0
(.venv) $ shortscale 0xffffffffffff
281474976710655 (48 bits)
$ deactivate
Python still amazes me with its terseness and readability.
The first iteration had 3 functions, of which the longest had 30 lines with generous spacing.
One of those functions decomposes a number into powers of 1000.
def powers_of_1000(n: int):
"""
Return list of (n, exponent) for each power of 1000.
List is ordered highest exponent first.
n = 0 - 999.
exponent = 0,1,2,3...
"""
p_list = []
exponent = 0
while n > 0:
p_list.insert(0, (n % 1000, exponent))
n = n // 1000
exponent += 1
return p_list
Playing aound in a Jupyter notebook, I was able to eliminate the extra function (and the list which it returns), simply by reversing the order of building the shortscale output.
Using a Jupyter environment in VS Code is a clear win. The result was simpler and faster.
There is nice support for Python testing and debugging in VS Code.
The function to run unit tests took just 3 lines.
I was pleased with the benchmarks as well. For this string manipulation micro-benchmark, CPython 3.11 is only 1.5x slower than V8 JavaScript!
Compiled languages like Go and Rust will outperform that, but again, not by a huge amount.
The results below are from my personal M1 arm64 running macOS.
$ python tests/bench_shortscale.py
50000 calls, 5000000 bytes, 1264 ns/call
100000 calls, 10000000 bytes, 1216 ns/call
200000 calls, 20000000 bytes, 1216 ns/call
$ python tests/bench_shortscale.py
50000 calls, 5000000 bytes, 1811 ns/call
100000 calls, 10000000 bytes, 1808 ns/call
200000 calls, 20000000 bytes, 1809 ns/call
$ node test/bench.js
20000 calls, 2000000 bytes, 796 ns/call
20000 calls, 2000000 bytes, 790 ns/call
20000 calls, 2000000 bytes, 797 ns/call
$ go test -bench . -benchmem
BenchmarkShortscale-8 4227788 252.0 ns/op 248 B/op 5 allocs/op
$ cargo bench
running 2 tests
test a_shortscale ... bench: 182 ns/iter (+/- 3)
test b_shortscale_string_writer_no_alloc ... bench: 63 ns/iter (+/- 2)
Open your browser on https://pyodide.org/en/stable/console.html and paste the following python commands into the python REPL, line by line.
import micropip
await micropip.install("shortscale")
import shortscale
shortscale.shortscale(0xffff0000)
shortscale.bench_shortscale()
It looks like Python in WASM in the browser is only 2 to 3 times slower than native CPython. Amazing!
GitHub shows the output of Jupyter notebook (.ipynb) files in your browser
https://github.com/jldec/shortscale-py/blob/main/shortscale.ipynb
You can also open the notebook from GitHub in a Google Colaboratory environment
https://colab.research.google.com/github/jldec/shortscale-py/blob/main/shortscale.ipynb
Keep on learning
🚀
{ "path": "/blog/getting-started-with-python-packaging", "attrs": { "alias": "/getting-started-with-python", "title": "Getting started with Python Packaging", "splash": { "image": "/images/rockford-office.jpg" }, "date": "2023-02-22", "layout": "BlogPostLayout", "excerpt": "From zero to publishing a Python module on pypi.org." }, "md": "# Getting started with Python Packaging\n\nIn preparation for using a lot more Python, I decided to refresh my Python knowedge and publish my first Python module at https://pypi.org/project/shortscale/.\n\nSome readers may recognize [shortscale](forays-from-node-to-rust) from earlier explorations in [JavaScript](https://github.com/jldec/shortscale), [Rust](https://github.com/jldec/shortscale-rs), and [Go](https://github.com/jldec/shortscale-go).\n\nThis post covers the following steps:\n\n1. Install Python on macOS\n2. Write the skeleton code, with just a one-line function.\n3. Build and publish the incomplete [v0.1](https://github.com/jldec/shortscale-py/tree/bb9b026b9097ce9c601e632a9d1f74a7da6adf29) module.\n4. Complete the logic [v1.0.0](https://github.com/jldec/shortscale-py/commit/6ab4a4b541590a60ebe2944473094465ba8f14f5).\n5. Benchmarks\n6. Python in the browser\n7. Jupyter notebooks\n\n## Install python v3.10 (the hard way)\n\nInstalling Python on macOS is easiest with [the official installer](https://www.python.org/downloads/macos/) or [with homebrew](https://realPython.com/installing-python/#how-to-install-from-homebrew). \n\nI wanted a way to switch between Python versions, so I followed the instructions for [pyenv](https://github.com/pyenv/pyenv). \n\nNOTE: This does a full local build of CPython, and requires dependencies different from the macOS command line tools.\n\n```\n# 1. Install pyenv \n# from https://github.com/pyenv/pyenv#set-up-your-shell-environment-for-pyenv\ngit clone https://github.com/pyenv/pyenv.git $HOME/.pyenv\nexport PYENV_ROOT=\"$HOME/.pyenv\"\nexport PATH=\"$PYENV_ROOT/bin:$PATH\"\neval \"$(pyenv init --path)\"\n\n# 2. Fix dependencies for macOS \n# from https://github.com/pyenv/pyenv/wiki#suggested-build-environment\nbrew install openssl readline sqlite3 xz zlib tcl-tk\n\n# 3. After the brew install, fix LDFLAGS, CPPFLAGS and add tcl-tk/bin onto PATH \nexport LDFLAGS=\"$LDFLAGS -L$HOME/homebrew/opt/openssl@3/lib -L$HOME/homebrew/opt/readline/lib -L$HOME/homebrew/opt/sqlite/lib -L$HOME/homebrew/opt/zlib/lib -L$HOME/homebrew/opt/tcl-tk/lib -L$HOME/homebrew/opt/openssl@3/lib -L$HOME/homebrew/opt/readline/lib -L$HOME/homebrew/opt/sqlite/lib -L$HOME/homebrew/opt/zlib/lib -L$HOME/homebrew/opt/tcl-tk/lib\"\nexport CPPFLAGS=\"$CPPFLAGS -I$HOME/homebrew/opt/openssl@3/include -I$HOME/homebrew/opt/readline/include -I$HOME/homebrew/opt/sqlite/include -I$HOME/homebrew/opt/zlib/include -I$HOME/homebrew/opt/tcl-tk/include -I$HOME/homebrew/opt/openssl@3/include -I$HOME/homebrew/opt/readline/include -I$HOME/homebrew/opt/sqlite/include -I$HOME/homebrew/opt/zlib/include -I$HOME/homebrew/opt/tcl-tk/include\"\nexport PATH=$HOME/homebrew/opt/tcl-tk/bin:$PATH\n\n# 4. Use pyenv to build and install python v3.10 and make it the global default\npyenv install 3.10\npyenv global 3.10\n\n# Point to the installed version in .bash_profile (instead of depending on the pyenv shim)\nexport PATH=$HOME/.pyenv/versions/3.10.9/bin:$PATH\n```\n\n## Virtual environments and pip\n\nPython [modules](https://docs.python.org/3/tutorial/modules.html) and their dependencies can be installed from [pypi.org](https://pypi.org/) using `pip install`.\n\nConfiguring a [virtual environment](https://docs.python.org/3/library/venv.html) will isolate modules under a `.venv` directory, which is easy to clean up, rather than installing everything globally.\n\nI created a venv under my home directory.\n\n```sh\npython3 -m venv ~/.venv\n```\n\nInstead of \"activating\" the venv, which changes the prompt, I prepended the .venv/bin directory onto my PATH.\n\n```\nexport PATH=$HOME/.venv/bin:$PATH\n```\n\n## Create a new module called shortscale\n\nFirst I wrote a skeleton `shortscale` function which just returns a string with the input.\n\nThe rest of the code is boilerplate, to make the function callable on the command line. Passing base=0 to [int()](https://docs.python.org/3/library/functions.html#int) enables numeric literal input with different bases. \n\n#### shortscale.py\n```py\n\"\"\"English conversion from number to string\"\"\"\nimport sys\n\n__version__ = \"0.1.0\"\n\ndef shortscale(num: int) -> str:\n return '{} ({} bits)'.format(num, num.bit_length())\n\ndef main():\n if len(sys.argv) < 2:\n print ('Usage: shortscale num')\n sys.exit(1)\n\n print(shortscale(int(sys.argv[1],0)))\n sys.exit(0)\n\nif __name__ == '__main__':\n main()\n\n```\n\nThe output looks like this:\n\n```sh\n$ python shortscale.py 0x42\n66 (7 bits)\n```\n\nNext, I built and published this incomplete [v0.1](https://github.com/jldec/shortscale-py/tree/bb9b026b9097ce9c601e632a9d1f74a7da6adf29) shortscale module.\n\nUnlike the npm JavaScript ecosystem, you can't just use pip to publish a module to the pypi repository. There are [different build tools](https://packaging.python.org/en/latest/tutorials/packaging-projects/#creating-pyproject-toml) to choose from.\n\nI chose `setuptools` because it appears to be the recommended tool, and shows what it's doing. This meant installing [build](https://pypi.org/project/build/) and [twine](https://pypi.org/project/build/). \n\nPython packages are described in a `pyproject.toml`. Note that project.scripts points to the CLI entrypoint at main().\n\n#### pyproject.toml\n```toml\n[project]\nname = \"shortscale\"\ndescription = \"English conversion from number to string\"\nauthors = [{name = \"Jürgen Leschner\", email = \"jldec@users.noreply.github.com\"}]\nreadme = \"README.md\"\nlicense = {file = \"LICENSE\"}\nclassifiers = [\"License :: OSI Approved :: MIT License\"]\ndynamic = [\"version\"]\n\n[project.urls]\nHome = \"https://github.com/jldec/shortscale-py\"\n\n[project.scripts]\nshortscale = \"shortscale:main\"\n\n[build-system]\nrequires = [\"setuptools>=61.0\"]\nbuild-backend = \"setuptools.build_meta\"\n\n[tool.setuptools.dynamic]\nversion = {attr = \"shortscale.__version__\"}\n```\n\n### Build the module\nThe build tool creates 2 module bundles (source and runnable code) in the ./dist directory.\n\n```\n$ python -m build\n...\nSuccessfully built shortscale-0.1.0.tar.gz and shortscale-0.1.0-py3-none-any.whl\n```\n\n### Publish to pypi.org\n```\n$ python -m twine upload dist/*\nUploading distributions to https://upload.pypi.org/legacy/\nUploading shortscale-0.1.0-py3-none-any.whl\nUploading shortscale-0.1.0.tar.gz\n...\nView at:\nhttps://pypi.org/project/shortscale/0.1.0/\n```\n\n## Install and run in a venv\n\nThe moment of truth. Install the module in a new venv, and invoke it.\n\n```sh\n$ mkdir test\n$ cd test\n$ python -m venv .venv\n$ source .venv/bin/activate\n\n(.venv) $ pip install shortscale\n...\nSuccessfully installed shortscale-0.1.0\n\n(.venv) $ shortscale 0xffffffffffff\n281474976710655 (48 bits)\n\n$ deactivate\n```\n\n## Complete the logic\n\nPython still amazes me with its terseness and readability. \n\nThe first iteration had [3 functions](https://github.com/jldec/shortscale-py/blob/main/shortscale.py), of which the longest had 30 lines with generous spacing. \n\nOne of those functions decomposes a number into powers of 1000.\n\n```py\ndef powers_of_1000(n: int):\n \"\"\"\n Return list of (n, exponent) for each power of 1000.\n List is ordered highest exponent first.\n n = 0 - 999.\n exponent = 0,1,2,3...\n \"\"\"\n p_list = []\n exponent = 0\n while n > 0:\n p_list.insert(0, (n % 1000, exponent))\n n = n // 1000\n exponent += 1\n\n return p_list\n```\n\nPlaying aound in a Jupyter notebook, I was able to eliminate the extra function (and the list which it returns), simply by reversing the order of building the shortscale output.\n\n![Screenshot of a Jupyter notebook in VS Code exploring shortscale](/images/jupyter-notebook-vs-code.png)\n\nUsing a Jupyter environment in VS Code is a clear win. The [result](https://github.com/jldec/shortscale-py/pull/3) was simpler and faster.\n\n\n## Testing\nThere is nice support for Python testing and debugging in VS Code. \n\nThe [function](https://github.com/jldec/shortscale-py/blob/main/shortscale/tests/test_shortscale.py) to run unit tests took just 3 lines. \n\n![Screenshot of VS Code Test integration for pytest](/images/python-test-vscode.png)\n\n## Benchmarks\n\nI was pleased with the [benchmarks](https://github.com/jldec/shortscale-py/blob/main/tests/bench_shortscale.py) as well. For this string manipulation micro-benchmark, CPython 3.11 is only 1.5x slower than V8 JavaScript! \n\nCompiled languages like Go and Rust will outperform that, but again, not by a huge amount. \n\nThe results below are from my personal M1 arm64 running macOS.\n\n### Python\n\n#### Python v3.11.2\n```\n$ python tests/bench_shortscale.py\n\n 50000 calls, 5000000 bytes, 1264 ns/call\n100000 calls, 10000000 bytes, 1216 ns/call\n200000 calls, 20000000 bytes, 1216 ns/call\n```\n\n#### Python v3.10.9\n```\n$ python tests/bench_shortscale.py\n\n 50000 calls, 5000000 bytes, 1811 ns/call\n100000 calls, 10000000 bytes, 1808 ns/call\n200000 calls, 20000000 bytes, 1809 ns/call\n```\n\n### Javascript\n```\n$ node test/bench.js\n\n20000 calls, 2000000 bytes, 796 ns/call\n20000 calls, 2000000 bytes, 790 ns/call\n20000 calls, 2000000 bytes, 797 ns/call\n```\n\n### Go\n```\n$ go test -bench . -benchmem\n\nBenchmarkShortscale-8 \t 4227788\t 252.0 ns/op\t 248 B/op\t 5 allocs/op\n```\n\n### Rust\n```\n$ cargo bench\n\nrunning 2 tests\ntest a_shortscale ... bench: 182 ns/iter (+/- 3)\ntest b_shortscale_string_writer_no_alloc ... bench: 63 ns/iter (+/- 2)\n```\n\n## Let's run shortscale in the browser\n\nOpen your browser on https://pyodide.org/en/stable/console.html and paste the following python commands into the python REPL, line by line.\n\n```python\nimport micropip\nawait micropip.install(\"shortscale\")\nimport shortscale\n\nshortscale.shortscale(0xffff0000)\n\nshortscale.bench_shortscale()\n```\n\n![Screenshot of browser at https://pyodide.org/en/stable/console.html running shortscale](/images/pyodide-shortscale.png)\n\nIt looks like Python in WASM in the browser is only 2 to 3 times slower than native CPython. Amazing!\n\n## Jupyter notebooks on GitHub\n\nGitHub shows the output of Jupyter notebook (.ipynb) files in your browser\n\nhttps://github.com/jldec/shortscale-py/blob/main/shortscale.ipynb\n\n![Screenshot of https://github.com/jldec/shortscale-py/blob/main/shortscale.ipynb](/images/shortscale-notebook-github.png)\n\n## Google Colaboratory\n\nYou can also open the [notebook](https://github.com/jldec/shortscale-py/blob/main/shortscale.ipynb) from GitHub in a [Google Colaboratory](https://colab.research.google.com/github/jldec/shortscale-py/blob/main/shortscale.ipynb) environment\n\nhttps://colab.research.google.com/github/jldec/shortscale-py/blob/main/shortscale.ipynb\n\n![Screenshot of Google Colaboratory running at https://colab.research.google.com/github/jldec/shortscale-py/blob/main/shortscale.ipynb](/images/shortscale-on-google-colaboratory.png)\n\n>> Keep on learning \n>> 🚀\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n", "html": "<h1>Getting started with Python Packaging</h1>\n<p>In preparation for using a lot more Python, I decided to refresh my Python knowedge and publish my first Python module at <a href=\"https://pypi.org/project/shortscale/\">https://pypi.org/project/shortscale/</a>.</p>\n<p>Some readers may recognize <a href=\"forays-from-node-to-rust\">shortscale</a> from earlier explorations in <a href=\"https://github.com/jldec/shortscale\">JavaScript</a>, <a href=\"https://github.com/jldec/shortscale-rs\">Rust</a>, and <a href=\"https://github.com/jldec/shortscale-go\">Go</a>.</p>\n<p>This post covers the following steps:</p>\n<ol>\n<li>Install Python on macOS</li>\n<li>Write the skeleton code, with just a one-line function.</li>\n<li>Build and publish the incomplete <a href=\"https://github.com/jldec/shortscale-py/tree/bb9b026b9097ce9c601e632a9d1f74a7da6adf29\">v0.1</a> module.</li>\n<li>Complete the logic <a href=\"https://github.com/jldec/shortscale-py/commit/6ab4a4b541590a60ebe2944473094465ba8f14f5\">v1.0.0</a>.</li>\n<li>Benchmarks</li>\n<li>Python in the browser</li>\n<li>Jupyter notebooks</li>\n</ol>\n<h2>Install python v3.10 (the hard way)</h2>\n<p>Installing Python on macOS is easiest with <a href=\"https://www.python.org/downloads/macos/\">the official installer</a> or <a href=\"https://realPython.com/installing-python/#how-to-install-from-homebrew\">with homebrew</a>.</p>\n<p>I wanted a way to switch between Python versions, so I followed the instructions for <a href=\"https://github.com/pyenv/pyenv\">pyenv</a>.</p>\n<p>NOTE: This does a full local build of CPython, and requires dependencies different from the macOS command line tools.</p>\n<pre><code># 1. Install pyenv \n# from https://github.com/pyenv/pyenv#set-up-your-shell-environment-for-pyenv\ngit clone https://github.com/pyenv/pyenv.git $HOME/.pyenv\nexport PYENV_ROOT="$HOME/.pyenv"\nexport PATH="$PYENV_ROOT/bin:$PATH"\neval "$(pyenv init --path)"\n\n# 2. Fix dependencies for macOS \n# from https://github.com/pyenv/pyenv/wiki#suggested-build-environment\nbrew install openssl readline sqlite3 xz zlib tcl-tk\n\n# 3. After the brew install, fix LDFLAGS, CPPFLAGS and add tcl-tk/bin onto PATH \nexport LDFLAGS="$LDFLAGS -L$HOME/homebrew/opt/openssl@3/lib -L$HOME/homebrew/opt/readline/lib -L$HOME/homebrew/opt/sqlite/lib -L$HOME/homebrew/opt/zlib/lib -L$HOME/homebrew/opt/tcl-tk/lib -L$HOME/homebrew/opt/openssl@3/lib -L$HOME/homebrew/opt/readline/lib -L$HOME/homebrew/opt/sqlite/lib -L$HOME/homebrew/opt/zlib/lib -L$HOME/homebrew/opt/tcl-tk/lib"\nexport CPPFLAGS="$CPPFLAGS -I$HOME/homebrew/opt/openssl@3/include -I$HOME/homebrew/opt/readline/include -I$HOME/homebrew/opt/sqlite/include -I$HOME/homebrew/opt/zlib/include -I$HOME/homebrew/opt/tcl-tk/include -I$HOME/homebrew/opt/openssl@3/include -I$HOME/homebrew/opt/readline/include -I$HOME/homebrew/opt/sqlite/include -I$HOME/homebrew/opt/zlib/include -I$HOME/homebrew/opt/tcl-tk/include"\nexport PATH=$HOME/homebrew/opt/tcl-tk/bin:$PATH\n\n# 4. Use pyenv to build and install python v3.10 and make it the global default\npyenv install 3.10\npyenv global 3.10\n\n# Point to the installed version in .bash_profile (instead of depending on the pyenv shim)\nexport PATH=$HOME/.pyenv/versions/3.10.9/bin:$PATH\n</code></pre>\n<h2>Virtual environments and pip</h2>\n<p>Python <a href=\"https://docs.python.org/3/tutorial/modules.html\">modules</a> and their dependencies can be installed from <a href=\"https://pypi.org/\">pypi.org</a> using <code>pip install</code>.</p>\n<p>Configuring a <a href=\"https://docs.python.org/3/library/venv.html\">virtual environment</a> will isolate modules under a <code>.venv</code> directory, which is easy to clean up, rather than installing everything globally.</p>\n<p>I created a venv under my home directory.</p>\n<pre><code class=\"language-sh\">python3 -m venv ~/.venv\n</code></pre>\n<p>Instead of "activating" the venv, which changes the prompt, I prepended the .venv/bin directory onto my PATH.</p>\n<pre><code>export PATH=$HOME/.venv/bin:$PATH\n</code></pre>\n<h2>Create a new module called shortscale</h2>\n<p>First I wrote a skeleton <code>shortscale</code> function which just returns a string with the input.</p>\n<p>The rest of the code is boilerplate, to make the function callable on the command line. Passing base=0 to <a href=\"https://docs.python.org/3/library/functions.html#int\">int()</a> enables numeric literal input with different bases.</p>\n<h4><a href=\"http://shortscale.py\">shortscale.py</a></h4>\n<pre><code class=\"language-py\">"""English conversion from number to string"""\nimport sys\n\n__version__ = "0.1.0"\n\ndef shortscale(num: int) -> str:\n return '{} ({} bits)'.format(num, num.bit_length())\n\ndef main():\n if len(sys.argv) < 2:\n print ('Usage: shortscale num')\n sys.exit(1)\n\n print(shortscale(int(sys.argv[1],0)))\n sys.exit(0)\n\nif __name__ == '__main__':\n main()\n\n</code></pre>\n<p>The output looks like this:</p>\n<pre><code class=\"language-sh\">$ python shortscale.py 0x42\n66 (7 bits)\n</code></pre>\n<p>Next, I built and published this incomplete <a href=\"https://github.com/jldec/shortscale-py/tree/bb9b026b9097ce9c601e632a9d1f74a7da6adf29\">v0.1</a> shortscale module.</p>\n<p>Unlike the npm JavaScript ecosystem, you can't just use pip to publish a module to the pypi repository. There are <a href=\"https://packaging.python.org/en/latest/tutorials/packaging-projects/#creating-pyproject-toml\">different build tools</a> to choose from.</p>\n<p>I chose <code>setuptools</code> because it appears to be the recommended tool, and shows what it's doing. This meant installing <a href=\"https://pypi.org/project/build/\">build</a> and <a href=\"https://pypi.org/project/build/\">twine</a>.</p>\n<p>Python packages are described in a <code>pyproject.toml</code>. Note that project.scripts points to the CLI entrypoint at main().</p>\n<h4>pyproject.toml</h4>\n<pre><code class=\"language-toml\">[project]\nname = "shortscale"\ndescription = "English conversion from number to string"\nauthors = [{name = "Jürgen Leschner", email = "jldec@users.noreply.github.com"}]\nreadme = "README.md"\nlicense = {file = "LICENSE"}\nclassifiers = ["License :: OSI Approved :: MIT License"]\ndynamic = ["version"]\n\n[project.urls]\nHome = "https://github.com/jldec/shortscale-py"\n\n[project.scripts]\nshortscale = "shortscale:main"\n\n[build-system]\nrequires = ["setuptools>=61.0"]\nbuild-backend = "setuptools.build_meta"\n\n[tool.setuptools.dynamic]\nversion = {attr = "shortscale.__version__"}\n</code></pre>\n<h3>Build the module</h3>\n<p>The build tool creates 2 module bundles (source and runnable code) in the ./dist directory.</p>\n<pre><code>$ python -m build\n...\nSuccessfully built shortscale-0.1.0.tar.gz and shortscale-0.1.0-py3-none-any.whl\n</code></pre>\n<h3>Publish to <a href=\"http://pypi.org\">pypi.org</a></h3>\n<pre><code>$ python -m twine upload dist/*\nUploading distributions to https://upload.pypi.org/legacy/\nUploading shortscale-0.1.0-py3-none-any.whl\nUploading shortscale-0.1.0.tar.gz\n...\nView at:\nhttps://pypi.org/project/shortscale/0.1.0/\n</code></pre>\n<h2>Install and run in a venv</h2>\n<p>The moment of truth. Install the module in a new venv, and invoke it.</p>\n<pre><code class=\"language-sh\">$ mkdir test\n$ cd test\n$ python -m venv .venv\n$ source .venv/bin/activate\n\n(.venv) $ pip install shortscale\n...\nSuccessfully installed shortscale-0.1.0\n\n(.venv) $ shortscale 0xffffffffffff\n281474976710655 (48 bits)\n\n$ deactivate\n</code></pre>\n<h2>Complete the logic</h2>\n<p>Python still amazes me with its terseness and readability.</p>\n<p>The first iteration had <a href=\"https://github.com/jldec/shortscale-py/blob/main/shortscale.py\">3 functions</a>, of which the longest had 30 lines with generous spacing.</p>\n<p>One of those functions decomposes a number into powers of 1000.</p>\n<pre><code class=\"language-py\">def powers_of_1000(n: int):\n """\n Return list of (n, exponent) for each power of 1000.\n List is ordered highest exponent first.\n n = 0 - 999.\n exponent = 0,1,2,3...\n """\n p_list = []\n exponent = 0\n while n > 0:\n p_list.insert(0, (n % 1000, exponent))\n n = n // 1000\n exponent += 1\n\n return p_list\n</code></pre>\n<p>Playing aound in a Jupyter notebook, I was able to eliminate the extra function (and the list which it returns), simply by reversing the order of building the shortscale output.</p>\n<p><img src=\"/images/jupyter-notebook-vs-code.png\" alt=\"Screenshot of a Jupyter notebook in VS Code exploring shortscale\"></p>\n<p>Using a Jupyter environment in VS Code is a clear win. The <a href=\"https://github.com/jldec/shortscale-py/pull/3\">result</a> was simpler and faster.</p>\n<h2>Testing</h2>\n<p>There is nice support for Python testing and debugging in VS Code.</p>\n<p>The <a href=\"https://github.com/jldec/shortscale-py/blob/main/shortscale/tests/test_shortscale.py\">function</a> to run unit tests took just 3 lines.</p>\n<p><img src=\"/images/python-test-vscode.png\" alt=\"Screenshot of VS Code Test integration for pytest\"></p>\n<h2>Benchmarks</h2>\n<p>I was pleased with the <a href=\"https://github.com/jldec/shortscale-py/blob/main/tests/bench_shortscale.py\">benchmarks</a> as well. For this string manipulation micro-benchmark, CPython 3.11 is only 1.5x slower than V8 JavaScript!</p>\n<p>Compiled languages like Go and Rust will outperform that, but again, not by a huge amount.</p>\n<p>The results below are from my personal M1 arm64 running macOS.</p>\n<h3>Python</h3>\n<h4>Python v3.11.2</h4>\n<pre><code>$ python tests/bench_shortscale.py\n\n 50000 calls, 5000000 bytes, 1264 ns/call\n100000 calls, 10000000 bytes, 1216 ns/call\n200000 calls, 20000000 bytes, 1216 ns/call\n</code></pre>\n<h4>Python v3.10.9</h4>\n<pre><code>$ python tests/bench_shortscale.py\n\n 50000 calls, 5000000 bytes, 1811 ns/call\n100000 calls, 10000000 bytes, 1808 ns/call\n200000 calls, 20000000 bytes, 1809 ns/call\n</code></pre>\n<h3>Javascript</h3>\n<pre><code>$ node test/bench.js\n\n20000 calls, 2000000 bytes, 796 ns/call\n20000 calls, 2000000 bytes, 790 ns/call\n20000 calls, 2000000 bytes, 797 ns/call\n</code></pre>\n<h3>Go</h3>\n<pre><code>$ go test -bench . -benchmem\n\nBenchmarkShortscale-8 \t 4227788\t 252.0 ns/op\t 248 B/op\t 5 allocs/op\n</code></pre>\n<h3>Rust</h3>\n<pre><code>$ cargo bench\n\nrunning 2 tests\ntest a_shortscale ... bench: 182 ns/iter (+/- 3)\ntest b_shortscale_string_writer_no_alloc ... bench: 63 ns/iter (+/- 2)\n</code></pre>\n<h2>Let's run shortscale in the browser</h2>\n<p>Open your browser on <a href=\"https://pyodide.org/en/stable/console.html\">https://pyodide.org/en/stable/console.html</a> and paste the following python commands into the python REPL, line by line.</p>\n<pre><code class=\"language-python\">import micropip\nawait micropip.install("shortscale")\nimport shortscale\n\nshortscale.shortscale(0xffff0000)\n\nshortscale.bench_shortscale()\n</code></pre>\n<p><img src=\"/images/pyodide-shortscale.png\" alt=\"Screenshot of browser at https://pyodide.org/en/stable/console.html running shortscale\"></p>\n<p>It looks like Python in WASM in the browser is only 2 to 3 times slower than native CPython. Amazing!</p>\n<h2>Jupyter notebooks on GitHub</h2>\n<p>GitHub shows the output of Jupyter notebook (.ipynb) files in your browser</p>\n<p><a href=\"https://github.com/jldec/shortscale-py/blob/main/shortscale.ipynb\">https://github.com/jldec/shortscale-py/blob/main/shortscale.ipynb</a></p>\n<p><img src=\"/images/shortscale-notebook-github.png\" alt=\"Screenshot of https://github.com/jldec/shortscale-py/blob/main/shortscale.ipynb\"></p>\n<h2>Google Colaboratory</h2>\n<p>You can also open the <a href=\"https://github.com/jldec/shortscale-py/blob/main/shortscale.ipynb\">notebook</a> from GitHub in a <a href=\"https://colab.research.google.com/github/jldec/shortscale-py/blob/main/shortscale.ipynb\">Google Colaboratory</a> environment</p>\n<p><a href=\"https://colab.research.google.com/github/jldec/shortscale-py/blob/main/shortscale.ipynb\">https://colab.research.google.com/github/jldec/shortscale-py/blob/main/shortscale.ipynb</a></p>\n<p><img src=\"/images/shortscale-on-google-colaboratory.png\" alt=\"Screenshot of Google Colaboratory running at https://colab.research.google.com/github/jldec/shortscale-py/blob/main/shortscale.ipynb\"></p>\n<blockquote>\n<blockquote>\n<p>Keep on learning<br>\n🚀</p>\n</blockquote>\n</blockquote>\n" }