API vs ABI 🧩

Ever installed a compiled Python package and hit a segfault instead of a nice traceback? 🫠

That's the ABI.

An API defines how code talks to code at the source level:

  • function names,
  • signatures,
  • what you import and call.

An ABI (Application Binary Interface) defines how already-compiled code talks to other compiled code:

  • how structs are laid out in memory,
  • how arguments get passed,
  • how objects are reference-counted.

CPython has its own ABI, and its internal structs (like PyObject) can shift shape between minor versions - which is why a C extension built for Python 3.11 often can't be used in 3.12 without a rebuild.

Why this matters if you build or ship packages πŸ› οΈ

πŸ” By default, compiled extensions need to be rebuilt per Python version - unless you opt into the limited API / stable ABI via Py_LIMITED_API, trading some features for abi3 wheels that work across versions (PEP 384).

🏷️ Wheel tags like cp311-cp311 vs cp311-abi3 literally spell out which ABI contract the binary is promising to honor.

πŸ’₯ If you get the ABI wrong, users don't get a friendly Python exception.

πŸ“š For a proper deep dive, check out this article on the Python ABI and abi3t - it covers the free-threaded build and what abi3t means for the stable ABI going forward.

πŸ’‘ Takeaway: If you distribute C extensions, rebuild per Python version or use abi3 - otherwise the ABI will break in a segfault, not an exception.


"Compile me!" πŸ«–

Deciding whether to re.compile a regex is a bit like Alice pondering the bottle labeled β€œDrink me!” - it's about picking the best option for the situation.

  • One-off search: A small sip with minimal effect and negligible overhead. If you only plan to run the regex once, there's no benefit to compiling it - Python will take care of that automatically when re.search is invoked.
python
import re

if re.search(r"\bwhite\s+rabbit\b", "Alice chased the white rabbit into the hole."):
    print("Found the white rabbit!")
  • Repeated use: A generous sip with a noticeable impact. Each call to re.search inside a loop would recompile the regex, creating overhead. When compiling the pattern once, multiple searches can reuse it and save time.
python
import re

texts = [
    "Alice quickly ran after the white rabbit as it disappeared behind the bushes.",
    "The white rabbit looked at its pocket watch anxiously before hopping away."
]

pattern = re.compile(r"\bwhite\s+rabbit\b")

for text in texts:
    if pattern.search(text):
        print("Found the white rabbit!")

βœ… Reuse β†’ compile. ❌ One-off β†’ skip it.

πŸ’‘ Takeaway: For repeated use, compile your regex - otherwise it'll end up like the white rabbit, frantically rushing: "Oh dear, oh dear, I shall be late β€” I got a match!"


Non-public 🚫 vs protected πŸ›‘οΈ

⚠️ Disclaimer: If you're coming from an OOP background, you may find the following content disturbing. Read at your own risk.

Python doesn't do strict access control, a single underscore means private by convention, while a double underscore triggers name mangling to protect attributes from being accidentally overridden in subclasses.

But what happens if a subclass defines its own double underscore attribute? How does a single-underscore variable differ from a double underscore variable in practice?

For example, the base Metric class has __storage and a _mean:

python
class Metric:
    def __init__(self):
        self.__storage = []  # double underscore β†’ protected
        self._mean = None  # single underscore β†’ private

    def add(self, num):
        self.__storage.append(num)
        self._mean = None

    def mean(self):
        if self._mean is None:
            self._mean = sum(self.__storage) / len(self.__storage)
        return self._mean

m = Metric()
m.add(5)
m.add(8)
print("Metric mean:", m.mean())  # 6.5
print("Metric __dict__:", m.__dict__)  # {'_Metric__storage': [5, 8], '_mean': 6.5}

A subclass can define its own double underscore attribute without overwriting the parent:

python
class MaxMetric(Metric):
    def __init__(self):
        super().__init__()
        self.__storage = None  # subclass storage, mangled separately

    def add(self, num):
        super().add(num)
        if self.__storage is None or num > self.__storage:
            self.__storage = num

    def max(self):
        return self.__storage

mm = MaxMetric()
mm.add(3)
mm.add(10)
mm.add(7)
print("MaxMetric mean:", mm.mean())  # 6.666666666666667
print("MaxMetric max:", mm.max())  # 10
print("MaxMetric __dict__:", mm.__dict__)
# {'_Metric__storage': [3, 10, 7], '_mean': 6.666666666666667, '_MaxMetric__storage': 10}

πŸ’‘ Takeaway: Use single underscore for helpers or temporary state, double underscore for critical storage that shouldn't be accidentally overridden.


Walrus operator: assign on the fly πŸ›«

I recently discovered that Python has a neat operator for assigning a value and using it immediately: the := operator, also called the walrus operator.

For example, instead of:

python
value = data.get("key")
if value:
    print(len(value))

You can do:

python
if (value := data.get("key")):
    print(len(value))

Boom πŸ’₯ value is assigned and ready to roll.

πŸ’‘ Takeaway: := = assign inline, use instantly, keep it neat.


The one when TRUNCATE locked me out πŸšͺπŸ”’

While running pytest with MySQL, I kept hitting tests stuck on: Waiting for table metadata lock 😫

Digging in, I found the culprit: my fixtures were using TRUNCATE TABLE to clean test data between runs.

Here's the catch: In MySQL, TRUNCATE is DDL, not DML. It drops/recreates the table under the hood, resets AUTO_INCREMENT, and requires an exclusive metadata lock. If any transaction has touched the table β€” even just a SELECT β€” TRUNCATE will block until that lock is released. In pytest, with long-lived connections, this happened constantly.

βœ… Fix: switched to DELETE FROM table for cleanup. DELETE is DML, transactional, and only takes row locks + short metadata locks. It doesn't reset AUTO_INCREMENT, but it doesn't block other transactions either.

πŸ’‘ Takeaway:: In MySQL tests, prefer DELETE over TRUNCATE unless you can guarantee no open transactions.

πŸ”„ Postgres comparison: TRUNCATE in Postgres is transactional β€” you can roll it back, and it doesn't block in the same way. It still takes stricter locks than DELETE, but because Postgres metadata locking is less rigid, it rarely causes the same β€œhung DDL” issues you see in MySQL.