What's New In Tohil 4.0

tclobj default return

This is a biggie. Many Tohil functions accept a to= argument where you can specify a Python data type to convert a tcl object returned from doing a call or accessing tcl data. You can set a return type of str, int, bool, float, list, set, dict, tuple, tohil.tclobj or tohil.tcldict.

Prior to Tohil 4, if you didn’t set a to= return type, the default return type was string, str. This seemed perfectly reasonable; after all, in Tcl, despite it having internal objects and maintaining in them a cache of a conversion to a data type such as integer, list, etc, in Tcl “every value is a string.”

However, as we have enhanced and extended Tohil’s tclobj type, it has become ever easier to use tclobjs directly from Python with no funny business. You can get a tclobj’s string representation with str(), integer with int(), float with float(), list with list(), and others. You can use Python list notation to access and manipulate elements of tclobjs when they contain lists, can iterate over them, etc.

Since Tohil’s tclobj type implements Python’s number protocol, if tclobjs contain numbers, they can be used in calculations without conversion via int() and float().

Consequently starting in Tohil 4, the default to return is now tohil.tclobj. In our experience, and a little bit to our surprise, most Python code that uses Tohil will “just work” without modifications.

If, though, for instance, you didn’t specify a default return and then knowing you would get a str invoked string methods on the str that was returned, you’ll probably get an error because the tclobj doesn’t implement all of the str datatype’s methods. In this case, adding a to=str to the Tohil call will be sufficient to get your code working under Tohil 4.

Python Subinterpreter Support

Full Python subinterpreter support!

First, starting with version 4, Tohil properly supports multi-phase init, meaning that multiple Python interpreters (the Python interpreter and any subinterpreters) can import tohil and they will get their own instance of Tohil, so there is no “crosstalk” between the interpreters.

Second, Tohil recognizes when a second Tcl interpreter within the same process has done a package require tohil and will create and exclusively interact with a separate, distinct Python subinterpreter for each corresponding Tcl interpreter.

Say for instance you create a new Tcl interpreter from Tcl, using something like set interp [interp create] and then do $interp eval "package require tohil", that second interpreter doing the package require causes a new Python subinterpreter to be created and initialized.

And it works great.

When any of the Tcl interpreters exercises their Python interpreter, Tohil will automatically switch Python’s executing interpreter to that interpreter (swap its thread state), if needed.

Upon deletion of a Tcl interpreter, if there is an attached Python subinterpreter, it is deleted as well.

Implementation Note: This was pretty tricky, because we previously had global variables, in particular one pointing to the Tcl interpreter. We had to figure out ways to stash the pointer to the Tcl interpreter in Python using C such that we could find it later when we didn’t have control over how we were called, for example we are being called from Python with to do some Python thing, you only get what it calls you with. So we stashed the interpreter pointer in a capsule in Tohil’s Python data types’ dictionaries and in __main__’s dictionary. It turned out really nice.

Support for Separate Virtual Interpreters in Rivet

A nice bit of fallout from the above, if you’re running the Apache webserver with the Apache Rivet module installed and running in the mode where different virtual hosts run in separate Tcl interpreters, known as separate virtual interpreters, each vhost that does a package require tohil will get its own Python subinterpreter, isolating Python between the vhosts just as Tcl is.

A function can now be specified in a to= arg

The to= argument to a Tohil function such as tohil.eval, tohil.call, etc, has until now been required to specify a Python data type such as int, float, str, tohil.tclobj, etc. It can now also be specified as a callable function.

If the to argument is not a recognized data type but is a callable function, Tohil will call that function with one argument, a tclobj object containing the object to be returned, and it is expected that the function will manipulate the object in some way and then return a result. Whatever the function returns is what the relevant tohil function will return.

This provides an additional way for a Tohil developer to customize the return of some Tcl activity in order to make it more standard and readily useful to the Python caller.

User-Facing Behavior Changes

• When Tcl is the parent, package require tohil will, in addition to initializing Python, automatically import tohil on the Python side.

Internal Changes

• When Tcl is the parent and Tohil initializes Python from scratch, we use PyInitializeEx(0) instead of Py_Initialize to prevent Python from registering signal handlers. (Signal handling probably ought to be Tcl’s business under this circumstance.)

• Internal code refactoring and cleanup.

Documentation Improvements

• Greatly improved documentation in Python-standard format.

• Makefile and docs for building the docs.

• make serve target to serve the docs via http (for devs)