Is my understanding correct that this would provide version agnostic python bindings? Currently, I am building a version of my bindings separately for each version (e.g. building and linking with python 3.7, 3.8, etc.). While automated, it still makes CI/CD take quite a long time.
As others have said, this has been supported since the limited/stable APIs were introduced. What this adds is a way of implementing a Python extension that can be loaded in (not just compiled for, which is already an improvement!) different Python implementations, namely CPython, Pypy and GraalVM.
But it is very limited. Understandably so, as they don't want to ossify the internal APIs, but it still is so limited that you can't actually build anything just using just that API as far as I know.
I wouldn't recommend ccache (or sccache) in CI unless you really need it. They are not 100% reliable, and any time you save from caching will be more than lost debugging the weird failures you get when they go wrong.
You can't cache based on the file contents alone. You will also need to cache based on all OS/compiler queries/variables/settings that the preprocessor depends on, since the header files might generate completely different content based on what ifdef gets triggered.
And that’s not impossible, just tedious. One tricky (and often unimportant) part is negative dependencies—when the build depends on the fact that a header or library cannot be found in a particular directory on a search path (which happens all the time, if you think about it). As far as I know, no compilers will cooperate with you on this, so build systems that try to get this right have to trace the compiler’s system calls to be sure (Tup does something like this) or completely control and hash absolutely everything that the compiler could possibly see (Nix and IIUC Bazel).
Why are you so skeptical? Think about how it works and then you'll understand that cache invalidation bugs are completely inevitable. Hell, cache invalidation is notoriously difficult to get right even when you aren't building it on top of a complex tool that was never designed for aggressive caching.
It would be interesting to see benchmarks comparing HPy extensions to equivalent Cython/pybind11 implementations in terms of performance and development time.
I'm a little unclear as to how this fits in with libraries like PyBind11 or nanobind? It seems like those libraries would need to be rewritten (or new libraries with the same goals created) in order to use this in the same way?
Yeah, sure, I mean, how many people write C to write an end-user Python module. There's stuff that genuinely wraps C libraries or predates higher level language wrappers, like numpy or matplotlib, but how many new modules are actually themselves written in C?
I would guess also that HPy would replace the includes of `Python.h` that pybind11 et al make in order to bind to CPython, and so existing extensions should be easier to port?
A lot. You don't have to write in C, just use the C-API functions. pybind etc. introduce a whole new set of problems, with new version issues and decreased debug ability.
tangentially related question: is there something as simple as luajit's ffi for python? as in: give it a c header, load the shared library, it simply makes structs usable and functions callable.
It's a no-go at this point, if you want this on MS Windows. CGo on MS Windows uses MinGW, while CPython uses MSVC. It's very hard to make this work due to name mangling.
I.e. you can do this for Python from MSYS2, for example, but not for the one your users will likely have.
Unless it was done at the very beginning, I doubt it would have been even possible because the current C API is the remnant from that very first public version.
python has one of the most fractured development ecosystems of any moderately used language. i’m pretty convinced python is a language that attracts poor development practices and magnifies them due to its flexibility. the people who love it don’t understand the extreme flexibility makes it fragile at scale and are willing to put up with its annoyances in an almost stockholm syndrome way
I think any programming language with a lot of popularity attracts poor development practices. Simply because a lot of programmers don't actually know the underlying processes of what they build. The flip-side of this is that freedom and flexibility also gives you a lot of control. Yes, it's very easy to write bad Python. In fact it's probably one of Python's weaknesses as you point out. If you're going to iterate over a bunch of elements, you probably expect your language standard libraries to do it in an efficient way, and Python doesn't necessarily do that. What you gain by this flexibility (and arguably sometimes poor design) is that it's also possible to write really good Python and tailor it exactly to your needs. I think Python scales rather well in fact. Django is a good example, as it's a massive workhorse for a lot of the web (Instagram still uses their own version of it as one example). It does so sort of anonymously similar to how PHP and Ruby do it outside of the hype circle, but it does it.
One of the advantages Python has, even when it's bad, is that it's often "good enough". 95% of the software which gets written is never really going to need to be extremely efficient. I would argue that in 2024 Go is actually the perfect combination of the good stuff from both Python and C. But those things aren't necessarily easy to get into if you're not familiar with something like memory management, (maybe strict typing?), explicit error handling and the differences between an interpreted and compiled language.
Anyway I don't think Python is anymore annoying than any other language. The freedom it gives you needs to be reigned in and if you don't then you'll end up with a mess. A mess which is probably perfectly fine.
But CPython itself has poor development practices: For about 8 years those in the inner circle can modify anything and pose as experts while brutally squashing criticism.
> most fractured development ecosystems of any moderately used language
Can you elaborate? What's done wrong with Python and right with other "moderately used language" ?
For start, C/C++ doesn't even have an official ecosystem. For Java or Golang, it looks better only because the "ecosystem" does not always include native extensions like cgo or JNI. Once you add them the complexity were no better than Python's
Python .pth files are horrific. Here's an actual .pth file I was dealing with the other day (from Google Cloud Storage) which completely prevents you from overriding the module using PYTHONPATH:
import sys, types, os;has_mfs = sys.version_info > (3, 5);p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('google',));importlib = has_mfs and __import__('importlib.util');has_mfs and __import__('importlib.machinery');m = has_mfs and sys.modules.setdefault('google', importlib.util.module_from_spec(importlib.machinery.PathFinder.find_spec('google', [os.path.dirname(p)])));m = m or sys.modules.setdefault('google', types.ModuleType('google'));mp = (m or []) and m.__dict__.setdefault('__path__',[]);(p not in mp) and mp.append(p)
If .pth files are the worst thing you can find to complain about, Python's doing pretty well. That horrific .pth file in question is better placed as the feet of its creators than the mechanism itself.
It shows that the language is highly dynamic and you can patch anything? The .pth mechanism allows the party controlling the Python installation (site) to run some init code before any user code, basically an rc mechanism. Nothing more, nothing radical. Maybe you’re unhappy with the dynamism, in which case your complaint is misplaced.
You have Anaconda packaging world vs PyPI. You have pyproject.toml for project management, which is not supported by Anaconda or the flagship documentation generation tool: Sphynx. You have half a dozen of package installers, none of them work to the full extent / all have different problems. You have plenty of ways to install Python, all of them suck. You have plenty of ways to do some common tasks, s.a. GUI, Web, automation: and all of them suck in different ways, w/o a hint of unifying link. Similarly, you have an, allegedly, common relational database interface, but most commonly used SQL bindings don't use it. And the list goes on.
As I said, it's only because .so extensions were hard. If every package were pure Python, I would simply copy paste them in my source code `lib` path.
Don't laugh at me, this is called "vendoring" or "static linking" by other languages, and the "requests" famously included a version of urllib3 for quite a while
>the people who love it don’t understand the extreme flexibility makes it fragile at scale and are willing to put up with its annoyances in an almost stockholm syndrome way
The people who love it understand that its extreme flexibility makes it applicable everywhere, while academic purity mostly doesn't work in the real work. They also prioritize getting things done over petty squabbling, but they know how to leverage available tooling where reliability is crucial.
The reason is a popularity not a technical one. It's inevitable to get a diverse interest to improve different parts of ecosystem by different parties.
After cpyext and cffi, this is the third attempt, largely driven by PyPy people, to get a C-API that people want to use.
If they succeed and keep the CPython "leaders" who ruined the development experience and social structure of CPython out of PyPy, PyPy might get interesting. If they don't keep them out, those "leaders" will merrily sink yet another project.
cffi is used to wrap c libraries. Only a masochist would use ctypes to wrap a whole library. While both are technically FFIs, it does not make sense to compare them. From a conceptual perspective, cffi was written to replace the C-API for C modules.
Is my understanding correct that this would provide version agnostic python bindings? Currently, I am building a version of my bindings separately for each version (e.g. building and linking with python 3.7, 3.8, etc.). While automated, it still makes CI/CD take quite a long time.
As others have said, this has been supported since the limited/stable APIs were introduced. What this adds is a way of implementing a Python extension that can be loaded in (not just compiled for, which is already an improvement!) different Python implementations, namely CPython, Pypy and GraalVM.
Cpython also has a limited stable abi and cp3X-abi3 wheels are compatible across multiple versions of Python.
https://docs.python.org/3/c-api/stable.html
But it is very limited. Understandably so, as they don't want to ossify the internal APIs, but it still is so limited that you can't actually build anything just using just that API as far as I know.
You can already build a single wheel as long as you only target cpython, if your needs fit with the limited / stable abi (abi3).
While pypy and graal have API support they don't have abi / abi3 support, so they still have to be built on their own (and per version I think).
I believe so, but it would presumably depend on what features you use.
While automated, it still makes CI/CD take quite a long time
See about using ccache -- https://ccache.dev/
I wouldn't recommend ccache (or sccache) in CI unless you really need it. They are not 100% reliable, and any time you save from caching will be more than lost debugging the weird failures you get when they go wrong.
please provide evidence for this assertion.
You can't cache based on the file contents alone. You will also need to cache based on all OS/compiler queries/variables/settings that the preprocessor depends on, since the header files might generate completely different content based on what ifdef gets triggered.
And that’s not impossible, just tedious. One tricky (and often unimportant) part is negative dependencies—when the build depends on the fact that a header or library cannot be found in a particular directory on a search path (which happens all the time, if you think about it). As far as I know, no compilers will cooperate with you on this, so build systems that try to get this right have to trace the compiler’s system calls to be sure (Tup does something like this) or completely control and hash absolutely everything that the compiler could possibly see (Nix and IIUC Bazel).
Maybe run every build version in their own container?
please read the documentation for ccache -- it works using the output of the preprocessor and optionally, file paths
Why are you so skeptical? Think about how it works and then you'll understand that cache invalidation bugs are completely inevitable. Hell, cache invalidation is notoriously difficult to get right even when you aren't building it on top of a complex tool that was never designed for aggressive caching.
Just search the bugs for "hash":
https://github.com/ccache/ccache/issues?q=is%3Aissue+hash+is...
It would be interesting to see benchmarks comparing HPy extensions to equivalent Cython/pybind11 implementations in terms of performance and development time.
I'm a little unclear as to how this fits in with libraries like PyBind11 or nanobind? It seems like those libraries would need to be rewritten (or new libraries with the same goals created) in order to use this in the same way?
Looks very cool.
How many new extensions are written in C these days? I was under the impression it's mostly things like Boost Python, pybind or PyO3.
PyO3 is bindings to the C API, so if you're using PyO3 you're still using the C API even if you're not actually writing C.
Yeah, sure, I mean, how many people write C to write an end-user Python module. There's stuff that genuinely wraps C libraries or predates higher level language wrappers, like numpy or matplotlib, but how many new modules are actually themselves written in C?
There's also Cython.
I would guess also that HPy would replace the includes of `Python.h` that pybind11 et al make in order to bind to CPython, and so existing extensions should be easier to port?
Quite a lot, for things like simulation code
Less so for general programming.
for C++ 11+, pybind11 > Boost.Python
for C++ 17+, nanobind > pybind11 (both created by the same developer)
">" meaning generally better, as described at https://nanobind.readthedocs.io/en/latest/why.html
A lot. You don't have to write in C, just use the C-API functions. pybind etc. introduce a whole new set of problems, with new version issues and decreased debug ability.
tangentially related question: is there something as simple as luajit's ffi for python? as in: give it a c header, load the shared library, it simply makes structs usable and functions callable.
cffi is closest to what you described.
cppyy does this for C++
Yeah, cffi.
I'm interested in calling go from python, gopy generates python bindings to cgo. Maybe HPy<->cgo would have less overhead.
It's a no-go at this point, if you want this on MS Windows. CGo on MS Windows uses MinGW, while CPython uses MSVC. It's very hard to make this work due to name mangling.
I.e. you can do this for Python from MSYS2, for example, but not for the one your users will likely have.
Imagine how different the Python ecosystem could be, if this was done 20 years ago.
Unless it was done at the very beginning, I doubt it would have been even possible because the current C API is the remnant from that very first public version.
python has one of the most fractured development ecosystems of any moderately used language. i’m pretty convinced python is a language that attracts poor development practices and magnifies them due to its flexibility. the people who love it don’t understand the extreme flexibility makes it fragile at scale and are willing to put up with its annoyances in an almost stockholm syndrome way
I think any programming language with a lot of popularity attracts poor development practices. Simply because a lot of programmers don't actually know the underlying processes of what they build. The flip-side of this is that freedom and flexibility also gives you a lot of control. Yes, it's very easy to write bad Python. In fact it's probably one of Python's weaknesses as you point out. If you're going to iterate over a bunch of elements, you probably expect your language standard libraries to do it in an efficient way, and Python doesn't necessarily do that. What you gain by this flexibility (and arguably sometimes poor design) is that it's also possible to write really good Python and tailor it exactly to your needs. I think Python scales rather well in fact. Django is a good example, as it's a massive workhorse for a lot of the web (Instagram still uses their own version of it as one example). It does so sort of anonymously similar to how PHP and Ruby do it outside of the hype circle, but it does it.
One of the advantages Python has, even when it's bad, is that it's often "good enough". 95% of the software which gets written is never really going to need to be extremely efficient. I would argue that in 2024 Go is actually the perfect combination of the good stuff from both Python and C. But those things aren't necessarily easy to get into if you're not familiar with something like memory management, (maybe strict typing?), explicit error handling and the differences between an interpreted and compiled language.
Anyway I don't think Python is anymore annoying than any other language. The freedom it gives you needs to be reigned in and if you don't then you'll end up with a mess. A mess which is probably perfectly fine.
But CPython itself has poor development practices: For about 8 years those in the inner circle can modify anything and pose as experts while brutally squashing criticism.
psst "reined in" https://www.merriam-webster.com/grammar/do-you-rein-in-or-re...
> most fractured development ecosystems of any moderately used language
Can you elaborate? What's done wrong with Python and right with other "moderately used language" ?
For start, C/C++ doesn't even have an official ecosystem. For Java or Golang, it looks better only because the "ecosystem" does not always include native extensions like cgo or JNI. Once you add them the complexity were no better than Python's
Python .pth files are horrific. Here's an actual .pth file I was dealing with the other day (from Google Cloud Storage) which completely prevents you from overriding the module using PYTHONPATH:
I agree those particular .pth files were horrific.
But python package made by Google were noturously bad. Its awefulness dates back to the GAE days.
If .pth files are the worst thing you can find to complain about, Python's doing pretty well. That horrific .pth file in question is better placed as the feet of its creators than the mechanism itself.
The fact they considered allowing executable code in path lookups shows a certain attitude.
It shows that the language is highly dynamic and you can patch anything? The .pth mechanism allows the party controlling the Python installation (site) to run some init code before any user code, basically an rc mechanism. Nothing more, nothing radical. Maybe you’re unhappy with the dynamism, in which case your complaint is misplaced.
You have Anaconda packaging world vs PyPI. You have pyproject.toml for project management, which is not supported by Anaconda or the flagship documentation generation tool: Sphynx. You have half a dozen of package installers, none of them work to the full extent / all have different problems. You have plenty of ways to install Python, all of them suck. You have plenty of ways to do some common tasks, s.a. GUI, Web, automation: and all of them suck in different ways, w/o a hint of unifying link. Similarly, you have an, allegedly, common relational database interface, but most commonly used SQL bindings don't use it. And the list goes on.
> You have Anaconda packaging world vs PyPI
As I said, it's only because .so extensions were hard. If every package were pure Python, I would simply copy paste them in my source code `lib` path.
Don't laugh at me, this is called "vendoring" or "static linking" by other languages, and the "requests" famously included a version of urllib3 for quite a while
> a language that attracts poor development practices
I agree, but note there’s another way to frame it: “python can be used by people who aren’t professional software developers”.
It’s also fractured because it has such a massive user base that use it for very different applications with very different priorities.
>the people who love it don’t understand the extreme flexibility makes it fragile at scale and are willing to put up with its annoyances in an almost stockholm syndrome way
The people who love it understand that its extreme flexibility makes it applicable everywhere, while academic purity mostly doesn't work in the real work. They also prioritize getting things done over petty squabbling, but they know how to leverage available tooling where reliability is crucial.
(See, I can generalize too)
C/C++ is more fractured.
While Python is fractured, it is nowhere near problems of C ecosystems.
As anyone who has tried to build multi-platform software with C or C++ can easily tell you.
It's almost a relief AIX, Solaris, and HP/UX are either very niche, or going the way of the Dodo.
Would some other language have become just as fragmented if it had gained the same level of popularity across such a broad range of user interests?
The reason is a popularity not a technical one. It's inevitable to get a diverse interest to improve different parts of ecosystem by different parties.
Python with types enforced by CI isn’t too bad. Or did you have something else in mind?
There are only two kinds of languages: the ones people complain about and the ones nobody uses.
it's not 'fractured', it's just fragmented, and it's not necessarily a bad thing, it gives plenty of room for R&D and experimentation
if something doesn't end up working well, you pivot
It would have taken time to do this and consequently Python would have missed the race and some other language would now be #1.
Python missed the race pretty heavily with 2to3 transition and still came out on top.
Survivorship bias. With version 2 they were already at the top.
[dead]
Is this thing “official”?
After cpyext and cffi, this is the third attempt, largely driven by PyPy people, to get a C-API that people want to use.
If they succeed and keep the CPython "leaders" who ruined the development experience and social structure of CPython out of PyPy, PyPy might get interesting. If they don't keep them out, those "leaders" will merrily sink yet another project.
cffi replaces ctypes, which is a completely different thing. cpyext is a reimplementation of the Python C-API, so no attempt at improving the API.
HPy on CPython uses the existing C-API under the hood, so there is zero need to build up some keep someone out...
cffi is used to wrap c libraries. Only a masochist would use ctypes to wrap a whole library. While both are technically FFIs, it does not make sense to compare them. From a conceptual perspective, cffi was written to replace the C-API for C modules.