I vibecoded https://punkx.org/apl/learn.html [1] its good slow entry into gnu-apl, at least it got me to be able to make tictactoe, vibecoded or not, it helped me to start.
I am amazed how quickly APL changed the way I think.
Also strongly recommend watching Aaron Hsu on youtube.
There is no better time to re-learn programming, try APL, Forth, LISP, z80 machine code, UXN TAL, just try new things.
"Men are born soft and supple; dead they are stiff and hard. Plants are born tender and pliant; dead, they are brittle and dry. Thus whoever is stiff and inflexible is a disciple of death. Whoever is soft and yielding is a disciple of life. The hard and stiff will be broken. The soft and supple will prevail."
APL is cool, but that quote hurts my brain. Disregard the funny typo, clearly different situations call for different strategies. For example, it might be great to use LLM slop interactively while learning, but that doesn’t mean it can be used as a finished product.
FYI, your "8‿64 •bit._cast -⟜@ raw" in bqn/util.bqn results in passing to •bit._cast integers in the range 0..255, whereas it expects integers ¯128..127 (and does arbitrary platform-specific things on integers outside that for the int16→int8 narrow); removing the "-⟜@" makes it work properly, and 2.5-5x faster while at it from not going to intermediate int16 (allowing the bitcast to be O(1) by itself). (•FBytes → •file.MapBytes also probably improves things, at the cost of the obvious mmap problem of issues if the file changes, as BQNs arrays are immutable). This is also what caused the problem at https://github.com/ClickHouse/ClickBench/pull/939/changes/40... (there's been some discussion about what to do about invalid •bit._cast before, considering it's rather perf-sensitive; it's the only place in CBQN with such quiet very-wrong results on invalid inputs)
The default CBQN "make o3" on x86-64 also results in it only using SSE2 (utilizing function multiversioning is on the ever-infinite TODO list, though somewhat-low on it considering it's strictly-unnecessary in any specific situation; there's also AVX-512 usage on a branch, but mostly only AVX2 on mainline; and no arm SVE)
That all said, CBQN doesn't currently do any loop fusion, so being significantly-slower for sequences of operations over larger-than-cache arrays would kinda just be expected. BQN also just isn't particularly intended for database work anyway.
(didn't look much at the specific query impls, though "Pair" in utils.bqn is at least an overlong version of "Pair ← ⋈¨"; and some if not all of those Pairs would be better as "≍˘" to avoid nested arrays and ensuing pointer chasing; and, of course, if some of the columns are bools/int8/int16/int32, it'd be beneficial to store & load them as such instead of float64)
BQN & CBQN are absolutely wonderful pieces of code.
L is mac/lin but linux is avx512 only specifically to try to deal with that problem. The compute on compressed algos helps fit more in those cache lines! https://lv1.sh/blog/compute-on-compressed/
What's the problem if a site is vibecoded? We will be seeing it more often, I recon. Let's get used to it. On the other hand, perhaps a disclaimer is in order.
Unknown quality is the absence of information, you can’t have a signal for it.
If you are using “unknown” as a euphemism for “poor”, which is the most obvious thing that makes the signal meaningful in the direction that seems apparent from context, why?.
Typically if a site is well designed, that used to be a signal that a lot of resources had been invested into the project. Now that signal is rather noisy.
Here, is particularly bad, as the main page is just complete slop bullet points. What does 'end-to-end column compression' mean? or 'uniform bytes at every layer'?
It's difficult to understand at a glance now, if this project is going to be abandonware in a week or if someone actually spending time on this. If the creator can't be bothered to read it, why should I?
Personally, I'm also very weary of Claude design-isms. Once you learn them, you see them everywhere.
fair point. I am not a front end designer and I did want something less spartan than proverbial q.txt [1] .
To be honest - many folks in the community have had access for some time , and a website was just put up this week as a 'larger community access'.
That said the mail group has more details and the blog posts are effectively internal posts turned into shorter (less code) versions. The binary itself is years of effort - with small improvements over many years and shaving a few kb each time ... 'built the old way' like amish furniture :)
Look online for Kx systems history of IP enforcement and legal threats. They are well known in the quant community for essentially being the Disney of time series databases. Nice tech yes, but the people behind it hold values that are typically diametrically opposite of hacker ethos. Good riddance.
> people behind it hold values that are typically diametrically opposite of hacker ethos
there are many open source Ks and K-likes these days which Kx are well aware of (source: i have written several K interpreters and discussed with Kx folks)
L targets the q/k4 dialect directly and is closed source which may be a special case. let's see
Quite cool, but for a new runtime of an existing language it might make sense to compare to, y'know, the other[0] runtimes of that language? Even if one has to omit the best, closed ones for lack of access / permission to benchmark?
that is fast! Are you going to do the STAC-M3 benchmarks? Kx use them as evidence that they are the fastest, which is a fair claim as nobody else participates in that particular test :-)
Closed source (and vibe-coded) software, especially for a language runtime, is a non-starter for many people including myself. But proprietary license is actually common in the APL and K family of languages. They are fascinating though, and plenty of free/open implemenations exist, for example: https://ktye.github.io/
the author of the site makes no secret that the website is vibecoded. You're unfairly extrapolating that the product itself is vibe-coded (hard to do unless he is a time-traveller). And you're missing the point of the unique advantages that L offers wrt computation on compressed data.
yup website is Claude Design for prototype. For the core ... AI has been less helpful than I hoped - I believe largely because array style languages have so little source in training? But where Claude was especially helpful (other than the web design w/ Claude Design for the prototype) was analyzing ASM output of functions and optimizing those (although it was hard to go ASM>C or ASM>rust). E.g. lots of small mistakes would have been missed by not having restrict/const in places. Claude was great at compile all functions, analyze ASM, suggest optimized ASM (and ASM2vec was helpful as well for finding any "similar" code paths that could be combined (e.g. var/dev/cov are just moments)
Unfortunately the code itself is in a style of C many find difficult to read. I blame my upbringing. ATW open sourced examples and it was not really helpful. More recently others are doing a step by step in more standard C https://github.com/ardentsia-cgs/kparser/
Not sure why all the hate (sure site may be vibecoded, not all of us are front-end masters and it's at least not spartan)... I've always found k to be fascinating, and cool idea to try and roll a new one. Wish it were wholly open source, but cool to have a new variant that seems to bench well.
I mean do a bunch of simulation work and statistics stuff but now jack about website design, so if I ever make a website about any project of mine it'll be vibedcoded or a barebones export of an org file README on github. Websites have little bearing on programming language runtime development, no?
There is so many low quality projects that rely on vibe coded websites and vibe written copy, that this becomes a smell, to the detriment of projects like OP's which is genuinely interesting and looks like high quality, well-crafted software.
Why would you not base your first impressions of a product on the way it chooses to describe itself?
It’s not feasible to install everything you lay eyes on for evaluation, and reviews often assume you already assume the product is a good fit and want to evaluate quality/validity of that assumption
Maybe because people focus on what's important rather than a fancy website? How many startups can't offer more than a fancy website. Warren Buffet, one of the greatest investors of all time has barely a website.
benchmarks at https://github.com/l-labs
unlike klong/ngn/bqn et al (which are GREAT) this has the goal of full production database compatibility (and full language compatibility).
q is the programming language underpinning kdb+ by Kx Systems - often claimed (by Kx) to be "the fastest database in the world". Possibly also the most expensive?
L is an independent implementation of a q interpreter. To those who reluctantly commercially license kdb+ - this will be a welcome alternative.
and fusion (e.g. f g h x has no intermediates e.g. mutates in place) is new
as is compute on compressed vectors (very helpful performance unlock) https://lv1.sh/blog/compute-on-compressed/
[1] seems to have the actual details on compressed vectors. looks like for each page you're storing min&max, and the values come as offsets from the min (smaller->less bits per value), did I get that right?
simple example at https://lv1.sh/blog/compute-on-compressed/
But in general compression is reducing the bit width of the input through an encoder (FOR or Frame of Reference is an old and good example). So we store the base in an offset location then the large payload is a much smaller size. E.g. i64 can goto i16. Then simd gets more #'s per cycle on the smaller, and the base is added to the scratch in stack (for sum). avg is similar (since it is just sum / count)
KlongPy and KlongPy + duckDB are wonderful. GREAT job. I think there's a whole world of backprop / ML work that can be done in that style! Given the heritage not surprising that most Klong code is ~= L in k mode! Same with ngn/k . E.g. sigmoid is almost the same (:: for assign in klongpy , : for l/q/k). The autograd work is great!
We'd like to see some benchmarks against open and closed k interpreters please! I'm curious how well a vibe coded k/q interpreter stands up to Shakti or whatever Mr. Whitney is letting out the door right now.
Definitely not a vibe coded language - I really wish the ai models were more helpful - but I should do a write up of the assembly analysis (and asm2vec). ATW is working on even more impressive things at the moment! Well beyond the scope of an interpreter itself.
the early versions of L (circa ... 2011/2012) couldn't really deliver any substantial performance improvement over what is out there (BQN/ngn/Kona). The memory bandwidth was the limit - I simply couldn't keep the cores active. ~2018/2019 I started from scratch with arrays (vectors) using compression by default. That was a massive unlock - and I could not keep most of the CPU doing actual compute! Then it was years of working on compression native operations - some of which were obvious[3] like sum/reductions ... others not so much!
Sorry about that. Basically you can now write and run code in the K/q/or qsql languages on your laptop for free. For a very long time (20 years+) this language has not been accessible to most - it’s almost exclusively used on Wall Street and has a very high price barrier to entry. Q programmers are still some of the most highly paid engineers consistently - as the workforce is small and the use cases are (generally) extremely lucrative. If you are a programmer wondering what comes next… q might be for you. Over the decades I’ve taught a few dozen people Q- and I’m happy to say they’ve gone on to have wonderful and stable careers in high paying jobs with deep stability. Mostly hedge funds and banks (and exchanges).
It’s a wonderful and often times world changing language and now you can play with it yourself.
It’s also very very fast. As a database it’s still insanely great for many usecases.
if you k you k ;-)
for the uninitiated, this looks like some wallstreet quant's new startup. Initially I thought it was the rebrand of shakti, Arthur Whites most recent rewrite of an array language. It's purpose built tooling for computing tick data for financial markets, but the best way I can describe it is codegolf for experienced programmers who don't want to give up the keyboard. these tools combine dataaccess and the ability to compute against that data with as few abstractions as possible.
As someone who does not know what k4, qSQL, or q are, reading through the landing page of this website was giving me mild schizophrenia. And then I tried to search for these things in the old way, and received incredibly dry technical sites that still don’t tell me what it is, and all these names are wildly SEO unfriendly. So I had Claude give me context and it’s apparently the database Wall Street uses for tick data. Sounds cool but, jeez.
K4 is the K programming language, Q is a language built on top of K. They are the practical over achieving members of the array family of programming languages.
Flagged, lots of unexplained grandiose thinking like “k and q made the vector the unit of thought.” K and q are unexplained and unlinked and “unit vector of thought” is pseudoscientific language
Extremely likely to be AI, though I’m not sure that matters for rules re: submissions
The language and its ideas come from Ken Iverson's famous paper "Notation as a Tool of Thought". This is a common understanding for people in the APL, K/J/Q etc array language communities, who are likely this website and product's intended audience.
> lots of unexplained grandiose thinking like “ k and q made the vector the unit of thought.”
That much makes sense in context: K is an array language, like APL, J, etc. From what little experience I have with J, ‘vector as a unit of thought’ seems like a reasonable description.
That’s fair: The most fair-to-submitter reading of your comment is my concerns are unfounded. If you didn’t intend that, that’s fine.
For moderators, I’d suggest that for the community, it’s spam. That’s one example, there are many more like it. The individual statement may be defensible but is still pseudoscientific language. This sort of content is a massive burden to community. Unanswerable anrguments about AI writing, whether the ability for an individual to have a parse-able reading is the same as writing being parable. The net effect is negative experiences for many and copy-editing for someone who did not do copy editing.
The 'unit of thought' thing is a nod to the Ken Iverson paper on APL that won him the Turing Award ("Notation as a Tool of Thought"), also referenced by Dyalog's tag line "The tool of thought for software solutions"). Variations of that phrase are endemic in the array languages space.
I see how jarring it is for you, but I want to comment-vouch against your flag. If you slow down a little, you can spot some meaning there. Its oddly phrased, but it does make sense that vectors (as in: simd vectors) are the building blocks of execution (to put it closer to how I might say it).
I’ve had similar ideas in the past: clearly simd is the way to get the most out of your cpu. Can we design a language where all operations are automatically simd, and it takes effort to do anything in scalars?
And I guess these array languages are what you might get.
It’s not ‘unit vector of thought,’ btw, which is weirder than what it says.
>If a story is spam or off-topic, flag it. Don't feed egregious comments by replying; flag them instead. If you flag, please don't also comment that you did.
reply