> Mathematicians Weinan Lin, Guozhen Wang, and Zhouli Xu have proven that 126-dimensional space can contain exotic, twisted shapes known as manifolds with a Kervaire invariant of 1—solving a 65-year-old problem in topology. These manifolds, previously known to exist only in dimensions 2, 6, 14, 30, and 62, cannot be smoothed into spheres and were the last possible case under what’s called the “doomsday hypothesis.” Their existence in dimension 126 was confirmed using both theoretical insights and complex computer calculations, marking a major milestone in the study of high-dimensional geometric structures.
So these are all powers of 2 minus 2, and it looks like from the article that the pattern doesn’t exist in 2^8 - 2 or higher. Is there any description a layperson might understand as to why it stops instead of going on forever!
Who are you talking about? The entire claim was "there was no reasonable description for hypotenuse length before the theorem"; I assume the Pythagorean theorem is intended. But that claim is crazy. You just contradicted it yourself.
I should note that hypotenuses look no different in 126 dimensions than they do in 2 - three points determine a plane, regardless of how big the space containing the plane might be - but that's not really relevant to anything here.
Here is my guess. Number of dimensions is more like a hyperparameter than a parameter. Each time you increase the dimension by 1 you get a new world. You cane expect a simple pattern to go on forever.
>
They’re all double the last dimension plus two, without skipping any in that sequence - but that offers no insight into why it wouldn’t hold for 254.
Wikipedia at least gives a literature reference and concise explanation for the reason:
"Hill, Hopkins & Ravenel (2016) showed that the Kervaire invariant is zero for n-dimensional framed manifolds for n = 2^k− 2 with k ≥ 8. They constructed a cohomology theory Ω with the following properties from which their result follows immediately:
* The coefficient groups Ω^n(point) have period 2^8 = 256 in n
* The coefficient groups Ω^n(point) have a "gap": they vanish for n = -1, -2, and -3
* The coefficient groups Ω^n(point) can detect non-vanishing Kervaire invariants: more precisely if the Kervaire invariant for manifolds of dimension n is nonzero then it has a nonzero image in Ω^{−n}(point)"
Paper:
Hill, Michael A.; Hopkins, Michael J.; Ravenel, Douglas C. (2016). "On the nonexistence of elements of Kervaire invariant one"
Is it conventional for mathematicians to talk about “the dimensions” like this? I think they are talking about a 126 dimensional space here, but I am a lowly computerer, so maybe this went over my head.
We usually don't talk about "the dimensions", we talk about the general case: n-dimensional spaces (theorems covering all dimensions simultaneously) or infinite-dimensional spaces (individual spaces covering all finite-dimensional spaces).
Of course, when you try to generalize your theorems you are also interested in the cases where generalization fails. In this case, there is something that happens in a 2-dimensional space, in a 6-, 14- or 30-dimensional space. Mathematicians would say "it happens in 2, 6, 14 or 30 dimensions". I never noticed that this is jargon specific to mathematicians.
Problems in geometry tend to get (at least) exponentially harder to solve computationally as the dimensions grow, e.g. the number of vertices of the n-dimensional cube is literally the exponential of base 2. Which is why they discovered something about 126-dimensional space now, when the results for lower dimensions have been known for decades.
But that's not how the article says it. It says "in dimensions 2, 6, 14, 30 and 62" instead of "in 2,6,14 or 30 dimensions". The later sounds fine, but "dimensions 8 and 24" to me sounds too much like something is happening in "8th and 24th dimension". It even uses singular "dimension 126" as if you took >=126 dimensional space, ordered the axis and something interesting happened along 126th and only that one.
Yeah, that's not what that means. In math "dimension" is used as a statistic. As in, "this manifold has a dimension of 4". So you can say things like "in dimension 4" to mean "when the dimension is equal to 4". We do also say "in 4 dimensions"; it just varies. The two phrases are equivalent. There is no ordering of dimensions or anything like that.
It's a good question. It's easy to assume they're talking about R^126 (where R is the reals) but digging a bit deeper I don't think it's true.
The Kervaire invariant is a property of an "n-dimensional manifold", so the paper is likely about 126-dimensional manifolds. That in turn has a formal definition, and although it's not my specialization, I think means it can be locally represented as an n-dimensional Euclidean space.
A simple example would be a circle, which I guess would be a 1-dimensional manifold, because every point on a circle has a tangent where the circle can be approximated by a line passing through the same point.
So they're saying that there are these surfaces which can be locally approximated by 126-dimensional Euclidean spaces. This in turn probably requires that the surface itself is embedded in some higher-dimensional space such as R^127.
Manifolds are generally considered objects of themselves, and it may be difficult to embed then in higher dimensional objects. This is especially the case for tricky manifolds like those with a Kervaire invariant of 1.
Not using the language of this article. Referring to e.g. a two-dimensional space as "Dimension 2" is irregular. One might say that the space has dimension 2 (as shorthand for "has a dimension of 2"), but "Dimension 2" is not used as the proper name of such a space.
It's common in math to say things like "in dimension 2" to mean "when the dimension is 2". It doesn't necessarily refer to a specific space (although it could based on context). It's just setting a contextual variable. Many problems occur in varying dimension and oftentimes you want to restrict discussion to a specific dimension.
Right - what I meant specifically is the use of names like "Dimension 2" (with the capital D) as if to refer to a specific location with that name. Among other things, it has too many associations with pulp science fiction. :)
I’m not a mathematician (just a programmer), but reading this made me wonder—doesn’t this kind of dimensional weirdness feel a bit like how LLMs organize their internal space? Like how similar ideas or meanings seem to get pulled close together in a way that’s hard to visualize, but clearly works?
That bit in the article about knots only existing in 3D really caught my attention. "And dimension 3 is the only one that can contain knots — in any higher dimension, you can untangle a knot even while holding its ends fast."
That’s so unintuitive… and I can't help thinking of how LLMs seem to "untangle" language meaning in some weird embedding space that’s way beyond anything we can picture.
Is there a real connection here? Or am I just seeing patterns where there aren’t any?
There is a real connection insofar as the internal space of an LLM is a vector space so things which hold for vector spaces hold for the internal space of an LLM. This is the power of abstract algebra. When an algebraic structure can be identified you all of a sudden know a ton of things about it because mathematicians have been working to understand those structures for a while.
The internal space of an LLM would also have things in common with how, say currents flow in a body of water because that too is a vector space. When you study this stuff you get this sort of zen sense of everything getting connected to everything else. Eg in one of my textbooks you look at how pollution spreads through the great lakes and then literally the next example looks at how drugs are absorbed into the bloodstream through the stomach and it’s exactly the same dynamic matrix and set of differential equations. Your stomach works the same as the great lakes on a really fundamental level.
The spaces being described here are a little more general than vector spaces, so some of things which are true about vector spaces wouldn’t necessarily work the same way here.
It's pretty simple, actually. Imagine you have a knot you want to untie. Lay it out in a knot diagram, so that there are just finitely many crossings. If you could pass the string through itself at any crossing, flipping which strand is over and which is under, it would be easy, wouldn't it? It's only knotted because those over/unders are in an unfavorable configuration. Well, with a 4th spatial dimension available, you can't pass the string through itself, but you can still invert any crossing by using the extra dimension to move one strand around the other, in a way that wouldn't be possible in just 3 dimensions.
> Or am I just seeing patterns where there aren’t any?
Your intuition is correct, it doesn't! A "3D hyperrope" is in fact just the surface of a ball[1], and it turns out that you can actually form non-trivial knots of that spherical surface in a 4-dimensional ambient space (and analogously they can be un-knotted if you then move up to 5-dimension ambient space, although the mechanics for doing so might be a little trickier than in the 1d-in-4d case). In fact, if you have a k-dimensional sphere, you can always knot it up in a k+2 dimensional ambient space (and can then always be unknotted if you add enough additional dimensions).
[1] note that a [loop of] rope is actually a 1-dimensional object (it only has length, no width), so the next dimension up should be a 2-dimensional object, which is true of the surface of a ball. a topologist would call these things a 1-sphere and a 2-sphere, respectively
I'm not sure what you mean here. This is discussing a 1-dimensional structure embeded in 4-dimensional space. If you're not sure it works for something else, well, that isn't what's under discussion.
If you just mean you're just unclear on the first step, of laying the knot out in 2D with crossings marked over/under, that's always possible after just some ordinary 3D adjustments. Although, yeah, if you asked me to prove it, I dunno that I could give one, I'm not a topologist... (and I guess now that I think about it the "finitely many" crossings part is actually wrong if we're allowing wild knots, but that's not really the issue)
I would be careful about drawing any analogies which are “too cute”. We use LLMs because they work, not because they are are theoretically optimal. They are full of lossy tradeoffs that work in practice because they are a good match for the hardware and data we have.
What is true is that you can get good results by projecting lower dimensional data into higher dimensions, applying operations, and then projecting it back down.
Warning: If you get too deep into this, you're going to find yourself dealing with a lot of technicalities like "are we talking about smooth knots, tame knots, topological knots, or PL knots?" But the above statement I think is true regardless!
I don't think this is true, I believe humans unravel language meaning in the plain old 3+1 dimensional Galilean manifold of events in nonrelativistic spacetime, just as animals do with vocalizations and body language, and LLM confabulations / reasoning errors are fundamentally due to their inability to access this level of meaning. (Likewise with video generators not understanding object permanence.)
> Or am I just seeing patterns where there aren’t any?
Meta: there are patterns to seeing patterns, and it's good to understand where your doubt springs from.
1: hallucinating connections/metaphors can be a sign you're spending too much time within a topic. The classic is binging on a game for days, and then resurfacing back into a warped reality where everything you see related back to the game. Hallucinations is the wrong word sorry: because sometimes the metaphors are deeply insightful and valuable: e.g. new inventions or unintuitive cross-discipline solutions to unsolved maths problems. Watch when others see connections to their pet topics: eventually you'll learn to internally dicern your valuable insights from your more fanciful ones. One can always consider whether a temporary change to another topic would be healthy? However sometimes diving deeper helps. How to choose??
2: there's a narrow path between valuable insight and debilitating overmatching. Mania and conspirational paranioa find amazing patterns, however they tend to be rather unhelpful overall. Seek a good balance.
3: cultivate the joy within yourself and others; arts and poetry is fun. Finding crazy connections is worthwhile and often a basis for humour. Engineering is inventive and being a judgy killjoy is unhealthy for everyone.
Hmmm, I usually avoid philosophical stuff like that. Abstract stuff is too difficult to write down well.
A lot of innovation is stealing ideas from two domains that often don’t talk to each other and combining them. That’s how we get simultaneous invention. Two talented individuals both realize that a new fact, when combined with existing facts, implies the existence of more facts.
Someone once asserted that all learning is compression, and I’m pretty sure that’s how polymaths work. Maybe the first couple of domains they learn occupy considerable space in their heads, but then patterns emerge, and this school has elements from these other three, with important differences. X is like Y except for Z. Shortcut is too strong a word, but recycling perhaps.
I'm unsure if I misunderstand you or your writing ingroup!
> learning is compression
I don't think I know enough about compression to find that metaphor useful
> occupy considerable space in their heads
I reckon this is a terribly misleading cliche. Our brains don't work like hard drives. From what I see we can keep stuffing more in there (compression?). Much of my past learning is now blurred but sometimes it surfaces in intuitions? Perhaps attention or interest is a better concept to use?
My favorite thing about LLMs is wondering how much of people's (or my own) conversations are just LLMs. I love the idea of playing games with people to see if I can predictably trigger phrases from people, but unfortunately I would feel like a heel doing that (so I don't). And catching myself doing an LLM reply is wonderful.
Some of the other sibling replies are also gorgeously vague-as (and I'm teasing myself with vagueness too). Abstracts are so soft.
If you have some probability distribution over finite sequences of bits, a stream of independent samples drawn from that stream can be compressed so that the number of bits in the compressed stream per sample from the original stream, is (in the long run) the (base 2) entropy of the distribution. Likewise if instead of independent samples from a distribution there is instead a Markov process or something like that, with some fixed average rate of entropy.
The closer one can get to this ideal, the closer one has to a complete description of the distribution.
I think this is the sort of thing they were getting at with the compression comment.
I think LLM layers are basically big matrices, which are one of the most popular many-dimensional objects that us non-mathematician mortals get to play with.
I'm also left wondering what happened to the thin band. It seemed like it would have been perfectly sufficient to just cut the torus and turn it into a tube. Likewise I wonder why we need to import a sphere rather than just pinch the ends of the tube shut and say it's now a sphere.
I don't really get the whole surgery concept, if you can just cut the torus (and this could be anywhere, eg along the length of its inner or outer diameters) and glue the edges together aren't you just waving away the problem? By that logic I can slice open a sphere and call it a sheet, or conversely declare all shees to be spheres that haven't been glued together yet.
> By that logic I can slice open a sphere and call it a sheet
You can do this. If you remove a point (or a line, or really any connected component), you get a space which is the same as the plane. What happens if you remove two distinct points? You end up with with a very thick circle. Three points? It starts to get harder to visualize, but you end up with two circles joined at a point. As you remove more points you will get more circles joined together. From a mathematical perspective, these spaces are very different. If we start to allow gluing arbitrary points in the sphere together it gets even worse, and you can get some pretty wild spaces.
The point of surgery is that by requiring this gluing in of these spheres along the boundary of the space we cut out, the resulting spaces are not as wild - or at least are easier to handle than if we do any operation. To give an example, one might have some space and we want to determine if it has property A. The problem is our space has some property B which makes it difficult to determine property A directly. But by performing surgery in a specific way, we can produce a new space which has property A if and only if the original space did, and importantly, no longer has property B.
For property As that mathematicians care about, surgery often does a good job of preserving the property. In contrast things like just cutting and gluing points together without care will typically change property A, so it does not help as much.
> Likewise I wonder why we need to import a sphere rather than just pinch the ends of the tube shut and say it's now a sphere.
I am not an expect on surgery, but I think from a mathematical perspective, pinching the ends of the tube shut and gluing in a new sphere would be equivalent operations. This pinching operation would be formalized as a "quotient space", and you can formalize the sphere as a "quotient" space equivalent to the pinching.
> I don't really get the whole surgery concept, if you can just cut the torus (and this could be anywhere, eg along the length of its inner or outer diameters) and glue the edges together aren't you just waving away the problem? By that logic I can slice open a sphere and call it a sheet, or conversely declare all shees to be spheres that haven't been glued together yet.
It's admittedly been a good 15 years since I cracked open a topology textbook, but the high-level, hand-wavey idea behind this sort of topological surgery isn't that you slice up manifolds and glue them together willy-nilly, but that you do so in very precise and controlled ways, which (and this is the very important bit) you've proven ahead of time preserve (or modify in a knowable fashion) some property of the manifold you care about. Rather than waving away the problem, you're decomposing it into a set of simpler ones which are ideally more tractable for some reason or another (perhaps you can compute a given property from first principles for a sphere, but not for a torus, for instance).
Seeing as mathematicians proving things in math has minimal relation to the real world, I'm not sure how important this is.
Mathematicians and physicists have been speculating about the universe having more than 4 dimensions, and/or our 4 dimensional space existing as some kind of film on a higher dimensional space for ages, but I've yet to see compelling proof that any of that is the case.
Edit: To be clear, I'm not attempting to minimize the accomplishment of these specific people. More observing that advanced mathematics seems only tangentially related to reality.
You might consider reading Hardy's A Mathematician's Apology. It gives an argument for studying math for the sake of math. Personally, reading a beautiful proof can be as compelling as reading a beautiful poem and needs no further justification.
However, there is another reason to read this essay. Hardy gives a few examples of fields of math which are entirely useless. Number theory, he claims, has absolutely no applications. The study of non-euclidean geometry, he claims, has absolutely no applications. History has proven him dramatically wrong, “pure” math has a way of becoming indispensable
I have no problem studying Math just to study Math. I read the title and jumped to some conclusions, I'm afraid. Was talking to a friend about String Theory and their 11+ dimensions the other day and that is immediately where my brain went to with this one. The article is interesting even though I have zero desire to personally study math just for math's sake.
There is a huge amount of mathematics that initially seems as though it could not possibly have any practical application that later turns out central to all sorts of things in the real world.
The most obvious examples are number theory and group theory, which are respectively the study of numbers and how they behave under basic operations like arithmetic, and the study of a type of set with a single operation that satisfies very basic rules[1]. How could this possibly have any relevance or practical application? And yet it turns out they are central to cryptography and information theory. Joseph Fourier trying to solve the equations that govern how heat diffuses through a metal came up with the theory that forms the basis for how we do video and audio compression (and a ton of other things).
Finally mathematicians don’t speculate about how many dimensions the universe has, they study 4- and higher- dimensional objects and spaces to understand them. This theory is used all over the place. You can’t have a function like a temperature map without 4 dimensions (3 for the spatial coordinates of your input and one for the output).
[1] this turns out (non-obviously) to be the study of symmetry.
> More observing that advanced mathematics seems only tangentially related to reality.
You might be surprised; there have proven to be a number of surprising connections between abstract mathematical structures and more concrete sciences. For instance, group theory - long thought to be an highly abstract area of mathematics with no practical application - turned out to have some very direct applications in chemistry, particularly in spectroscopy.
When you try to solve one problem involving two objects in three-dimensional space, you have a six-dimensional problem space. If you have two moving objects, you have a twelve-dimensional problem space. Higher dimensional spaces show up everywhere when dealing with real-life problems.
> Mathematicians Weinan Lin, Guozhen Wang, and Zhouli Xu have proven that 126-dimensional space can contain exotic, twisted shapes known as manifolds with a Kervaire invariant of 1—solving a 65-year-old problem in topology. These manifolds, previously known to exist only in dimensions 2, 6, 14, 30, and 62, cannot be smoothed into spheres and were the last possible case under what’s called the “doomsday hypothesis.” Their existence in dimension 126 was confirmed using both theoretical insights and complex computer calculations, marking a major milestone in the study of high-dimensional geometric structures.
So these are all powers of 2 minus 2, and it looks like from the article that the pattern doesn’t exist in 2^8 - 2 or higher. Is there any description a layperson might understand as to why it stops instead of going on forever!
The proof that it stops instead of going on forever is here: https://arxiv.org/abs/0908.3724
It’s more than 200 pages of pretty technical mathematics, so I’m reasonably confident that there is no description a layperson might understand.
Yet.
There was no reasonable description for hypotenuse length before the theorem.
I assume you mean the Pythagorean theorem, and by description you mean equation?
I was trying to be vague to see how pedantic the audience here is when the meaning of a statement can be easily inferred from context. Seems very.
What? "The longest side of a triangle" isn't a challenging concept to understand or describe.
(from the nearby toilet cubicle) "That's a right triangle, ya idiot!"
D'oh!
[remarkably incorrect answer misunderstanding the above comments]
Who are you talking about? The entire claim was "there was no reasonable description for hypotenuse length before the theorem"; I assume the Pythagorean theorem is intended. But that claim is crazy. You just contradicted it yourself.
I should note that hypotenuses look no different in 126 dimensions than they do in 2 - three points determine a plane, regardless of how big the space containing the plane might be - but that's not really relevant to anything here.
Sorry, yes, that was a major misunderstanding of the previous comment by me. I am utterly wrong!
(The longest side of a right triangle)
If you believe that you can only subtend a 90 degree angle, sure.
("Subtend" is a direct part-for-part translation of the verb which also gives us the name "hypotenuse".)
Here is my guess. Number of dimensions is more like a hyperparameter than a parameter. Each time you increase the dimension by 1 you get a new world. You cane expect a simple pattern to go on forever.
They’re all double the last dimension plus two, without skipping any in that sequence - but that offers no insight into why it wouldn’t hold for 254.
> They’re all double the last dimension plus two, without skipping any in that sequence - but that offers no insight into why it wouldn’t hold for 254.
Wikipedia at least gives a literature reference and concise explanation for the reason:
> https://en.wikipedia.org/w/index.php?title=Kervaire_invarian...
"Hill, Hopkins & Ravenel (2016) showed that the Kervaire invariant is zero for n-dimensional framed manifolds for n = 2^k− 2 with k ≥ 8. They constructed a cohomology theory Ω with the following properties from which their result follows immediately:
* The coefficient groups Ω^n(point) have period 2^8 = 256 in n
* The coefficient groups Ω^n(point) have a "gap": they vanish for n = -1, -2, and -3
* The coefficient groups Ω^n(point) can detect non-vanishing Kervaire invariants: more precisely if the Kervaire invariant for manifolds of dimension n is nonzero then it has a nonzero image in Ω^{−n}(point)"
Paper:
Hill, Michael A.; Hopkins, Michael J.; Ravenel, Douglas C. (2016). "On the nonexistence of elements of Kervaire invariant one"
* https://arxiv.org/abs/0908.3724
* https://annals.math.princeton.edu/2016/184-1/p01
Can't wait to read up on volume 127 of the Dimension periodical.
ah yes, now I understand completely
Is it conventional for mathematicians to talk about “the dimensions” like this? I think they are talking about a 126 dimensional space here, but I am a lowly computerer, so maybe this went over my head.
> Is it conventional for mathematicians to talk about “the dimensions” like this
There is an old joke:
How do you imagine a 126-dimensional space? - Simple: imagine an n-dimensional space and set n=126.
We usually don't talk about "the dimensions", we talk about the general case: n-dimensional spaces (theorems covering all dimensions simultaneously) or infinite-dimensional spaces (individual spaces covering all finite-dimensional spaces).
Of course, when you try to generalize your theorems you are also interested in the cases where generalization fails. In this case, there is something that happens in a 2-dimensional space, in a 6-, 14- or 30-dimensional space. Mathematicians would say "it happens in 2, 6, 14 or 30 dimensions". I never noticed that this is jargon specific to mathematicians.
Problems in geometry tend to get (at least) exponentially harder to solve computationally as the dimensions grow, e.g. the number of vertices of the n-dimensional cube is literally the exponential of base 2. Which is why they discovered something about 126-dimensional space now, when the results for lower dimensions have been known for decades.
But that's not how the article says it. It says "in dimensions 2, 6, 14, 30 and 62" instead of "in 2,6,14 or 30 dimensions". The later sounds fine, but "dimensions 8 and 24" to me sounds too much like something is happening in "8th and 24th dimension". It even uses singular "dimension 126" as if you took >=126 dimensional space, ordered the axis and something interesting happened along 126th and only that one.
Yeah, that's not what that means. In math "dimension" is used as a statistic. As in, "this manifold has a dimension of 4". So you can say things like "in dimension 4" to mean "when the dimension is equal to 4". We do also say "in 4 dimensions"; it just varies. The two phrases are equivalent. There is no ordering of dimensions or anything like that.
It's a good question. It's easy to assume they're talking about R^126 (where R is the reals) but digging a bit deeper I don't think it's true.
The Kervaire invariant is a property of an "n-dimensional manifold", so the paper is likely about 126-dimensional manifolds. That in turn has a formal definition, and although it's not my specialization, I think means it can be locally represented as an n-dimensional Euclidean space.
A simple example would be a circle, which I guess would be a 1-dimensional manifold, because every point on a circle has a tangent where the circle can be approximated by a line passing through the same point.
So they're saying that there are these surfaces which can be locally approximated by 126-dimensional Euclidean spaces. This in turn probably requires that the surface itself is embedded in some higher-dimensional space such as R^127.
Manifolds are generally considered objects of themselves, and it may be difficult to embed then in higher dimensional objects. This is especially the case for tricky manifolds like those with a Kervaire invariant of 1.
As someone who is also not a mathematician it sounded perfectly normal to me.
Not using the language of this article. Referring to e.g. a two-dimensional space as "Dimension 2" is irregular. One might say that the space has dimension 2 (as shorthand for "has a dimension of 2"), but "Dimension 2" is not used as the proper name of such a space.
It's common in math to say things like "in dimension 2" to mean "when the dimension is 2". It doesn't necessarily refer to a specific space (although it could based on context). It's just setting a contextual variable. Many problems occur in varying dimension and oftentimes you want to restrict discussion to a specific dimension.
Right - what I meant specifically is the use of names like "Dimension 2" (with the capital D) as if to refer to a specific location with that name. Among other things, it has too many associations with pulp science fiction. :)
I’m not a mathematician (just a programmer), but reading this made me wonder—doesn’t this kind of dimensional weirdness feel a bit like how LLMs organize their internal space? Like how similar ideas or meanings seem to get pulled close together in a way that’s hard to visualize, but clearly works?
That bit in the article about knots only existing in 3D really caught my attention. "And dimension 3 is the only one that can contain knots — in any higher dimension, you can untangle a knot even while holding its ends fast."
That’s so unintuitive… and I can't help thinking of how LLMs seem to "untangle" language meaning in some weird embedding space that’s way beyond anything we can picture.
Is there a real connection here? Or am I just seeing patterns where there aren’t any?
There is a real connection insofar as the internal space of an LLM is a vector space so things which hold for vector spaces hold for the internal space of an LLM. This is the power of abstract algebra. When an algebraic structure can be identified you all of a sudden know a ton of things about it because mathematicians have been working to understand those structures for a while.
The internal space of an LLM would also have things in common with how, say currents flow in a body of water because that too is a vector space. When you study this stuff you get this sort of zen sense of everything getting connected to everything else. Eg in one of my textbooks you look at how pollution spreads through the great lakes and then literally the next example looks at how drugs are absorbed into the bloodstream through the stomach and it’s exactly the same dynamic matrix and set of differential equations. Your stomach works the same as the great lakes on a really fundamental level.
The spaces being described here are a little more general than vector spaces, so some of things which are true about vector spaces wouldn’t necessarily work the same way here.
> That’s so unintuitive…
It's pretty simple, actually. Imagine you have a knot you want to untie. Lay it out in a knot diagram, so that there are just finitely many crossings. If you could pass the string through itself at any crossing, flipping which strand is over and which is under, it would be easy, wouldn't it? It's only knotted because those over/unders are in an unfavorable configuration. Well, with a 4th spatial dimension available, you can't pass the string through itself, but you can still invert any crossing by using the extra dimension to move one strand around the other, in a way that wouldn't be possible in just 3 dimensions.
> Or am I just seeing patterns where there aren’t any?
Pretty sure it's the latter.
That makes sense for a 2D rope in 4D space, but I’m not convinced the same approach holds for a 3D ”hyperrope” in 4D space.
Your intuition is correct, it doesn't! A "3D hyperrope" is in fact just the surface of a ball[1], and it turns out that you can actually form non-trivial knots of that spherical surface in a 4-dimensional ambient space (and analogously they can be un-knotted if you then move up to 5-dimension ambient space, although the mechanics for doing so might be a little trickier than in the 1d-in-4d case). In fact, if you have a k-dimensional sphere, you can always knot it up in a k+2 dimensional ambient space (and can then always be unknotted if you add enough additional dimensions).
[1] note that a [loop of] rope is actually a 1-dimensional object (it only has length, no width), so the next dimension up should be a 2-dimensional object, which is true of the surface of a ball. a topologist would call these things a 1-sphere and a 2-sphere, respectively
I'm not sure what you mean here. This is discussing a 1-dimensional structure embeded in 4-dimensional space. If you're not sure it works for something else, well, that isn't what's under discussion.
If you just mean you're just unclear on the first step, of laying the knot out in 2D with crossings marked over/under, that's always possible after just some ordinary 3D adjustments. Although, yeah, if you asked me to prove it, I dunno that I could give one, I'm not a topologist... (and I guess now that I think about it the "finitely many" crossings part is actually wrong if we're allowing wild knots, but that's not really the issue)
I would be careful about drawing any analogies which are “too cute”. We use LLMs because they work, not because they are are theoretically optimal. They are full of lossy tradeoffs that work in practice because they are a good match for the hardware and data we have.
What is true is that you can get good results by projecting lower dimensional data into higher dimensions, applying operations, and then projecting it back down.
> "And dimension 3 is the only one that can contain knots — in any higher dimension, you can untangle a knot even while holding its ends fast."
Maybe you could create "hyperknots", e.g. in 4D a knot made of a surface instead of a string? Not sure what "holding one end" would mean though.
Yes, circles don't knot in 4D, but the 2-sphere does: https://en.wikipedia.org/wiki/Knot_theory#Higher_dimensions
Warning: If you get too deep into this, you're going to find yourself dealing with a lot of technicalities like "are we talking about smooth knots, tame knots, topological knots, or PL knots?" But the above statement I think is true regardless!
Yep — you can always “knot” a sphere of two dimensions lower, starting with a circle in 3D and a sphere in 4D.
When you untie a knot, it’s ends are fixed in time.
Humans also unravel language meaning from within a hyper dimensional manifold.
I don't think this is true, I believe humans unravel language meaning in the plain old 3+1 dimensional Galilean manifold of events in nonrelativistic spacetime, just as animals do with vocalizations and body language, and LLM confabulations / reasoning errors are fundamentally due to their inability to access this level of meaning. (Likewise with video generators not understanding object permanence.)
It's not just LLMs. Deep learning in general forms these multi-d latent spaces
> Or am I just seeing patterns where there aren’t any?
Meta: there are patterns to seeing patterns, and it's good to understand where your doubt springs from.
1: hallucinating connections/metaphors can be a sign you're spending too much time within a topic. The classic is binging on a game for days, and then resurfacing back into a warped reality where everything you see related back to the game. Hallucinations is the wrong word sorry: because sometimes the metaphors are deeply insightful and valuable: e.g. new inventions or unintuitive cross-discipline solutions to unsolved maths problems. Watch when others see connections to their pet topics: eventually you'll learn to internally dicern your valuable insights from your more fanciful ones. One can always consider whether a temporary change to another topic would be healthy? However sometimes diving deeper helps. How to choose??
2: there's a narrow path between valuable insight and debilitating overmatching. Mania and conspirational paranioa find amazing patterns, however they tend to be rather unhelpful overall. Seek a good balance.
3: cultivate the joy within yourself and others; arts and poetry is fun. Finding crazy connections is worthwhile and often a basis for humour. Engineering is inventive and being a judgy killjoy is unhealthy for everyone.
Hmmm, I usually avoid philosophical stuff like that. Abstract stuff is too difficult to write down well.
A lot of innovation is stealing ideas from two domains that often don’t talk to each other and combining them. That’s how we get simultaneous invention. Two talented individuals both realize that a new fact, when combined with existing facts, implies the existence of more facts.
Someone once asserted that all learning is compression, and I’m pretty sure that’s how polymaths work. Maybe the first couple of domains they learn occupy considerable space in their heads, but then patterns emerge, and this school has elements from these other three, with important differences. X is like Y except for Z. Shortcut is too strong a word, but recycling perhaps.
I'm unsure if I misunderstand you or your writing ingroup!
> learning is compression
I don't think I know enough about compression to find that metaphor useful
> occupy considerable space in their heads
I reckon this is a terribly misleading cliche. Our brains don't work like hard drives. From what I see we can keep stuffing more in there (compression?). Much of my past learning is now blurred but sometimes it surfaces in intuitions? Perhaps attention or interest is a better concept to use?
My favorite thing about LLMs is wondering how much of people's (or my own) conversations are just LLMs. I love the idea of playing games with people to see if I can predictably trigger phrases from people, but unfortunately I would feel like a heel doing that (so I don't). And catching myself doing an LLM reply is wonderful.
Some of the other sibling replies are also gorgeously vague-as (and I'm teasing myself with vagueness too). Abstracts are so soft.
If you have some probability distribution over finite sequences of bits, a stream of independent samples drawn from that stream can be compressed so that the number of bits in the compressed stream per sample from the original stream, is (in the long run) the (base 2) entropy of the distribution. Likewise if instead of independent samples from a distribution there is instead a Markov process or something like that, with some fixed average rate of entropy.
The closer one can get to this ideal, the closer one has to a complete description of the distribution.
I think this is the sort of thing they were getting at with the compression comment.
I think LLM layers are basically big matrices, which are one of the most popular many-dimensional objects that us non-mathematician mortals get to play with.
The "Mathematical Surgery" illustration is funny. Mathematicians can make a sphere from a torus and two halves of a sphere. Amazing!
I'm also left wondering what happened to the thin band. It seemed like it would have been perfectly sufficient to just cut the torus and turn it into a tube. Likewise I wonder why we need to import a sphere rather than just pinch the ends of the tube shut and say it's now a sphere.
I don't really get the whole surgery concept, if you can just cut the torus (and this could be anywhere, eg along the length of its inner or outer diameters) and glue the edges together aren't you just waving away the problem? By that logic I can slice open a sphere and call it a sheet, or conversely declare all shees to be spheres that haven't been glued together yet.
They have played us for absolute fools
> By that logic I can slice open a sphere and call it a sheet
You can do this. If you remove a point (or a line, or really any connected component), you get a space which is the same as the plane. What happens if you remove two distinct points? You end up with with a very thick circle. Three points? It starts to get harder to visualize, but you end up with two circles joined at a point. As you remove more points you will get more circles joined together. From a mathematical perspective, these spaces are very different. If we start to allow gluing arbitrary points in the sphere together it gets even worse, and you can get some pretty wild spaces.
The point of surgery is that by requiring this gluing in of these spheres along the boundary of the space we cut out, the resulting spaces are not as wild - or at least are easier to handle than if we do any operation. To give an example, one might have some space and we want to determine if it has property A. The problem is our space has some property B which makes it difficult to determine property A directly. But by performing surgery in a specific way, we can produce a new space which has property A if and only if the original space did, and importantly, no longer has property B.
For property As that mathematicians care about, surgery often does a good job of preserving the property. In contrast things like just cutting and gluing points together without care will typically change property A, so it does not help as much.
> Likewise I wonder why we need to import a sphere rather than just pinch the ends of the tube shut and say it's now a sphere.
I am not an expect on surgery, but I think from a mathematical perspective, pinching the ends of the tube shut and gluing in a new sphere would be equivalent operations. This pinching operation would be formalized as a "quotient space", and you can formalize the sphere as a "quotient" space equivalent to the pinching.
> I don't really get the whole surgery concept, if you can just cut the torus (and this could be anywhere, eg along the length of its inner or outer diameters) and glue the edges together aren't you just waving away the problem? By that logic I can slice open a sphere and call it a sheet, or conversely declare all shees to be spheres that haven't been glued together yet.
It's admittedly been a good 15 years since I cracked open a topology textbook, but the high-level, hand-wavey idea behind this sort of topological surgery isn't that you slice up manifolds and glue them together willy-nilly, but that you do so in very precise and controlled ways, which (and this is the very important bit) you've proven ahead of time preserve (or modify in a knowable fashion) some property of the manifold you care about. Rather than waving away the problem, you're decomposing it into a set of simpler ones which are ideally more tractable for some reason or another (perhaps you can compute a given property from first principles for a sphere, but not for a torus, for instance).
The wikipedia page on this stuff (https://en.wikipedia.org/wiki/Surgery_theory) is quite technical, but you might be able to squint at it and get a sense for how it works.
Network optimizing problems are just better with 4D hypercubes.
Well, shit.
>And dimension 3 is the only one that can contain knots — in any higher dimension, you can untangle a knot even while holding its ends fast.
Do we have anything in the universe that is knotted? Both on large and small scales. Or it is just coincidence?
Not wishing to be flippant, but I have lots of bits of string that are knotted.
This is some Dr Strange stuff
This got me thinking — is there a version of “in mice” for math papers?
Seeing as mathematicians proving things in math has minimal relation to the real world, I'm not sure how important this is.
Mathematicians and physicists have been speculating about the universe having more than 4 dimensions, and/or our 4 dimensional space existing as some kind of film on a higher dimensional space for ages, but I've yet to see compelling proof that any of that is the case.
Edit: To be clear, I'm not attempting to minimize the accomplishment of these specific people. More observing that advanced mathematics seems only tangentially related to reality.
You might consider reading Hardy's A Mathematician's Apology. It gives an argument for studying math for the sake of math. Personally, reading a beautiful proof can be as compelling as reading a beautiful poem and needs no further justification.
However, there is another reason to read this essay. Hardy gives a few examples of fields of math which are entirely useless. Number theory, he claims, has absolutely no applications. The study of non-euclidean geometry, he claims, has absolutely no applications. History has proven him dramatically wrong, “pure” math has a way of becoming indispensable
I have no problem studying Math just to study Math. I read the title and jumped to some conclusions, I'm afraid. Was talking to a friend about String Theory and their 11+ dimensions the other day and that is immediately where my brain went to with this one. The article is interesting even though I have zero desire to personally study math just for math's sake.
I have always been fond of the following quote by Jacobi: “Mathematics exists solely for the honor of the human mind”
There is a huge amount of mathematics that initially seems as though it could not possibly have any practical application that later turns out central to all sorts of things in the real world.
The most obvious examples are number theory and group theory, which are respectively the study of numbers and how they behave under basic operations like arithmetic, and the study of a type of set with a single operation that satisfies very basic rules[1]. How could this possibly have any relevance or practical application? And yet it turns out they are central to cryptography and information theory. Joseph Fourier trying to solve the equations that govern how heat diffuses through a metal came up with the theory that forms the basis for how we do video and audio compression (and a ton of other things).
Finally mathematicians don’t speculate about how many dimensions the universe has, they study 4- and higher- dimensional objects and spaces to understand them. This theory is used all over the place. You can’t have a function like a temperature map without 4 dimensions (3 for the spatial coordinates of your input and one for the output).
[1] this turns out (non-obviously) to be the study of symmetry.
> More observing that advanced mathematics seems only tangentially related to reality.
You might be surprised; there have proven to be a number of surprising connections between abstract mathematical structures and more concrete sciences. For instance, group theory - long thought to be an highly abstract area of mathematics with no practical application - turned out to have some very direct applications in chemistry, particularly in spectroscopy.
When you try to solve one problem involving two objects in three-dimensional space, you have a six-dimensional problem space. If you have two moving objects, you have a twelve-dimensional problem space. Higher dimensional spaces show up everywhere when dealing with real-life problems.
>Seeing as mathematicians proving things in math has minimal relation to the real world, I'm not sure how important this is.
Évariste Galois says hi and Satoshi-sensei greets him back.