IBM's Power11 Processor Architecture

(servethehome.com)

80 points | by ksec a day ago ago

69 comments

  • sllabres a day ago ago

    For those who have asked who runs this system: Power systems today run either "AS/400" aka iSeries, IBMs traditional Unix (AIX) or Linux (RedHat or SuSE).

    In addition to the technical parameters mentioned in the article (an IBM Red Book would certainly provide more):

    From what I know from conversations, SAP users are a large customers base for Power systems. I think the size of the possible main memory might be one reason, persistent main memory [1] could be another, in my opinion. But this feature is not new, it was available at least with Power 9. The systems are available with up to 64 TB memory (I don't know if all this memory would by available for one process, but from what I have read at least 32 TB is) and that seems to mix well with SAP HANA.

    Another reason is licensing. LPAR partition is considered hard partitioning for Oracle databases, which makes licensing much easier and, in many configurations, also more cost-effective. [2]

    Then there are several RAS features. Think of: if one core of a processor in a x86 server dies, your server dies. This is not the case in the IBM Power environment. These days many customers are happy with horizontal redundancy, but not all are equal.

    [1] https://community.sap.com/t5/technology-blog-posts-by-member...

    [2] https://oraclelicensingexperts.com/oracle-licensing-virtuali...

    • sillywalk a day ago ago

      Nitpick: it's no longer iSeries, it's the unsearchable IBM i now. I believe it goes (System/38)->AS/400->AS400e->System i->iSeries->i5->i

      > if one core of a processor in a x86 server dies, your server dies. This is not the case in the IBM Power environment.

      I believe that having a spare processor core is new to POWER11. EDIT: No. I was wrong.

      • sllabres a day ago ago

        You are right. I'm not working myself on these systems but they are often still called with their older name here due to the frequent renaming...

        Regarding the processor: yes the processor spare is new to the Power11 but features like "Processor Instruction Retry" and "Alternative processor recovery" are older. Here are documents describing the feature for old 8284-22A (S822 with Power8 CPU) [1] or even older 8204-E8A (System Power 550 with Power6 CPU) [2]

        [1] https://www.redbooks.ibm.com/redpapers/pdfs/redp5098.pdf (topic 4.3.4 and 4.3.5) [2] https://www.redbooks.ibm.com/redpapers/pdfs/redp4404.pdf

    • imglorp a day ago ago
  • fidotron a day ago ago

    I can't be the only one that simultaneously appreciates that IBM are still in the game but remains mystified as to who is actually buying and using this stuff.

    They simply have to have some farms in government running this to make it make sense.

    • CursedSilicon a day ago ago

      On the low-end. Ever shopped at Costco? It's all IBM AS/400's handling the back-end. That's why all their Windows 11 PC's have big black and green terminal apps running front and center

      On the high-end? Banks, airports, hospitals, research labs. There's a lot of places that need the kind of fault tolerance that specifically IBM POWER systems provide

      EDIT: Okay, IBM POWER "systems". They've been described as mainframes to me so I went with that terminology

      • nabla9 a day ago ago

        IBM Z mainframes use Z processors and now Telum, Telum II processors, not POWER.

        • inkyoto a day ago ago

          z Series have used both, POWER and TELUM (I and II) processors.

          For many years, the 64-bit extension of the original S/360/370/390 architecture was emulated in the software layer via the static binary translation – just like the i Series AS/400 have been doing since the inception, and there was no native S/360 implementation in silicon for a fairly long time.

          If my understanding is correct, with TELUM processors, IBM has gone back to implementing the ISA in silicon, although the available details on TELUM are scarce.

          • ch_123 18 hours ago ago

            This (and variations) is commonly believed but not the case - IBM's Z hardware has always used processors which natively implement the Z instruction set. I think part of the source of the confusion is a presentation from years ago which showed that some IP is shared between the Z and Power CPUs.

            • sillywalk 11 hours ago ago

              > a presentation from years ago which showed that some IP is shared between the Z and Power CPUs.

              The eCLIPz project, for the POWER6 & Z10[0].

              "The z10 processor was co-developed with and shares many design traits with the POWER6 processor, such as fabrication technology, logic design, execution unit, floating-point units, bus technology (GX bus) and pipeline design style, i.e., a high frequency, low latency, deep (14 stages in the z10), in-order pipeline.

              However, the processors are quite dissimilar in other respects, such as cache hierarchy and coherency, SMP topology and protocol, and chip organization."[1]

              [0] https://news.ycombinator.com/item?id=18494225

              [1] https://en.wikipedia.org/wiki/IBM_z10

          • nabla9 18 hours ago ago

            Your understanding is not correct.

            IBM z15 mainframes had z15 chip now they have Tellum. z chips are their own line. z14, z13, ...

      • wolf550e a day ago ago

        "mainframes" are z/Architecture, not POWER.

        • a day ago ago
          [deleted]
      • gosub100 a day ago ago

        I seriously doubt there's a POWER mainframe in the back of Costco to handle the 3 UPC barcode scans per second. It's possible that every Costco store funnels its orders to a single mainframe somewhere.

        I think a more realistic case is the visa and MasterCard credit card networks that have almost 0% downtime.

        • robotnikman 6 hours ago ago

          I can provide some insight into this. Most stores will have a server which handles the barcodes, pricing, etc, locally. Then all transactions are usually sent in a batch everyday from the store server to a central server/servers somewhere for processing, usually around that time reports are also generated and stats made available for BI analysis.

          Payments processed by the payment terminals handle the authorization of the payments separately from any store servers, usually through a service such as Connected Payments.

        • dardeaup a day ago ago

          No such thing as a POWER mainframe. IBM's POWER lines (i and p) are different from their mainframe line (z).

        • CursedSilicon a day ago ago

          Not a POWER Mainframe, no. But AS/400's come in many sizes. The smallest ones are roughly the size of a standard workstation

          • pjmlp a day ago ago

            Back in 1994, they were small enough like a big PC tower, I used to seat on one occasionally, that was out of order.

            One of my Summer intern jobs was to run backups every few days on a AS/400 system.

            That "seat" was the old one that was yet to be collected.

            • RaftPeople a day ago ago

              > Back in 1994, they were small enough like a big PC tower,

              Back then their model lineup ranged from the small size like you mention to approx 3 or 4 refrigerator size at the high end.

              When I did some system work out at Costco in the 90's they had 4 of the largest models connected together in one system image (sysplex I think).

              • pjmlp 21 hours ago ago

                Indeed, never saw other sizes in real, only on magazine ads.

        • gt0 a day ago ago

          Mainframes are not POWER, they are z/Architecture.

          For AS/400 (IBM i), they're POWER, but come in pretty small models like IBM Power S1012, it's available as "deskside", i.e. a big tower.

        • chiffre01 a day ago ago

          I think it's more likely they have a rack of IBM iSeries servers in the back someplace, or maybe in a colo data center.

    • nxobject a day ago ago

      For background to sibling comments - AS/400 aka "System i" was historically a separate line of processors, but is now POWER running a software translation layer.

      https://en.wikipedia.org/wiki/IBM_i#TIMI

      To illustrate why the AS/400 had its market niche, the Cali Cartel had a very successful installation of an AS/400. Apparently they used it to do both "business analytics" and back-office tasks.

      https://www.vice.com/en/article/the-cartel-supercomputer-of-...

      • ch_123 18 hours ago ago

        > For background to sibling comments - AS/400 aka "System i" was historically a separate line of processors, but is now POWER running a software translation layer.

        The software translation layer has been a feature of the platform dating back to the System/38 days, and was specifically intended to allow the CPU architecture to change without breaking software compatibility.

        IBM i also has the "PASE" layer, which is a binary compatibility layer for AIX. Those applications do not use the translation layer.

    • DebtDeflation a day ago ago

      There used to be stories about small manufacturing companies that bought AS/400s in the early 1990s, installed them in their factories, then at some point there was construction and walls were built around them, and then they were discovered 20+ years later, still running, when they were upgrading their production lines and trying to figure out what was actually running them.

    • dlcarrier a day ago ago

      It's more beuroacracies in general than government in particular.

    • pram a day ago ago

      People stuck on DB2 is the answer.

    • nine_k a day ago ago

      Google used to use POWER9 chips, but they apparently have stopped.

      I suppose the military should buy a significant amount, because they require 100% US-based development and manufacture.

      • sidkshatriya a day ago ago

        Power chips are made by Samsung according to the article. The fab for that is probably in Korea.

    • vb-8448 a day ago ago

      There are still plenty of IBM-I (formerly as400) and AIX installations around Europe.

      • Someone a day ago ago

        The statistic to look at is not how many there are, but how many get sold nowadays.

        Also, how does “plenty” compare to the millions (50 million or so, it seems from a quick search) x64s that Intel sells per year? Do they even sell 1% of that?

  • jauntywundrkind a day ago ago

    The memory improvement seems so so awesome. 3x the bandwidth, 2x capacity, via Open Memory Interface (OMI) buffers.

    Really looking forward to x86 getting similar: Intel's MR-DIMM and industry standard MCR-DIMM are both sort of happening. Intel's Granite Rapids (Xeon 6900P series) for example takes either DDR5-6400 or MRDIMM-8800, a 38% increase. And which should also allow a capacity increase, up to 256GB dimms supposedly. But so far, seems like 96GB is as big as we get, and MCR-DIMMs are still work-in-progress. https://www.phoronix.com/review/intel-xeon6-mrdimm-ddr5 https://www.serversupply.com/MEMORY/PC5-70400/96GB/MICRON/MT...

  • favorited a day ago ago

    It was z, not Power, but I really enjoyed STH's recent video going through the design of the z17 mainframe with IBM folks: https://www.youtube.com/watch?v=C8oLfMXUo0U

    • russellbeattie a day ago ago

      That was really interesting! All that proprietary hardware is mind blowing. The cost of those mainframes must be mind boggling, let alone the maintenance costs. It answers the question, "What does IBM do nowadays?"

      But one thing I never knew about before was covered in just a few seconds in the video: 40+ layer motherboard PCBs. Holy cow, it was like a half inch thick! It makes sense now that I think about it considering how many components are slotted into it, but for some reason it never dawned on me that you could have a PCB that was so huge vertically.

      • favorited a day ago ago

        Yeah, that PCB blew my mind as well! Especially how it enables them to have no wires in these systems.

  • ChuckMcM a day ago ago

    As a systems architecture dilettante I really like the Power architecture. It is old school super computer / mainframe architecture on a chip. Parallel processing architecture that makes cross domain inferences impossible etc etc.

    A crazy friend of mine had a workstation with a Power10 main board in it running Redhat Linux. Other than the fact that it used what was essentially a rack server main board so had these 8 fans that blew through the chassis like a 747 on take off it was pretty neat.

    • SlowTao a day ago ago

      Dont need a Power10 for that. Just get a Powermac G5 tower in a kernel panic and those fans will be flying!

  • doctorshady a day ago ago

    I love working with the POWER ISA, it makes me happy to see they're still making high performance chips. I really wish these were more accessible to the average user though.

    • TheAmazingRace 12 hours ago ago

      Allegedly, Raptor Computing Systems (RCS) is working with another outfit called Solid Silicon to produce a Power ISA CPU that is more accessible to mere mortals.

      Though based on what I've read so far, it all feels like vaporware.

  • MrDrMcCoy a day ago ago

    Would love to have one in my homelab, but they're functionally impossible to acquire.

    • Fade_Dance a day ago ago

      Presumably they don't run forever and are eventually upgraded... Well, let's assume they do run forever since we're talking mainframes, even then there would be the occasional upgrade. What happens to the old ones? Presumably it's not like standard servers and workstations where you end up finding 100 pages of Old Dell servers on eBay, but they have to go somewhere, no? Straight to the shredder?

      • unnah a day ago ago

        You can find quite a few used Power 8 and Power 9 servers for sale on ebay. The prices range from a thousand dollars or euros to a few thousand, so not for everyone but well within the reach of a well-off hobbyist.

  • Keyframe a day ago ago

    judging by the comments and (lack of proper) answers, the realm mystery is who are the customers?! Before, POWER excelled at bandwidth.. like real torrent of bandwidth. What's the edge these days?

  • JLO64 a day ago ago

    I hate phrasing a question like this, but I would genuinely like to know who the customer for this architecture is. My understanding is that cloud (and for that matter consumer) is all x86/ARM.

    So is it science/defense that makes up their user base? I have an acquaintance who works at NASA mention that IBM stuff was really good for processing large amounts of data fast. Is that their primary advantage over other architectures?

    • doctorshady a day ago ago

      Raptor Systems makes a bunch of ATX compatible POWER motherboards. As to who buys them, it's not entirely clear. If they were a little cheaper, I'd pick one up in a heartbeat. $3,000 for a CPU and motherboard is a bit hard to rationalize for what's effectively just nerding out though: https://www.raptorcs.com/content/base/products.html .

      For the embedded market, NXP makes a bunch of QorIQ chips based around the POWER ISA - mostly for telecom products. These are actually reasonably common in certain devices, but not really what you'd want in a desktop.

      • Palomides a day ago ago

        raptor is still on power9 chips from 2017, I have one, but it's not a reasonable purchase unless you're a very very serious open source believer

        • doctorshady a day ago ago

          My understanding is there's issues around POWER10 that stopped it from being adopted: https://www.talospace.com/2021/09/its-not-just-omi-thats-tro... . For me personally, having worked enough with embedded POWER chips to get weirdly comfortable with ISA, given the choice, I'd love to have a system like this to write pure ASM stuff on. Not for any practical reason in the slightest; pretty much in the same spirit of souping up a go kart.

          On that note, out of curiosity, how do these compare with x86 and ARM chips from that era?

      • TheAmazingRace a day ago ago

        What's funny about all this is, back when the Raptor Blackbird MicroATX board dropped in 2018, you could cobble together one of those with a quad core POWER9 CPU for a reasonable $1200 out the door. Prices went up exponentially after the pandemic due to sourcing issues and some channels used for parts drying up. A shame, really.

        I used to own one of these systems before I sold it to my dad, and I'm eagerly awaiting a next generation offering, if it ever comes out.

        • Fade_Dance a day ago ago

          What are the use cases for the second hand market such that prices can rise exponentially? Just hacking and perhaps at home testing for open source projects that need to run on the widest array of ISAs? Or is it just a scarcity thing?

          • TheAmazingRace 12 hours ago ago

            It's primarily scarcity. Unless you're in the know and are a special kind of computer geek (see Cameron Kaiser of Floodgap Gopher fame) - most folks aren't really going to know about Raptor's offerings. Very few folks are buying them new, thus very few make it on to the second-hand market.

            I really want IBM's Power ISA to have more mindshare, as I think it's a cleaner design and more straightforward than the likes of ARM or RISC-V. Unfortunately, IBM is more concerned with wringing out every last dollar from their contracts instead of pushing the ISA out as much as they can. OpenPOWER was supposed to be this initiative, but it has seemingly fizzled out to a large extent, such that NVidia and Google no longer are Platinum members and hardly anyone talks about it anymore.

    • esseph 19 hours ago ago

      Fed Gov, State Gov, mil, industrial, transportation, banking, etc.

    • wmf a day ago ago

      IBM pulled out of the HPC market several years ago.

    • dlcarrier a day ago ago

      It's customers that never migrated to x86/ARM.

    • themafia a day ago ago

      The last I checked most of the supporting mainframe systems use POWER. For example the DASD storage devices are all using POWER CPUs.

    • dmitrygr a day ago ago

      mainframes

      edit: thought some ibm mainframes used power in helper roles.

      • wmf a day ago ago

        Power isn't a mainframe.

        • dmitrygr a day ago ago

          Hm, I thought some of the peripheral controllers in IBM mainframes used power, guess i was wrong.

        • gosub100 a day ago ago

          I think you meant to say "mainframe isn't a customer" ? Power architecture runs almost exclusively on mainframes.

          • dardeaup a day ago ago

            Correction - POWER architecture runs exclusively on 'i' and 'p' series. 'z' series is mainframe.

          • wmf a day ago ago

            That still isn't correct.

  • wmf a day ago ago

    Besides the memory improvements these slides are really vague.

  • bsder a day ago ago

    Is there some technical limitation as to why IBM Power systems aren't leading the charge with AI? Or are the people using them just keeping really quiet that they have a superpower and not letting anybody know?

    It seems like these systems should be stupidly good at AI training.

    What am I missing?

    • wmf a day ago ago

      GPUs are much better for AI than CPUs and Power CPUs are also far behind Intel/AMD.

      • bsder a day ago ago

        Your statement tells me precisely nothing.

        From my, admittedly uninformed, position, AI training requires big memory and enough processing units to keep the bandwidth to that memory fully utilized--things which Power should be very good at.

  • russellbeattie a day ago ago

    How are all the features of an architecture like this utilized? Does the hardware automatically take advantage of all the cache, RAM, threads, etc., or are mainframe applications specifically written with chunks of low-level assembly to optimize for the specific features of the chip and supporting hardware? The devs that write the compilers must be geniuses.