A real issue here is lack of training data (at least for LLMs). There's lots of high quality (and plenty more poor quality) open source software that can be used to train on. There's significantly less open source hardware and often the stuff that does exist is mostly front end design. Good examples of complete test benches (ones you'd close verification on and go to a production tape out with) are few and far between and there's basically nothing for modern physical design and backend considerations (i.e. how you take your design and actually manufacture a chip with it).
Commercial companies who may be interested in AI tools for EDA do have these things of course but are any going through the expensive process of fine tuning LLMs with them?
Indeed perhaps it's important to include a high quality corpus in pre training? I doubt anyone wants to train an LLM from scratch for EDA.
Perhaps NVidia are doing experiments here? They've got the unique combination of access to a decent corpus, cheaper training costs and in house know how.
I fine-tuned an LLM to do Verification IP wiring at a LLM hardware startup. We built the dataset in house. It was quite effective actually, with enough investment in expanding the dataset this is a totally viable application.
I'm curious: did you have to tailor your dataset around instruction-following/reasoning capabilities as well? No conflict of interest myself – I'm interested in hobby programming for vintage computers – but my understanding comes from Unsloth's fine-tuning instructions. [1]
No problem - although I'm out of that particular role, it's appropriate to discuss since the company shared these details already in an openAI press release a few months back.
I fine-tuned reasoning models (o1-mini and o3-mini) which were already well into instruction-following and reasoning behavior. The dataset I prepared was taking this into account, but it was just simple prompt/response pairs. Defining the task tightly, ensuring the dataset was of high quality, picking the right hyper parameters, and preparing the proper reward function (and modeling that against the API provided) were the keys to success.
> Indeed perhaps it's important to include a high quality corpus in pre training? I doubt anyone wants to train an LLM from scratch for EDA.
That does sound reasonable to me. The main problem is that you (at least for software) can't train on source code alone, as comments are human language, so you need some corpus of human language as well, so that the LLM learns that, next to the programming language(s). I'd assume it's the same as well.
Depending on what you're going for, you could take an existing pre-trained model, and further pretrain it on your EDA corpus. That means you'll have to reinvent or lift from somewhere else the entire finetuning data and pipeline, which is significantly harder than doing a finetune.
I would love to see a future where the barrier of entry for purpose-built chips is 100x lower. That said there's an interesting observation in the interview:
> We essentially have rolled out an L1 through L5, where L5 is the Holy Grail with fully autonomous end-to-end workflows. L1 is where we are today, and maybe heading into L2. L3 involves orchestration and then planning and decision-making. When we get to L5, we’ll be asking questions like, ‘Are junior-level engineers really needed?’
We're seeing this in the software development world too, where it's becoming harder and harder for junior engineers to both learn programing and to be successful in their careers. If the only thing that's needed are senior engineers, how do people grow to become senior engineers? It's a harrowing prospect.
> the barrier of entry for purpose-built chips is 100x lower.
You still have to wait half of year to an year to have your purpose built chips produced and shipped to you. Masks for your chip, that's what makes the whole process slow.
With FPGA, you can have your purpose built chip overnight.
Thus, in my not so humble opinion, one should use whatever means one can to make FPGAs more efficient.
This assumes that the AI growth stays exponential. This is not necessarily wrong but it is certainly not true either. If you had made that point in the 80s in regards to compilers, we would have expected software engineering jobs to have pretty much disappeared, yet the exact opposite happened.
A real issue here is lack of training data (at least for LLMs). There's lots of high quality (and plenty more poor quality) open source software that can be used to train on. There's significantly less open source hardware and often the stuff that does exist is mostly front end design. Good examples of complete test benches (ones you'd close verification on and go to a production tape out with) are few and far between and there's basically nothing for modern physical design and backend considerations (i.e. how you take your design and actually manufacture a chip with it).
Commercial companies who may be interested in AI tools for EDA do have these things of course but are any going through the expensive process of fine tuning LLMs with them?
Indeed perhaps it's important to include a high quality corpus in pre training? I doubt anyone wants to train an LLM from scratch for EDA.
Perhaps NVidia are doing experiments here? They've got the unique combination of access to a decent corpus, cheaper training costs and in house know how.
I fine-tuned an LLM to do Verification IP wiring at a LLM hardware startup. We built the dataset in house. It was quite effective actually, with enough investment in expanding the dataset this is a totally viable application.
I'm curious: did you have to tailor your dataset around instruction-following/reasoning capabilities as well? No conflict of interest myself – I'm interested in hobby programming for vintage computers – but my understanding comes from Unsloth's fine-tuning instructions. [1]
[1] https://docs.unsloth.ai/basics/datasets-guide
No problem - although I'm out of that particular role, it's appropriate to discuss since the company shared these details already in an openAI press release a few months back.
I fine-tuned reasoning models (o1-mini and o3-mini) which were already well into instruction-following and reasoning behavior. The dataset I prepared was taking this into account, but it was just simple prompt/response pairs. Defining the task tightly, ensuring the dataset was of high quality, picking the right hyper parameters, and preparing the proper reward function (and modeling that against the API provided) were the keys to success.
> Indeed perhaps it's important to include a high quality corpus in pre training? I doubt anyone wants to train an LLM from scratch for EDA.
That does sound reasonable to me. The main problem is that you (at least for software) can't train on source code alone, as comments are human language, so you need some corpus of human language as well, so that the LLM learns that, next to the programming language(s). I'd assume it's the same as well.
Depending on what you're going for, you could take an existing pre-trained model, and further pretrain it on your EDA corpus. That means you'll have to reinvent or lift from somewhere else the entire finetuning data and pipeline, which is significantly harder than doing a finetune.
I would love to see a future where the barrier of entry for purpose-built chips is 100x lower. That said there's an interesting observation in the interview:
> We essentially have rolled out an L1 through L5, where L5 is the Holy Grail with fully autonomous end-to-end workflows. L1 is where we are today, and maybe heading into L2. L3 involves orchestration and then planning and decision-making. When we get to L5, we’ll be asking questions like, ‘Are junior-level engineers really needed?’
We're seeing this in the software development world too, where it's becoming harder and harder for junior engineers to both learn programing and to be successful in their careers. If the only thing that's needed are senior engineers, how do people grow to become senior engineers? It's a harrowing prospect.
With FPGA, you can have your purpose built chip overnight.
Thus, in my not so humble opinion, one should use whatever means one can to make FPGAs more efficient.
The usual answer is "they don't".
As in: by the time this becomes an issue, AI will begin to displace senior engineers - the same way it's displacing junior engineers now.
Considering where AI was a decade ago? I'd be reluctant to bet on this happening within a decade from now, but I certainly wouldn't bet against.
This assumes that the AI growth stays exponential. This is not necessarily wrong but it is certainly not true either. If you had made that point in the 80s in regards to compilers, we would have expected software engineering jobs to have pretty much disappeared, yet the exact opposite happened.
I really don't see why anyone thinks this is a good or desirable outcome
Humans trying to build and navigate systems that they do not understand and is going to be a disaster
It's the inevitable outcome. It's not an "if". It's a "when", and "how poorly would that go".