> I don’t know if structural engineering works like this, but I do know that software engineering doesn’t.
Structural Engineering (generally construction engineering) does work like that. Following the analogy, the engineers draw; they don't lay bricks. But, all the best engineers have probably been site supervisors at some point and have watched brick being layed, and spoken to the layers of bricks, etc. Construction methods change, but they don't change as quickly as software engineering methods. There is also a very material and applicable "reality" constraint. Most struct's knowledge/heuristics remains valid over long periods of time. The software engineers' body of knowledge can change 52 times in a year. To completely stretch the analogy - the site conditions for construction engineering are better known than the site conditions for a large software project. In the latter case the site itself can be adjusted more easily, and more materially, by the engineering itself i.e. the ground can move under your feet. Site conditioning on steroids!
Ultimately, that's why I agree fully with the piece. Generic advise may be helpful, but it always applies to some generic site conditions that are less relevant in practice.
"Generic Software Design" as the author called it, is nice for setting the general direction of some implementation. This is why I like to read software engineering books. It's easier to solve a problem if you have some kind of framing to guide you. And it's easier to talk about the solution if everyone share the same terminology.
But yes, the map is not the territory, and giving directions is not the same as walking the trail. The actual implementation can deviate from the plan drafted at the beginning of the project. A good explanation is found in Naur's Theory of Programming, where he says the true knowledge of the system is inside the head of the engineers that worked on it. And that knowledge is not easily transferrable.
I completely disagree with almost the entirety of the article. It’s all about prior experience building large things many times yourself, not using some framework or other external abstraction.
When you have done this many times you absolutely can design a large application without touching the code. This is part planning and risk analysis experience and part architecture experience. You absolutely need a lot of experience creating large applications multiple times and going through that organizational grind but prior experience in management and writing high level plans is extremely helpful.
I've always felt it's unrealistic to separate upfront architecture from implementation, because my experience is a lot of systems turn out to have requirements that are a lot more complex in reality than they might seem at first, even if you think quite hard about the requirements.
Imagine if you worked for an online retailer like Amazon, and you were assigned to architect a change so you can add free sample items into customers' orders. Take a moment to think about how you'd architect such a system, and what requirements you'd anticipate fulfilling. In the next paragraph, I'll tell you what the requirements are. Or you can skip the next paragraph, the size of which should tell you the requirements are more complex than they seem.
The samples must be items in the basket, so the warehouse knows to pick them. They must be added at the moment of checkout, because that's when the order contents and weight can change. Often a customer should receive a sample only once, even if they check out multiple orders - so a record should be kept of which customers have already been allocated a given sample. It should be possible to assign a customer the same sample multiple times, in which case they should receive it once per order until they've received the assigned number. Some samples go out of stock regularly, so the sample items should not be visible to the customer when they view their order on the website, but if shipped it should appear on their receipt to assure them they haven't been charged for it. Samples should never be charged for, even if their barcode is identical to something we normally charge for. If the warehouse is unable to ship the sample, the customer should not receive a missing-item apology or a separate shipment, and the record saying that customer has had that sample already should be decremented. If the warehouse can't ship anything except the sample, the entire orders should be delayed/cancelled, never shipping the sample alone. If a customer ordered three of an item and was assigned one sample item with the same barcode but the warehouse only had three items with that barcode in stock, something sensible should happen. One key type of 'sample' is first-time-customer gifts; internal documentation should explain that if the first order a customer places is on 14-day delivery and their second order is on faster delivery and arrives first, the first-order gift will be in the second order to arrive but that's expected because it's assigned at checkout. If the first-order-checked-out is cancelled, either by the customer or the warehouse, the new-customer gift should be added to the next order they check out. Some customers will want to opt out of free samples, those who do should not be assigned any samples. But the free sample system is also used by customer services to give out token apology gifts to customers whose orders have had problems, customers who've been promised a gift should receive it even if they've opted out of free samples.
No reasonable person can design such a system upfront, because things like 'opt-out mechanism sometimes shouldn't opt you out' and 'more than one definition of a customer's first order' do not occur to reasonable people.
I would hope that a large successful online retailer got that way by factoring their implementation so that many aspects can be dealt with mostly as a configuration matter. This is mistaking quantity for difficulty. First of all, separate the domains fulfillment doesn't care about pricing, but they do care about grouping items in a shipment so they should already have had grouping rules, so apply the 'do not ship this item alone' rule, etc. The other pattern to apply repeatedly here is to separate the decision-making from effecting a change, i.e. separation of policy from mechanism. So you can have a library of mechanisms (e.g. add item to order at checkout) vs the policies which decide who, which item, and what to charge. If you don't conflate all these separate concerns as a single 'thing' to begin with, then none of the individual things is complicated, just has to be the right things in the right places.
This thought processes does use some knowledge of online retail but not really that much. It's mostly patterns of system decomposition and good engineering.
Edit: the point of the article itself stands, if the codebase is in no shape to have these free samples built as I described then my input is useless, other than to consider working toward that architectural goal.
> I don’t know if structural engineering works like this, but I do know that software engineering doesn’t.
Structural Engineering (generally construction engineering) does work like that. Following the analogy, the engineers draw; they don't lay bricks. But, all the best engineers have probably been site supervisors at some point and have watched brick being layed, and spoken to the layers of bricks, etc. Construction methods change, but they don't change as quickly as software engineering methods. There is also a very material and applicable "reality" constraint. Most struct's knowledge/heuristics remains valid over long periods of time. The software engineers' body of knowledge can change 52 times in a year. To completely stretch the analogy - the site conditions for construction engineering are better known than the site conditions for a large software project. In the latter case the site itself can be adjusted more easily, and more materially, by the engineering itself i.e. the ground can move under your feet. Site conditioning on steroids!
Ultimately, that's why I agree fully with the piece. Generic advise may be helpful, but it always applies to some generic site conditions that are less relevant in practice.
"Generic Software Design" as the author called it, is nice for setting the general direction of some implementation. This is why I like to read software engineering books. It's easier to solve a problem if you have some kind of framing to guide you. And it's easier to talk about the solution if everyone share the same terminology.
But yes, the map is not the territory, and giving directions is not the same as walking the trail. The actual implementation can deviate from the plan drafted at the beginning of the project. A good explanation is found in Naur's Theory of Programming, where he says the true knowledge of the system is inside the head of the engineers that worked on it. And that knowledge is not easily transferrable.
I completely disagree with almost the entirety of the article. It’s all about prior experience building large things many times yourself, not using some framework or other external abstraction.
When you have done this many times you absolutely can design a large application without touching the code. This is part planning and risk analysis experience and part architecture experience. You absolutely need a lot of experience creating large applications multiple times and going through that organizational grind but prior experience in management and writing high level plans is extremely helpful.
The article incurs in the very same problem it's describing. It's generic advice that might not be appliable to specific situations.
I've always felt it's unrealistic to separate upfront architecture from implementation, because my experience is a lot of systems turn out to have requirements that are a lot more complex in reality than they might seem at first, even if you think quite hard about the requirements.
Imagine if you worked for an online retailer like Amazon, and you were assigned to architect a change so you can add free sample items into customers' orders. Take a moment to think about how you'd architect such a system, and what requirements you'd anticipate fulfilling. In the next paragraph, I'll tell you what the requirements are. Or you can skip the next paragraph, the size of which should tell you the requirements are more complex than they seem.
The samples must be items in the basket, so the warehouse knows to pick them. They must be added at the moment of checkout, because that's when the order contents and weight can change. Often a customer should receive a sample only once, even if they check out multiple orders - so a record should be kept of which customers have already been allocated a given sample. It should be possible to assign a customer the same sample multiple times, in which case they should receive it once per order until they've received the assigned number. Some samples go out of stock regularly, so the sample items should not be visible to the customer when they view their order on the website, but if shipped it should appear on their receipt to assure them they haven't been charged for it. Samples should never be charged for, even if their barcode is identical to something we normally charge for. If the warehouse is unable to ship the sample, the customer should not receive a missing-item apology or a separate shipment, and the record saying that customer has had that sample already should be decremented. If the warehouse can't ship anything except the sample, the entire orders should be delayed/cancelled, never shipping the sample alone. If a customer ordered three of an item and was assigned one sample item with the same barcode but the warehouse only had three items with that barcode in stock, something sensible should happen. One key type of 'sample' is first-time-customer gifts; internal documentation should explain that if the first order a customer places is on 14-day delivery and their second order is on faster delivery and arrives first, the first-order gift will be in the second order to arrive but that's expected because it's assigned at checkout. If the first-order-checked-out is cancelled, either by the customer or the warehouse, the new-customer gift should be added to the next order they check out. Some customers will want to opt out of free samples, those who do should not be assigned any samples. But the free sample system is also used by customer services to give out token apology gifts to customers whose orders have had problems, customers who've been promised a gift should receive it even if they've opted out of free samples.
No reasonable person can design such a system upfront, because things like 'opt-out mechanism sometimes shouldn't opt you out' and 'more than one definition of a customer's first order' do not occur to reasonable people.
I would hope that a large successful online retailer got that way by factoring their implementation so that many aspects can be dealt with mostly as a configuration matter. This is mistaking quantity for difficulty. First of all, separate the domains fulfillment doesn't care about pricing, but they do care about grouping items in a shipment so they should already have had grouping rules, so apply the 'do not ship this item alone' rule, etc. The other pattern to apply repeatedly here is to separate the decision-making from effecting a change, i.e. separation of policy from mechanism. So you can have a library of mechanisms (e.g. add item to order at checkout) vs the policies which decide who, which item, and what to charge. If you don't conflate all these separate concerns as a single 'thing' to begin with, then none of the individual things is complicated, just has to be the right things in the right places.
This thought processes does use some knowledge of online retail but not really that much. It's mostly patterns of system decomposition and good engineering.
Edit: the point of the article itself stands, if the codebase is in no shape to have these free samples built as I described then my input is useless, other than to consider working toward that architectural goal.