I used to work for a not for profit. The level of legal graft was enormous.
We once got audited by a government agency. Said government agency had been extensively burdened with restrictions in its operation by lobbying from the NFP space.
After completing the audit, the gentleman running the government agency had a press release more less saying "I think it would be best if we were allowed to release our findings where they pertained to the expectations citizens have for the not for profit space, and not just where we find outright illegal behaviour. People should be able to understand exactly how much of a charities funding is used for its actual charitable purpose, and how much of its funds are effectively gifts for directors and staff"
Which sort of sums it up. Graft goes on it just finds a legal path.
I've seen situations like these before. This is why off-site backups are so very important. I've also been in the same position of providing data from a backup that someone was intentionally trying to destroy to escape responsibility.
This story even hints at a common theme that happens even when people aren't trying to destroy data - that some people will tear down whatever they inherit, then blame their predecessors for the problems that result.
But if you don’t blame them it can also backfire. I inherited a bad codebase once and tried my best to improve it. But there was only so much time. When I left the guy after me blamed me for the still bad parts immediately.
It's always interesting to me how easily corruption occurs. I always assume that accounting double checks things and so on, but I've seen so many business where someone just creates an account and money goes out and ... nobody notices for years.
I've even created automated invoices for some companies and realized that some data was missing for months. And yet they got paid significant amounts. I realized that the invoices could have been for just about anything and they would have gotten paid ...
When Robert McNamara took over Ford, accounting was so messed up, they would weigh their invoices and if the amount wasn't too far off from the expected dollars/pound ratio, they would pay it.
> Ford Motor Company was hemorrhaging millions of dollars every month. It was impossible to give an exact number because there was no accounting system. “Can you believe it?” Henry II later remembered. “In one department they figured their costs by weighing the pile of invoices on a scale”
One guy was caught doing that to the tune of $100 million to Facebook and Google. If he had stopped at $1 million or something he probably would have got away with it. I suspect others have.
That's how he got to DOD in the first place, by being the stereotypical "businessman who will clean up government." DOGE was not the first time politicans have talked about this kind of thing; it comes along every 20-30 years.
The Fog of War is definitely worth a watch. It was a fairly harrowing experience when I saw it. Actually, it is probably one of the hardest films I have watched. He clearly details the war crimes he presided over, and is open about the fact that he would have been tried as a war criminal had they lost.
He also pioneered the use of seat belts while at Ford. Which does make the morality-math a little more unusual.
To be clear, while McNamara was an advocate for seat belts, he wasn’t particularly instrumental in their development or adoption. In 1955 he introduced a paid option for Ford vehicles that included seat belts. But the option wasn’t popular with consumers, only 2% of new Fords the following year had it installed.
Saab was the first to include them as standard equipment in 1958, with Volvo introducing the modern 3-point seatbelts as standard equipment the following year. They remained unpopular in the US until after Ralph Nader’s 1965 book Unsafe At Any Speed which became a bestseller and prompted congress to pass the National Traffic & Motor Safety Act in 1966, and ultimately were made compulsory by states (presumably under a lot of lobbying pressure from insurers) starting in 1970.
The larger the usual bills, the larger the rounding-error-level amounts. I've had some fun time with a vendor recently where they just forgot to bill a few $k for months, but remembered when asked for quota increase.
I think I missed something. They later offered the guy the world to solve problems. He declined and then complains they wouldn’t provide the tools he needed.
Part of “name your price” should include whatever tools - up to and including ownership of processes.
Yeah I think something was missed. My wild speculation is that the person thats "causing issues" has a privileged position with the owners. The owners are unwilling to completely cut this person out of the business, and that is what he means when he says that the owners won't provide the tools he needed.
I’m the author of the post. I hinted, in a cryptic sentence near the end, that I necessarily had to leave out the worst parts of the story. No, no organized crime. But yes, there were people who appropriated resources that weren’t theirs and used every tool at their disposal to avoid scrutiny. To keep it vague, let’s just say some of the people involved had means that could seriously harm the businesses and their owners. And since these were primary businesses, that would have been a serious problem. The owners, knowing this, tried to find solutions but couldn’t really “afford” to remove the people involved. To be specific, in the end the owners themselves were aware of what was happening, but hoped to resolve it with a few more checks. Eventually, I realized that as long as there was enough money for everyone, they were okay with the ongoing theft.
Sounds like very straightforward tax evasion. The business brings in lots of cash, doesn't pay taxes, obscures the books enough so that there's no smoking gun. Some of the people participating in the scheme are skimming, but maybe less than the taxes would be, or maybe the owners are also implicated and would face criminal penalties themselves, so it's better for everyone just to keep it going and keep the books messy. Don't need mafiosos from TV for that.
Great read! Yeah, these days if I get asked for technical advice, I’m always glad to put good effort into suggestions. But as soon as you tell me “well I want to follow some of your advice, but I want to do this other stuff the wrong way”, I usually say “Good luck with all that!” and away I go.
Author's note: Many readers, understandably struck by the severity of the events, have speculated about the involvement of organized crime. I want to clarify that, while the situation was extremely problematic and dishonest, that wasn't the case. The "worst parts" I alluded to referred to other internal dynamics, abuses of trust, and improprieties that I prefer not to detail further for privacy reasons and to avoid weighing down the narrative.
> I even worked on translating Archivista’s interface into Italian, since it wasn’t yet localized, just to make it easier for users.
A certain Italian-speaking area is legendary for its organized crime syndicates. Would anyone be surprised if that was the case here? Kinda hard to imagine it not.
How you gonna leave out the good parts like circa<year> so we can gauge the tech available then? Also, what about the tools you used to sync/backup to owner's house? My personal query, why did you move to freebsd? was it a different application/use? This is an awesome story, our modern approach would be to install nextcloud/owncloud with collaboration and rsync/syncthing to an offsite NAS (owner's house).
As for your decision, I would have agreed to a directorship and hired a local MSP to do things the way I wanted. This would have allowed you to have your cake and eat it too. A lot of times, in these situations, all you need is trusted eyes and ears from outside the corrupted fold. This principle is used in the military and diplomatic core, there is a staffing structure, and then there is an XO, who is hired and controlled from HQ. This XO answers to HQ, not the local structure.
We're talking around 2009 — I don't recall the exact period, but that’s the era. For backups, I used rsync-based syncs and kept history by using hard links and rsync on top of those. I also had a Perl script that automated the whole thing, but I’ve long since forgotten its name.
As for the rest — I hear you, and I totally agree. But at the time, I was young and more focused on building things with healthy clients who genuinely wanted to create something good, rather than trying to salvage a situation that, honestly, was nearly beyond saving.
I switched the ALIX to FreeBSD for other tasks, and FreeBSD (with its native read only support) was perfect for the new workload.
One of the reasons why having boxes in a data center would be good.
If there was big(?) money flowing through the company regularly,
Keeping the server at the office and the backup in the owner's house
seems like a shoestring budget.
Which was way more common in the past years, esp in small companies
when "IT" was to be cheap cheap, even if there was.
But it seems that the client in this story did not worry about cost.
Want a new server? No problem, A second one (windows) no problem?
Was stuffing the box into a data center ever brought up?
They didn't seem to state a year this was occurring in, and from what is written it sounds like the internet connection was insanely slow.
Back in the 95-2010 range so many places outside of towns had pretty much no internet. Maybe you'd get a meg or two up and down. Can't do much offsite with that.
It sounds like one of the VMs was a samba file server, to serve shared files for the workstations in the office. That was a common thing to run locally in the office, to keep latency down.
Yeah, that's really a strange choice for formatting and makes it very hard to read. Not the typical practice to insert a <br> after every sentence... (that said, the post itself is a great read!)
The goal of truncating the sentences in that way was precisely to increase the suspense a bit, but I believe I miserably failed, making it just less readable.
Can you elaborate at all as to why you didn't make the phone call you eluded to that made the other person change their tone? I assume out of respect for the deceased/leaving skeletons in the closet?
Sure, I can say this. The person I would have called, someone very close to me, would have been extremely disappointed to learn what was happening. They were very proud of having helped, during difficult times, the very person who was now threatening me. And since this person close to me was facing serious health issues (though still had authority), I chose to avoid causing them further pain that, ultimately, would have been pointless at that moment.
This is needlessly negative. It’s clear that dishonest people do not always win. Disproving such a claim requires finding only one case of a successful prosecution for fraud.
The only useful definition is “breaks the rules of the game”. Using this definition one honest victory in a perfect information game disproves the assertion.
Not any individual one - a particular dishonest person might only win 20% of the time - but in aggregate - the winner is almost always a dishonest person.
Even when a game rewards honesty, dishonest people are willing to be honest if that's truly what gives them the greatest chance of winning, so they still win.
I believe they are saying that there are multiple rounds, each with different games - some with honest optimal strategies and some with dishonest optimal strategies. A dishonest person can always choose the optimal strategy for each game, but the honest person can only choose the best honest strategy. So in aggregate the dishonest person comes out ahead.
I used to work for a not for profit. The level of legal graft was enormous.
We once got audited by a government agency. Said government agency had been extensively burdened with restrictions in its operation by lobbying from the NFP space.
After completing the audit, the gentleman running the government agency had a press release more less saying "I think it would be best if we were allowed to release our findings where they pertained to the expectations citizens have for the not for profit space, and not just where we find outright illegal behaviour. People should be able to understand exactly how much of a charities funding is used for its actual charitable purpose, and how much of its funds are effectively gifts for directors and staff"
Which sort of sums it up. Graft goes on it just finds a legal path.
I've seen situations like these before. This is why off-site backups are so very important. I've also been in the same position of providing data from a backup that someone was intentionally trying to destroy to escape responsibility.
This story even hints at a common theme that happens even when people aren't trying to destroy data - that some people will tear down whatever they inherit, then blame their predecessors for the problems that result.
But if you don’t blame them it can also backfire. I inherited a bad codebase once and tried my best to improve it. But there was only so much time. When I left the guy after me blamed me for the still bad parts immediately.
Ah that reminds me of a classic Dilbert comic, The code mocking
https://www.reddit.com/r/ProgrammerHumor/comments/195lc8/whe...
(Reddit, because Dilberts creator and his website have gone off the rails)
He would blamed your new code if you rewrote. People who blame are juniors. You are not really a senior if you blame.
It's always interesting to me how easily corruption occurs. I always assume that accounting double checks things and so on, but I've seen so many business where someone just creates an account and money goes out and ... nobody notices for years.
I've even created automated invoices for some companies and realized that some data was missing for months. And yet they got paid significant amounts. I realized that the invoices could have been for just about anything and they would have gotten paid ...
When Robert McNamara took over Ford, accounting was so messed up, they would weigh their invoices and if the amount wasn't too far off from the expected dollars/pound ratio, they would pay it.
From _Go Like Hell_ by A.J. Baime (2010):
> Ford Motor Company was hemorrhaging millions of dollars every month. It was impossible to give an exact number because there was no accounting system. “Can you believe it?” Henry II later remembered. “In one department they figured their costs by weighing the pile of invoices on a scale”
Even Google evnetually caught a few people who just cold sent in invoices and found that Google would pay.
One guy was caught doing that to the tune of $100 million to Facebook and Google. If he had stopped at $1 million or something he probably would have got away with it. I suspect others have.
https://www.npr.org/2019/03/25/706715377/man-pleads-guilty-t...
I had no idea that piece of shit was associated with Ford at all.
That's how he got to DOD in the first place, by being the stereotypical "businessman who will clean up government." DOGE was not the first time politicans have talked about this kind of thing; it comes along every 20-30 years.
https://en.wikipedia.org/wiki/Whiz_Kids_(Department_of_Defen...
He talks about it in The Fog of War (which is not going to make you change your ideia about him).
The Fog of War is definitely worth a watch. It was a fairly harrowing experience when I saw it. Actually, it is probably one of the hardest films I have watched. He clearly details the war crimes he presided over, and is open about the fact that he would have been tried as a war criminal had they lost.
He also pioneered the use of seat belts while at Ford. Which does make the morality-math a little more unusual.
To be clear, while McNamara was an advocate for seat belts, he wasn’t particularly instrumental in their development or adoption. In 1955 he introduced a paid option for Ford vehicles that included seat belts. But the option wasn’t popular with consumers, only 2% of new Fords the following year had it installed.
Saab was the first to include them as standard equipment in 1958, with Volvo introducing the modern 3-point seatbelts as standard equipment the following year. They remained unpopular in the US until after Ralph Nader’s 1965 book Unsafe At Any Speed which became a bestseller and prompted congress to pass the National Traffic & Motor Safety Act in 1966, and ultimately were made compulsory by states (presumably under a lot of lobbying pressure from insurers) starting in 1970.
> He also pioneered the use of seat belts while at Ford. Which does make the morality-math a little more unusual.
Is it unusual? A surviving driver from a car wreck needs a new car. Dead ones don't.
The larger the usual bills, the larger the rounding-error-level amounts. I've had some fun time with a vendor recently where they just forgot to bill a few $k for months, but remembered when asked for quota increase.
I think I missed something. They later offered the guy the world to solve problems. He declined and then complains they wouldn’t provide the tools he needed.
Part of “name your price” should include whatever tools - up to and including ownership of processes.
Yeah I think something was missed. My wild speculation is that the person thats "causing issues" has a privileged position with the owners. The owners are unwilling to completely cut this person out of the business, and that is what he means when he says that the owners won't provide the tools he needed.
My mind immediately went to organized crime. Money laundering for people who he rather didn't know his name.
I’m the author of the post. I hinted, in a cryptic sentence near the end, that I necessarily had to leave out the worst parts of the story. No, no organized crime. But yes, there were people who appropriated resources that weren’t theirs and used every tool at their disposal to avoid scrutiny. To keep it vague, let’s just say some of the people involved had means that could seriously harm the businesses and their owners. And since these were primary businesses, that would have been a serious problem. The owners, knowing this, tried to find solutions but couldn’t really “afford” to remove the people involved. To be specific, in the end the owners themselves were aware of what was happening, but hoped to resolve it with a few more checks. Eventually, I realized that as long as there was enough money for everyone, they were okay with the ongoing theft.
Sounds like very straightforward tax evasion. The business brings in lots of cash, doesn't pay taxes, obscures the books enough so that there's no smoking gun. Some of the people participating in the scheme are skimming, but maybe less than the taxes would be, or maybe the owners are also implicated and would face criminal penalties themselves, so it's better for everyone just to keep it going and keep the books messy. Don't need mafiosos from TV for that.
Its better to know who is stealing from you (and how much) than not - sometimes the evil you know is better than the evil you dont.
> as long as there was enough money for everyone, they were okay with the ongoing theft.
Cue @patio11: The optimal amount of fraud is non-zero [0]
0. https://www.bitsaboutmoney.com/archive/optimal-amount-of-fra...
if you weren't getting paid for this - why get involved in the first place?
I've been paid for doing my job: creating the infrastructure, configuring stuff, etc.
Alright, I was thinking this was just the case of helping a friend
This looks like a clue:
"I even worked on translating Archivista’s interface into Italian, since it wasn’t yet localized, just to make it easier for users."
No, that's not a clue :-) I've just replied, clarifying this part, to the previous comment
Got it, it was disorganized crime, not organized. ;)
Yes, exactly :-D
I thought that just means they're buying him out?
Great read! Yeah, these days if I get asked for technical advice, I’m always glad to put good effort into suggestions. But as soon as you tell me “well I want to follow some of your advice, but I want to do this other stuff the wrong way”, I usually say “Good luck with all that!” and away I go.
Author's note: Many readers, understandably struck by the severity of the events, have speculated about the involvement of organized crime. I want to clarify that, while the situation was extremely problematic and dishonest, that wasn't the case. The "worst parts" I alluded to referred to other internal dynamics, abuses of trust, and improprieties that I prefer not to detail further for privacy reasons and to avoid weighing down the narrative.
> I even worked on translating Archivista’s interface into Italian, since it wasn’t yet localized, just to make it easier for users.
A certain Italian-speaking area is legendary for its organized crime syndicates. Would anyone be surprised if that was the case here? Kinda hard to imagine it not.
How you gonna leave out the good parts like circa<year> so we can gauge the tech available then? Also, what about the tools you used to sync/backup to owner's house? My personal query, why did you move to freebsd? was it a different application/use? This is an awesome story, our modern approach would be to install nextcloud/owncloud with collaboration and rsync/syncthing to an offsite NAS (owner's house). As for your decision, I would have agreed to a directorship and hired a local MSP to do things the way I wanted. This would have allowed you to have your cake and eat it too. A lot of times, in these situations, all you need is trusted eyes and ears from outside the corrupted fold. This principle is used in the military and diplomatic core, there is a staffing structure, and then there is an XO, who is hired and controlled from HQ. This XO answers to HQ, not the local structure.
We're talking around 2009 — I don't recall the exact period, but that’s the era. For backups, I used rsync-based syncs and kept history by using hard links and rsync on top of those. I also had a Perl script that automated the whole thing, but I’ve long since forgotten its name.
As for the rest — I hear you, and I totally agree. But at the time, I was young and more focused on building things with healthy clients who genuinely wanted to create something good, rather than trying to salvage a situation that, honestly, was nearly beyond saving.
I switched the ALIX to FreeBSD for other tasks, and FreeBSD (with its native read only support) was perfect for the new workload.
Italia. Money is not a problem, still they don't hire any consulting company. No organized crime involved. Sure ;-)
One of the reasons why having boxes in a data center would be good.
If there was big(?) money flowing through the company regularly, Keeping the server at the office and the backup in the owner's house seems like a shoestring budget.
Which was way more common in the past years, esp in small companies when "IT" was to be cheap cheap, even if there was.
But it seems that the client in this story did not worry about cost. Want a new server? No problem, A second one (windows) no problem?
Was stuffing the box into a data center ever brought up?
They didn't seem to state a year this was occurring in, and from what is written it sounds like the internet connection was insanely slow.
Back in the 95-2010 range so many places outside of towns had pretty much no internet. Maybe you'd get a meg or two up and down. Can't do much offsite with that.
> They didn't seem to state a year this was occurring in
Maybe it was modified in the last two hours, but at least now it says "About 16 years ago".
It sounds like one of the VMs was a samba file server, to serve shared files for the workstations in the office. That was a common thing to run locally in the office, to keep latency down.
Why the hell is there a line break after every sentence?
Yeah, that's really a strange choice for formatting and makes it very hard to read. Not the typical practice to insert a <br> after every sentence... (that said, the post itself is a great read!)
The goal of truncating the sentences in that way was precisely to increase the suspense a bit, but I believe I miserably failed, making it just less readable.
Just because they didn't see your vision doesn't mean it wasn't good. You clearly had an intent with it.
For my anecdote, it worked for me and I didn't even notice the spacing until they pointed it out.
I think it’s called ventilated prose. More commonly found in code comments.
Probably originally written for LinkedIn. The whole pointless "moral lesson" when they didn't actually achieve anything vibe fits too.
Hello my high school research paper teacher
Reading through it I had a feeling it was in Italy. I was bit sad to confirm it.
Italy, but no organized crime involved
But tax evasion, right?
Probably there was a lot the family did not know about the deceased father.
Can you elaborate at all as to why you didn't make the phone call you eluded to that made the other person change their tone? I assume out of respect for the deceased/leaving skeletons in the closet?
Sure, I can say this. The person I would have called, someone very close to me, would have been extremely disappointed to learn what was happening. They were very proud of having helped, during difficult times, the very person who was now threatening me. And since this person close to me was facing serious health issues (though still had authority), I chose to avoid causing them further pain that, ultimately, would have been pointless at that moment.
Fantastic war story. There's always like these dozen hangers-on who've made their fortune parasitizing successful people.
good read.
Creepy
> Because sometimes, dishonest people do win.
Let me fix this for you… Because always, dishonest people do win.
Good read and it would make a good short film :-)
This is needlessly negative. It’s clear that dishonest people do not always win. Disproving such a claim requires finding only one case of a successful prosecution for fraud.
Disproving “winners are always dishonest” would be a bit trickier! (Mainly because nailing a definition of “dishonest” is just too hard)
The only useful definition is “breaks the rules of the game”. Using this definition one honest victory in a perfect information game disproves the assertion.
> Because sometimes, dishonest people do win.
Dishonest people almost always win.
Not any individual one - a particular dishonest person might only win 20% of the time - but in aggregate - the winner is almost always a dishonest person.
Even when a game rewards honesty, dishonest people are willing to be honest if that's truly what gives them the greatest chance of winning, so they still win.
Is this some kind of inverse no true scotsman?
If you win by being honest that’s not dishonest.
I believe they are saying that there are multiple rounds, each with different games - some with honest optimal strategies and some with dishonest optimal strategies. A dishonest person can always choose the optimal strategy for each game, but the honest person can only choose the best honest strategy. So in aggregate the dishonest person comes out ahead.
Life has a lot of iterated games. A reputation for dishonesty can sometimes end up following you around.
Ok but people are both honest and dishonest so how do you decide what type a person is?
because dishonest people can choose to be honest
honest when it benefit you is not truly honest
> because dishonest people can choose to be honest
According to you. But why can’t the opposite be true?
> honest when it benefit you is not truly honest
Show me a person who has never been dishonest.
If a person chooses to be dishonest when that benefits them, they're a dishonest person.
Nobody is perfectly honest so this definition isn’t useful without further qualification.