Would it be a poor professional choice to send this to my bosses boss who’s current raison d’etre is getting our product on the cloud? I ask because I get the alert emails when we go over budget. And we always go over budget.
Make another meme, maybe like that “days without an accident” counter…. Except it’s months without going over cloud budget
Then tell him that you made it, and that it’s true.
Cloud has some great features. Important to know what they are. Also important to know if you need those features and what the cheapest and best ways to get them are.
I love the meme. Good job.
The IT managers got tired of being blamed for all server outages and want to shift some of those responsibilities. Now when there’s an outage, they can say “it’s not us, it’s AWS because they suspend our account for non-payment”.
Plus you need to hire less people to manage all services yourself. You can also avoid vendor lock in if you have z proper policy, but most managers don’t think long-term or even care. I started not caring about costs much anymore, it’s not a me problem, it’s a manager problem. I just do enough effort to choose the right setup
And you can even abuse AWS support by trying to get them to troubleshoot your code for you.
This happens all the time and it’s a pretty simple answer,: the engineering chops of the team may not be up to snuff. Too many old school sysadmins were handed “Cloud/DevOps/SRE” titles and didn’t update their skills, it’s why a lot of top companies expect their SWEs to manage resources. Hate to say but if your team isn’t capable of scaling things using all the additional options cloud providers (and open source tools) give you, many at no additional cost on top of compute/data transfer, to scale your operations to optimize for cost, and also have the ability to build things in a way that avoid vendor lock-in (IaC/Terraform/containerization, along with having someone who actually understands Cloud Architecture), then you may need better engineers. 9/10 chance your team “migrated everything to cloud” as a 1:1 match with what you were running in a DC and then went shocked_pikachu when it was more money. Additionally, have y’all factored in all the time/money spent on maintaining the server hardware, power, DC cooling, etc. too? Cloud providers just plain have better engineers than any average company, especially ones doing the whole “this is more!” dance post cloud migration.
You can absolutely do the whole 1:1 migration to cloud, but always expect things to balloon at least a bit post-migration, but then immediately work on learning all the tools these providers give you to tighten down your cloud spend. How much are you spending on disk? Would bucket storage be cheaper? RE: Containers, even if you DO go that route, do you really need Kubernetes, which will come at an additional monetary and also maintenance cost? The likely answer at least initially is a big fat “no”. Are you running every VM, even lower envs, 24/7? Is that required? If your services are not stateless, work to make them such so you can learn about scaling in the cloud, which can even be done w/ VM-based services.
I’m not even going to touch on how much more agility using cloud vs a DC gives you.
This all may sound a bit aggressive, but it’s not meant to be. It’s just, when you’ve seen this same complaint many times, ya know. This is a learning opportunity to figure out so much about how to build your environment using relevant cloud services.
I agree, that good cloud engineers can save costs in the cloud. But I also think good non-cloud engineers, can save much much more.
When you are rewriting your entire stack to leverage cloud performance, you could probably spend a similar effort for a rewrite that increases regular performance by a similar factor.
RE: Containers, even if you DO go that route…
I was under the impression, that stateless stuff without containers requires a strong vendor login (aws lambda, google functions, azure function). Are you saying, I could do stateless without vendor-lockin and without containers and without kubernetes? This is news to me. Please point me to some resources
These days there are many solutions to deploy kubernetes on a fleet of bare-metal servers, so if you use kubernetes, the option to take everything in house again is available. Distributed storage are the toughest one to setup in house but there are many mature solutions that integrate with kubernetes well these days.
is stateless possible without kubernetes? (and without vendor lock in?)
GP said:
RE: Containers, even if you DO go that route, do you really need Kubernetes, which will come at an additional monetary and also maintenance cost? The likely answer at least initially is a big fat “no”.
There are self-hosted runtime such as workerd that allows you to run your own stateless lambda-like platform. It’s kinda losing steam these days though, and everyone seems to be pushing self-hosted kubernetes as the best way to get off the cloud these days.
I’d like to subscribe to your newsletter
TLDR: skill issue
Skill, but mostly due to the company not investing in the time to train to do it right. The company just wants to start next week by saying “flip the cloud switch” and immediately see their costs go down, without any outages and without putting in due diligence.
And sometimes the CEO/CIO/manger is too busy to coordinate training because the decision maker is busy on their “cloud provider training” for only them, in a Swiss Alps super swanky spa and resort.
Sounds like another skill issue, why aren’t you the CTO? Not throwing around enough buzz words
Agree with all your points. If I had a penny for every time lift-and-shift is mentioned when migrating to cloud from on-prem…
If your services are not stateless, work to make them such so you can learn about scaling in the cloud, which can even be done w/ VM-based services. how much more agility using cloud vs a DC gives you
This can’t be understated. Embracing elastic idology to remove single points of failure and decoupling stateful aspects of applications has been the biggest takeaway of being part of several migrations of services to AWS. Implementing these into your practices as you grow is a huge benefit that may is worth the cost.
Over time, if the scale you’re operating at grows, using experience/knowledge from AWS and applying it to running services in a datacenter could be beneficial. In my experience, if you have a large, consistent, asynchronous workload which you’ve maxed out on reserved instances or savings plans, it is likely cheaper to operate on your own hardware than in the cloud (or get credits from GCP or Azure to migrate services to reduce costs). This is where avoiding vendor lock-in is key.
have y’all factored in all the time/money spent on maintaining the server hardware, power, DC cooling, etc. too?
For sure, this isn’t 2007 where you need to purchase servers and network equipment to start a website. For most startups and small businesses, operating in the cloud will be less expensive upfront and likely over the first 3 years. This isn’t a one size fits all approach though, and it’d be prudent to evaluate the cloud spend periodically and compare with what’d it’d cost to manage it entirely. Obviously you’d need a team competent enough to manage this, without it going to shit.
I’m a big fan of proxmox
I going find it hilarious that a small amount of cloud costs the same as a hugely resourced hyper converged proxmox private cloud
I’m currently using both but proxmox is the powerhouse
Honestly I was told when I was younger it was often cheaper and easier to run things in the cloud, doesn’t really seem like that’s true anymore if it ever was
Most of the time, management is looking for the next “silver bullet” that is going to magically solve all their problems. They will latch onto the latest marketing gimmick and run with it despite having no understanding of how the “silver bullet” works or the impact it will have on their business. A decade and a half ago it was “the cloud”. Now It’s “AI”.
Are there advantages to “hosted solutions” AKA “the cloud” AKA renting someone else’s data center? Sure there are. For example, It’s great for small businesses who need enterprise grade technology but can’t afford their own data center. Cloud providers also offer services and scale that would be very difficult and costly to build out in your own data center(s).
But is it cheaper all the time? The answer to that is a definite “No”. Like most other business decisions it’s situational and there are a lot of facets that impact the cost. In my experience, one downside to hosted solutions is that it’s very easy to make architectural errors that have high costs and no one notices until accounting is on the phone wanting to know why the Azure bill doubled over the last month and “Whoops!” Is not really a satisfactory answer.
Oh no, there’s no shortcut around actually thinking! That means you have to properly understand your own job!
They push the idea of handing no CapEx, only OpEx! Think of the savings! They then try to make things look cheap like it’s super cheap.
I was doing a test in Azure and forgot to delete the insurance… I’m just a could of days… $30. But my crappy repurposed gaming PC too home server… I can do whatever I want when I want and never have to consider cost.
Except for the electricity bill because someone forgot to shutdown the mining VM :p
At least you can hear the fans spinning too fast and do something about it.
Worth it if I never have to negotiate another colo contract or have to spec new servers 9 months in advance ever again.
Specing out server is a real pain. (not sarcasm)
Yup or colo is full and won’t accept any more changes. So one has to wait until someone else gives up space and then a fire sale on the newly freed space.
Ran into this. Was constantly denied time to properly load test and configure things. So it all went in with default values and high resources. Then they got the bill, throttled everything down, and then normal compute processing was missing SLAs measured in half-days.
But look on the bright side. Every minute of the day programmers were typing, creating value, instead of wasting company money reading or thinking.
Sadly, reading and thinking, which is half of a programmers job, is often undervalued
a programmer’s* job
I’d you just try to “lift and shift” to the cloud instead of engineering a solution that fits your needs, then you won’t find cost savings or risk reduction (and like you mentioned in the meme, vendor lock in can even increase risk) which makes it pointless, it does have its place but it’s often a ham fisted and half baked bill of goods sold to the bean counters instead of the infrastructure and dev teams and is worse in the long run
Honestly, the major issue is that they don’t pay us enough for us to stay longer than 2-3 years. When I was brought onto my job, I was having the hardest time navigating the home grown primitive system they custom built. The people who created this system were no longer working at the company. I was left trying to figure out a system that was custom built, semi functional and poorly maintained using an old confluence and Google doc.
I left the company after 3 years leaving in place of my own custom code, and my own sets of instructions for the next sap that has to try and figure it out.
That hat makes me wonder, can elephants get sunburned?
“Animals with pale skin and little fur are most at-risk for sunburns. A few examples are pigs, rhinos, and elephants.” Source
This is why you are elephants cover their backs in mud, it acts as a sunscreen and also helps to keep them cool.
Huh, TIL! Also interesting about hippo sweat and giraffe tongues.
Off to found an elephant parasol charity. /s
Elephants do throw loose dirt on themselves. It was either to help reduce the sun rays, or to help keep mosquitoes off - sorry I can’t remember.
