Comment by scarface74
4 years ago
For context. My first exposure to the cloud was at my last company of 100 employees. We aggregated publicly available (ie no PII) health care provider data from all 50 states and government agencies as well as various disease/health dictionaries and we combined it with data sent to us from large health systems.
These are the services we used.
Infrastructure
- Route 53 (DNS)
- SQS/SNS (messaging)
- Active Directory.
- Cognito (SAML/SSO for our customers)
- Parameter Store/DynamoDB (configuration)
- CloudWatch (logging, monitoring, alerts, scheduling)
- Step functions (orchestration)
- Kinesis (stream processing). We were just introducing this when I left. I’m not sure what they were using it for.
CI/CD
We used GitHub for source control.
- CodePipeline (CI/CD orchestration)
- CodeBuild (Serverless builds. It would spin up a Windows or Linux Docker container and basically run PowerShell or Bash commands)
- self hosted OctopusDeploy server.
Data Storage
- S3 (Object/File storage)
- Redshift (OLAP database)
- Aurora/MySqL (OLTP RDMS). When we had large indexing to do to ELasticSearch, Read Replicas would autoscale.
- ElasticSearch
- Redis
Data Processing
- Athena (Serverless Apache Presto processing against S3)
- Glue (Serverless PySpark environment)
Compute
- EC2 (Pet VMs and one autoscaling group of VMs to process data as it came in from clients. It ran a legacy Windows process)
- ECS/Fargate (Serverless Docker cluster)
- Lambda (for processes where we needed to scale from 0 to $alot for backend processes)
- Workspaces (Windows VMs hosted in the US as Dev machines for our Indian Developers who didn’t want to deal with the latency.)
- Level 7 load balancers
Front end
- S3 (hosted static assets like html, JS, CSS. You can serve S3 content as a website.)
- CloudFront (CDN)
- WAF (Web Application Firewall)
All of the above infrastructure was duplicated for five different environments (DEV, QAT, UAT, Stage, Prod). In Prod, where needed, infrastructure was duplicated in multiple available zones (not regions).
Where applicable, backups were automated.
We had two full time operations people. The rest was maintained by developers. ——- as far as the rest.
> [Procuring resources] is no different than it is locally.
I can go from no infrastructure to everything I just named in a matter of hours locally? I can set up a multi availability zone Mysql database with automated backups just by running a yaml file locally and then turn it off when not needed?
Most of what you're listing are Layer 7 services. The time cost there is in the configuration. You can put Active Directory in the cloud, but it's still going to be Active Directory, i.e. a massively complicated proprietary framework that touches every Windows system in your network like an octopus.
And some of those things actually make sense. You can't really locally host a CDN, can you? If you need a big amount of compute for an hour and then never again, it doesn't make much sense to buy hardware for that.
But the point isn't that it never makes sense to put anything in the cloud at all. It's that companies regularly overuse it as some kind of buzzword panacea when there are only a specific set of things that it's actually good for.
It’s not just “configuration”. There is also the issue of continuous monitoring and upkeep. Not to mention someone has to worry about servers going down, hard drives going bad, backups. Would any one person know how to configure and maintain everything above?
I’m a developer who happens to have AWS in my toolbelt. I could set all that up by myself. In the the two years that I worked there, we never had an issue with any of it.
How much in house expertise would we have had to hire to manage everything that we used?
> There is also the issue of continuous monitoring and upkeep. Not to mention someone has to worry about servers going down, hard drives going bad, backups.
Monitoring and backups you configure once and then they're automated. Disk failures happen maybe a couple times a year and take five minutes to stick in another disk. None of this is particularly labor intensive.
> Would any one person know how to configure and maintain everything above?
It has long been a fact of life in companies small enough to have a single-person IT department.
1 reply →