09 juli 2018
DevOpsDays Amsterdam 2018 - conference report (deel 1)
Conference Report: DevOpsDays Amsterdam 2018 - Pakhuis de Zwijger, Amsterdam, Wednesday, Jun 27 -Jun 29 2018
The Conference location is close to Central Station Amsterdam, so using the train to get the DevOpsDays Amsterdam is the best option. After a nice walk along 't IJ from the train to the venue I arrived in time to have a coffee before a day full of workshops would start.
Workshop Day - Wednesday June 27, 2018
09:00 - 11:30 Michael Hausenblas - Go for Ops
All though I'm not a "real" developer, I'm always interested in languages that can be used for Operational tasks. Go (https://golang.org/) is the language created by google can be easily run on a big number of platforms. A very enthusiastic Michel ("i'm not here to say Go is the way to go, but i can highly recommend it") first gave a brief history of Go, talked about the structure of GOPATH and its tree and then dived into the various language features. All features addressed had code examples, and after a GIT-checkout (pull) the audience was able to run the code them self. The part about containers was mainly targeted a running Go in a container, where I hoped it was about creating containers with Go. While Michel's excitement for Go made me want to try it out, I missed the practical implementations for Operations as promised in the title of the workshop. Not a real problem, because the explanation of features was very clear and will allow you find practical use of them fast. For non-developers this was a very informative workshop about Go. And remember variables starting with a Capital Letter are public variables.
Code and Slides: https://github.com/mhausenblas/go4ops
BREAK: Time for some coffee, to greet some familiar faces and showing my workingspirit.nl black Polo-Shirt.
12:00 - 14:30 Philipp Krenn - Monitor Your Microservices — Logs, Metrics, Pings, and Traces
Philipp is an employee of Elastic (https://elastic.co/) and therefore all monitoring of microservices focused on the Elasticsearch/Kibana/Logstash/Beats/X-Pack stack. To make it a real workshop, everybody in the audience could login to a Kibana instance in the AWS cloud where the Elastic-stack was also running. Giving examples and doing them live, while the audience could do the same, created good interaction but also insights in what is possible with the Elastic-stack. The overview and examples using Zipkin (https://zipkin.io/)as an APM (Application Performance Monitoring) was a very nice addition that made me want to look more into APM. While the workshop used to new 6.3 version of the elastic-stack, I decided to upgrade my Vagrant environment for Elasticsearch from 6.2 to 6.3, unfortunately it failed. I use Alpine Linux that doesn't have glibc. Because 6.3 now has the x-packs that use glibc enabled by default, the Java program Elasticsearch fails to start.
BREAK: This break after 2.5 hours of workshop was a bit short, taking into account that the previous break was a short lunch break.
14:45 - 17:15 Bridget Kromhout - Kubernetes 101
Every participant to the workshop got three IPs and login credentials to running a kubernetes cluster in the Azure Cloud. The where second to non issues with those clusters, quite a accomplishment considering the big audience and therefore the big number of participants. This workshop of 2.5 hours is actually a workshop of 2 days. By keeping an incredible pace and skipping some less interesting topics Bridget was able to go through the 367 slides. Because I'm already bit familiar with Kubernetes I could keep up with the pace and learn something in between typing commands. And who would have thought of this 10 years ago: Bridget works at Microsoft, uses a Mac Book and is trying to Advocate Linux. The workshop is based on the material by Jérôme Petazzoni which is on GitHub and allows others to facilitate this workshop to.
Code (also a base for the slides): https://github.com/jpetazzo/container.training
THE END of the Workshop day: After 7.5 hours of workshops I was exhausted but went home with a feeling I learned a lot and should attend the workshops next year for sure.
Conference Day #1 - Thursday June 28, 2018
Au contraire the previous day, this morning there was a blue sky which made the walk along 't IJ even better. Arriving, getting a Goodie Bag and a Barista Coffee, I noticed the number of participants attending the Conference was a lot more than to the workshops yesterday. At 09:00 the conference organizers represented by Peter Nijhuis open the 6th DevOpsDays Amsterdam.
09:15 - 09:55 Bridget Kromhout - Cloud, containers, k8s
Before getting to containers Bridget explains the reason for DevOps: The "wall of confusion" between Developers, those who say "YOLO" and Operations those who say "Nope", needs to be torn down. And a pit-fall that is introduced with continuous delivery, the enabler of DevOps: Roll-backward is an illusion, Roll(Move) forward is often the only option. Trying the get the state that was before is than the challenge. And then there are containers, not new but with the introduction of Docker more usable. A new technique like containers introduce new challenges, like upgrade the base layer, and makes "things" more complex. The transformation never stops, what is now the latest, will soon be superseded. So what do you choose: take a look at the Cloud Native Foundation. But also take into account that there is a wider Ecosystem, legacy is there and most of the times it matters very much (financial systems for example). Adding an extra layer to containers: Kubernetes. Looks easy on the outside but when you take a look on the inside (Kelsey Hightower: Kubernetess the hard way) complexity rules. I very much agree more with these statements Bridget makes, adding layers is adding complexity. She ends with a request to the Nerds. Looking at IT today, it is clear the Nerds have won, a great responsibility. Infrastructure as code can and will shape the IT world: Don't screw up.
10:00 - 10:30 Serhat Can - The Rocky Path to Migrating Production Applications to Serverless Architecture
Serhat begins with stating he doesn’t hate AWS Lambda (the AWS serverless offer) even if in his talk it might look like he does. The company he works for, OpsGenie, grew from 3 to 60 people and the customer base grew likewise. While using the very scalable (as promised/advertised) AWS lambda it became a critical component, building observability into Lambda even let to a spin-off company (https://www.thundra.io/). But when things (Lambda, Java, DynamoDB and SQS) run in production, issues are there and/or going to happen. Cold Start is the time to start a Lambda function for the first time. A small memory size, the use of various VPC, a Class path scan, and the language choice (Cold start is 10s for Java) all let to Slow response times, an can lead to Function Time-outs. Solutions where found: Increase Memory (and pay more), do smart warm-ups and replace the Spring Framework. Next Issue: Scaling. Quote "Functions scale until they don't". All though unlimited was promised, 3000 concurrent events were the limit. And functions have an effect on the other function with regards to scaling. Then there is the limit to ENI and IPs that stops scaling. With the last not logged in CloudWatch and therefore took some time to discover. Which brings us to Observability. There are too many moving pieces to find a issue immediately and the is no way to attach an agent. Because events need to paid, Serhat had an issue that costed $40.000, A failing event was logged but that event failed and that event was logged..... Because in the end the invoice was the indication something went wrong, now Pricing is monitored. And last but not least: Functions can be triggered more than once, up to 3 times was observed, make sure your app can handle this. After hearing all these issues with serverless computing I felt a bit of an Grumpy Old Sysadmin. Haven't we had all these issues on servers for 20 years or more? Memory, IP space, Scaling Limits, missing logs just to name a few. This sounds to me serverless may mean there are no servers, but the problems are still there.
BREAK: Unlike the workshop day, there are vendor booths placed in two rooms of the conference. Time to get some LapTop Hipster Stickers.
10:50 - 11:20 Michael Ducy - DevOps in a Cloud Native World
Alltough the first talk of the day already explained a lot of DevOps, Michael from Sysdig added the famous CA(L)MS. Culture, Automation, (Lean), Measurement and Sharing. Where off course Culture is the most important. An environment where failure is allowed and post-mortems are shared with the whole world. So, what about Cloud Native? It fits into the ALM of CALMS and if you manage the complexity, you can unlock velocity thanks to automation. Michael suggests using a service mesh, circuit breakers and distributed tracing to solve the murder mystery. revering to the now very popular quote: "We replaced our monolith with micro services so that every outage could be more like a murder mystery". Then the statement "Cookbooks for applications are wrong, they should abstract the infra not the application" was made, I’m not sure what the implications are of this and what Michael tried to explain, but it made me think (and that is what I like about DevOps Days). Like everybody knows who is looking at DevOps longer than a week: DevOps Teams don't exist, what companies now call DevOps-teams are teams supporting Developers. Maybe DevOps-teams can be better described as: Developers Services Team, which are creating value for Developers.
11:25 - 11:55 Armon Dadgar - Service Networking for Microservices
Armon is the CTO of Hashicorp. The company is developing the open source tool Consul (https://www.consul.io/).Which now with version 1.2 as a very complete service mesh. The explanation of the need for a service mesh starts with the old way of writing applications: a big monolith. This had they advantages that internal communication was easy where external was more the issue. Security between frontend and backend could be done using strict zones. Writing microservices creates development agility but also introduces new problems/challenges. Using ephemeral cloud/container technology asks for service discovery. A register should know what runs where and when, services need to register themselves when created and be removed when destroyed. How to add security to the communications between microservices if they are ephemeral. Using network segmentation can reduce the "blast radius". A better solution is to use the register knows what needs to communication with what and can create segments. Using Kubernetes and Consul as a register creates a Service Mesh. Using Consul-client (part of the same binary as consul-server) as a sidecar proxy container in a pod, communications can be setup between pods by means of the proxies, after consulting (Aha that's why it is called Consul) with the Consul-Server. These sidecar proxies can communicate unencrypted inside the pod but can use TLS for proxy to proxy communication. outside the pods. While attending this presentation I was able to install 4 Alpine Linux VMs with a upgraded consul version 1.2 cluster on my Laptop. After disallowing communication for the only running service named "consul" I was able to get a failing cluster, because the 4 servers no longer could communicate as intended, but not very convenient.
LUNCH BREAK: Besides eating a nice salad, I was able to talk to some vendors. When I told the elastic people about my "missing glibc" problems upgrading to 6.3 they were very interested and promised to look into it. Let’s wait for the results.
13:05 - 13:35 Rachael Byrne - Don’t be a bystander, be an Incident Commander!
When Rachael from PagerDuty (who remembers having a Pager in Europe?) talks about an IC, she is addressing an Incident Manager (https://response.pagerduty.com/). Traditionally the IC is highly skilled senior technical person. But, as PagerDuty found out, that is not very scalable. A better choice is to use a non-technical group of people that rotate shift. The skills needed are mostly non-technical. but a basic understanding of the environment is needed. The communication skills need to be direct. An IC must address people by name, ask clear questions, have time management skills to time box events and be an active Listener. The IC doesn't take part technically in solving the incident and is not responsible for finding the solution. To train a new IC shadowing an experienced IC during a real incident is very helpful. The trainee feels the pain, but remains silent, and writes a real-time log. Being an IC on failure Friday where an incident is simulated can also help. After this the shadow can be reversed, meaning that during the incident the new IC is in charge, but an experienced IC is standby in the background and has a direct (background) communication channel with the new IC. When the incident is resolved, do like Americans do, Celebrate the success.
13:40 - 14:00 Ignites Day 1
Ignites are short talks about any subject, either serious or "tongue in cheek". The presentations are timeboxed (5min) using a technical solution: 20 slides that are auto-forwarded every 15 seconds, started by on hosted on the laptop of the ignite host.
- Jason Yee - Going Dutch: observaties over Nederlandse cultuur & DevOps
The Dutch have street-side windows in their homes with no curtains (gordijnen), great for Observability and transparency just like we need in DevOps monitoring. A typical Dutch snack-shop is "de FEBO" where a wall of small doors, opened by injecting money, separate the customer from a snack. Don't be FEBOps! Go Dutch!
- Joep Weijers - Extreme feedback in DevOps
As a Build Master Joep is looking for ways to give developers extreme feedback on their failed builds. Examples are presented from flashlights to drones. Drones are easily ignored by developers, they wear noicecanceling headphones all the time. Solution: add a Glock to the drone, allthough this might reduce your workforce fast.
- Matty Stratton - Talk Selection as Mockumentary Film Editing
When organising Devopsdays Chicago Matt was doing talk-submission selection. The best way to do this was using his mockumentary making skills. There he found that the prewritten story never is in the footage and that the story is created from the footage. So, don't stick to the predefined talk-submissions subject but find one in the submissions.
14:00 -17:00 Open Spaces
Open spaces is the part of the conference where the participants suggest subjects. Each subject (or a selection when there are too many) gets a room and time-slot assigned. When the time-table is finished anybody can go to any room and take part in the discussion/shouting/loving about the subject. Don't forget the Code-Of-Conduct though. While others were talking about for example Post-It-Notes (what colour to use, how to tear them and what more), I had a very nice talk with a former colleague about life the universe and everything.
THE END, BBQ Evening Event: Live BBQ-ed meat and the DevHops beer made it a great evening event on the terrace of the DevOpsDays Amsterdam venue. Felt a bit like the grumpy old Sysadmin again because a lot of conversations were about "the old days" when we had the same IT problems that now are presented as new. Took a not to late train home, tomorrow would be another intensive day.
Next week we will publish part 2.