Welcome 👋️

Over the last few years there has been a push away from using machine identity for continuous integration (CI) agents, or runners, and instead use a more targeted, least privileged approach to authentication and authorization. This is where OIDC (OpenID Connect) comes in, which is a method of authentication used to bridge between the CI provider and cloud services such as AWS, Azure, and Google Cloud. In this model the CI provider acts as an identity provider, issuing tokens to the CI runner/agent which include a set of claims identifying the owner, pipeline, workflow and job that is being executed. This is then used to authenticate with the cloud service, and access the resources that the pipeline, workflow and job require. ...

As an Amazon Web Services (AWS) developer, I am often asked what is the best way to organise datasets in S3. A dataset could comprise data exported by business systems, or data emitted by AWS services, such as CloudFront logs, or CloudTrail logs. Far too often I have seen datasets just dumped into one massive S3 bucket, and left for someone else to tidy up later, however with a little consideration, and empathy for those dealing with this in the future, we can do better than this. ...

As a developer who works with Go in my day-to-day development, I constantly struggle with third party packages or tools which bring in a lot of dependencies. This is especially true when you’re trying to keep your project dependencies up to date, while dependabot, and other security software, is screaming about vulnerabilities in dependencies of dependencies. This is especially a problem with two common packages I use: Any HTTP adaptor package, which ships with integrations for multiple server packages, such as Gin, Echo, and others. Any package which uses docker to test with containers. Projects which include examples with their own dependencies. To break this cycle in my own projects, and packages I publish privately in work projects, I have adopted the use of Go workspaces, which allows me to create a monorepo broken up into multiple packages, and then publish one or more of these packages. ...

As a software developer, I have seen a lot of changes over the years, however few have been as drastic as the rise of artificial intelligence. There are a growing list of tools and services using this technology to help developers with day to day tasks, and speed up their work, however few of these tools help them understand how this technology works, and what it can do. So I wanted to share some of my own tips on how to get started with AI. ...

I have been working with Amazon API Gateway for a while and one thing I noticed is there are a few options for authentication, which can be confusing to developers, and lead to security issues. This post will cover one of the common security pitfalls with API Gateway and how to mitigate it. If your using AWS_IAM authentication on an API Gateway, then make sure you set the default authorizer for all API resources. This will avoid accidental exposing an API if you mis-configure, or omit an authentication method for an API resource as the default is None. ...

Amazon Web Services (AWS) is deprecating the go1.x runtime on Lambda, this is currently scheduled for December 31, 2023. Customers need to migrate their Go based lambda functions to the al2.provided runtime, which uses Amazon Linux 2 as the execution environment. I think this is a bad thing for a couple of reasons: There is no automated migration path from existing Go Lambda functions to the new custom runtime. Customers will need to manually refactor and migrate each function to this new runtime, which this is time-consuming and error-prone. This will remove Go1.x name from the lambda console, Go will now just be another “custom” runtime instead of a first class supported language. This makes Go development on Lambda seem less official/supported compared to other languages like Node, Python, Java etc. Case in point, try searching for “al2.provided lambda” on Google and see how little documentation comes up compared to “go1.x lambda”. The migration essentially removes the branding and discoverability of Go as a Lambda language, I am sure this will improve over time, but it is still ambiguous. ...

I recently started using Terraform Cloud but discovered that the getting started tutorial which describes how to integrate it with Amazon Web Services (AWS) suggested using IAM user credentials. This is not ideal as these credentials are long-lived and can lead to security issues. What is the problem with IAM User Credentials? IAM User Credentials are long lived, meaning once compromised they allow access for a long time They are static, so if leaked it is difficult to revoke access immediately But there are better alternatives, the one I recommend is OpenID Connect (OIDC), which if you dig deep into the Terraform Cloud docs is a supported approach. This has a few benefits: ...

Recently I have been looking into automated security remediation to understand its impacts, positive and negative. As I am a user of AWS, as well other cloud services, I was particularly interested in how it helped maintain security in these environments. As with anything, it is good to understand what problem it is trying to solve and why it exists in the first place. So firstly what does automated security remediation for a cloud service do? This is software which detects threats, more specifically misconfigurations of services, and automatically remediates problems. ...

I was inspired by others to document the tools I use working as a software developer professionally, and hacking on side projects out side of work. One thing to note is in my day job I work on an Apple Mac, but my personal machine is a Linux laptop running PopOS. I find using Linux as a desktop works as most software I use is web based or supported on linux. I also use it for IoT development as pretty much all the tool chains I use supports it. ...

Billing is an integral part of day to day AWS account operation, and to most it seems like a chore, however there is a lot to be learnt interacting with AWS Billing data. So why would you ever want to dive into AWS Billing data in the first place? It is pretty easy for both novices, and experience developers to rack up a sizable bill in AWS, part of the learning experience is figuring out how this happened. The billing data itself is available in parquet format, which is a great format to query and dig into with services such as Athena. This billing data is the only way of figuring out how much a specific AWS resource costs, this again is helpful for the learning experience. The Cost Explorer in AWS is great if you just want an overview, but having SQL access to the data is better for developers looking to dive a bit deeper. The billing service has a feature which records created_by for resources, this is only available in the CUR data. If you have already you can enable it via Cost Allocation Tags. These points paired with the fact that a basic understanding of data wrangling in AWS is an invaluable skill to have in your repertoire. ...