I’ts 3pm on a typical Thursday and I’m sitting in a meeting with a Customer discussing how to update their Lambda runtimes using the Serverless Framework. This isn’t strange in and of itself, but we’ve recently ran into an issue that I thought would be worth writing about.
You see, up until very recently customers using the Serverless framework may have run into issues with their framework not supporting Lambda runtimes above Node16. This was the result of some hard coded values in their AWS provider plugin and the custom-resources library in all versions prior to v4. A look through their issues list on the REPO turned up this (still open) ticket.
https://github.com/serverless/serverless/issues/12307
This manifests itself as a deployment error during the ‘serverless deploy‘ execution and for my customer, was preventing them from addressing the scores of Trusted Advisor recommendations related to Lambda Deprecation.
What was interesting about this issue was the response was to wait for v4 to resolve this. Normally id say ok that makes sense lets just acknowledge the issues and wait, however reading through the v4 release notes highlighted that in v4 a new fee structure model would be introduced making the upgrade now a procurement discussion as well as a technical one for my customer.
Knowing this is turning into a larger operational effort than originally anticipated, I suggest possibly patching around the current limitation in the CICD pipeline which would buy them some time for the business to evaluate v4 more fully while still allowing them to address the operational issues.
I offer to build a sandbox environment and test this approach for them in a burner aws account which ive detailed below;
The Setup
My Customer uses serverless within a AWS Codebuild job installed using NPM so to recreate their build pipeline I quickly spin up a similar environment using the following container image;
Once this is deployed I throw together a buildspec file that replicates their build steps and drop in a serverless.yaml file that will exploit the lambda-runtime limitation as well as force the creation of a custom resource which spins up lambda in the background.
Once all IAM roles and job configurations are in place I execute the job and confirm im able to reproduce the error. Now we go looking for references to old Node versions.
using a recursive grep -Rnl /usr/local/lib/node_modules/serverless/ -e ‘node1’ I find 2 files that need some attention;
– /serverless/lib/plugins/aws/provider.js
– /serverless/lib/plugins/aws/custom-resources/index.js
The Fix
I update my buildspec file as follows (ensuring I install the appropriate serverless version first) using sed to replace any reference of node16 with node18 within their libraries.
“`
phases:
build:
commands:
– npm install -g serverless@3.38.0 –unsafe-perm
– sed -i ‘s/nodejs16.x/nodejs18.x/g’ /usr/local/lib/node_modules/serverless/lib/plugins/aws/custom-resources/index.js
– sed -i ‘s/nodejs14.x/nodejs18.x/g’ /usr/local/lib/node_modules/serverless/lib/plugins/aws/provider.js
“`
I then re-execute the job and low and behold this does the trick. The next execution of my codebuild job completed successfully. I check the logs and see all resources are now using Node 18 as intended.
Conclusion
This is 100% a tactical patch and not something that I’d suggest a customers go and do for any critical workload, however this highlights some of the realities of working with infrastructure as code DSLs and 3rd party libraries that underpin infrastructure management. DSLs aren’t always able to remain in lockstep with the pace of cloud vendors and to ensure your operational hygiene doesn’t slip too far behind, engineers may need to get creative. In this case it provides a relatively safe alternative to doing nothing or being forced down an upgrade path that has broader technical and commercial implications. For my customer, it allows them to continue to test things in a development environment in parallel to evaluating the move to Serverless v4.
3481
0
October 31, 2024
