Back to blog posts
Back to current openings

Have questions? I’m here to help.

Diliana Beeva

Talent Community Manager

Linkedin
diliana@lab08.com

NestJS AWS S3 – From an idea to reality

Apply for this position

By Martin Andreev

NestJS AWS S3 – From an idea to reality

When I started at Lab08 as a Technical Lead, my first project was a greenfield project – every programmer’s dream. You don’t need to clean other people’s code, and no depreciated and forgotten technologies. A dream come true for everyone, but especially for a guy who spent his last two years fixing legacy code.

The tech

So, long story short, we gathered the team and discussed what technologies we would use and need for such a project. After some internal brainstorming, we ended up going with:

  • NX – for managing a mono repo
  • Angular for all the front-end apps
  • NestJS for the API
  • MySQL for databases
  • MongoDB for event logging, messages, notifications, and others
  • Redis for cache and message broker for queues

The teams here at Lab08 are pretty fond of NestJS. It is a simple, Express-based, robust, and extendable NodeJS framework. Also, it lets you prototype things fast, has a growing community, and build-in modules or third-party libraries for most stuff like integrations with Redis, different types of ORMS, MongoDB, WebSockets, and more.

Another good feature of this framework is its reliance on module and dependency injection containers, as Angular heavily inspires it. This makes code separation and general separation of concern relatively straightforward.

So, after choosing the tech, the next step was… *drum roll*

The infrastructure

The obvious next step was to set up the infrastructure. 

Here at Lab08, we use AWS for almost all products we build with our clients (with a few exceptions), so naturally, we were set out to use S3 buckets to store all static files like images, pdf’s, zip files, etc.

So, now that we’ve begun the boring part.

The nightmare

Did you know that NodeJS has a lot of libraries – like an insane amount? You can find packages for almost anything, but then comes the research. Do you have to find out if the library is actively supported? Is the library stable? What version of NodeJS does it support? Are there any vulnerabilities? Are there a lot of issues? When was the last update?

Here, we came across a problem. Since NestJS is a somewhat new framework, there are actively maintained packages for the most common issues, like ORMs, WebSockets, serialization, emails, caching, etc.

However, when you need something specific, there is a high chance that you may not find it easily.

The search began. After a few hours of digging, we found a couple of NestJS packages and started exploring them. Unfortunately, we quickly found they were not well written. They were limited and not actively maintained (worked on an older version of NestJS, used the old AWS SDKs, or had a lot of open issues from months ago).

After researching, we found that we don’t have a package that fits our needs, so we decided to use the AWS NodeJS SDK v3 and create a simple wrapper to call it a service. We started a few simple methods that wrapped the AWS SDK and allowed it to swim with the rest of our code – all seemed well at this point. Magnificient.

One bite at a time

After some time, I was having lunch with a Lab08 Tech Lead from another team one day, asking me about implementing certain aspects of our system. One of these questions was about the connection with the S3 buckets.

When he started on another project at Lab08, the active package for AWS was still in v2, and he wanted to migrate to a bigger version. As such, he wanted to know if there were a lot of breaking changes. After a brief chat on the topic, we headed back to the office and discussed how our project module was implemented, what problems we encountered, what could be improved, etc.

As with all pieces of code, bugs are inevitable (disguised as unexpected features if they pop up while you are doing a demo, you know). Sometimes, updates lead to breaking changes that must be addressed. So, since most Lab08 projects share the same infrastructure, all the teams spend time maintaining a different implementation of the same module, which is required for every NestJS based project. Another essential thing to note is that the amount of time supporting this would only increase, as new projects would also have such dependencies, so something had to be done.

Open source, baby

After identifying a critical cross-dependency between projects, we decided to make our lives easier and create an official library for NestJS. As we all know, most of us programmers are pretty lazy, so we want to be able to contribute and expand a common library instead of maintaining something separately. After a quick discussion on our fundamental needs, we made a repo and paused the music.

On the first day, there was the repo

The first thing we created was the repo, and we gave it a catchy name:

https://github.com/LabO8/nestjs-s3

We decided to simply name it nestjs-s3. Straightforward but classy. Now it was time to set up everything. I had forgotten how boring it was to set up linters, git hooks, formatters, and write install instructions. Anyway, I thought: ‘No worries,’ as I knew the fun part would come when we started writing the code.

On the second day, there was the version crisis

Writing an open-source package for a popular framework is cool, but there can always be hiccups. In this case, while discussing how the module should be implemented, NestJS released its newest version, 8, which came with some minor breaking changes.

Oh well, it happens, right? After a quick discussion with the team, we decided to target the latest version of the framework and release the package as version 2.0.0. Later, we create a separate branch for version 1.0.0 that will support versions 6 and 7 of the framework. Crisis averted!

On the third day, there was the module

The fun part began as we actively made sure to make the module abstract and decide what and how to configure it. What we ended up with was a pretty simple configuration. The module can be set up in two ways – the so-called static way of using promises. It looks something like this:

S3Module.forRoot({
 region: 'eu-west-1',
 accessKeyId: 'test',
 secretAccessKey: 'test',
});

Or with a promise:

S3Module.forRootAsync({
 useFactory: () =>
   new Promise((resolve) => {
     resolve({
       region: 'eu-west-1',
       accessKeyId: 'test',
       secretAccessKey: 'test',
     });
   }),
});

Or it could even be coupled with libraries like @nestjs/config:

S3Module.forRootAsync({
 useFactory: (config) => {
   return config.get<S3Config>('config-key');
 },
 inject: [ConfigService],
 import: [ConfigModule],
});

We had a way to initialize our module and have an instance to the AWS S3 client in our dependency injection container.

On the fourth day, there were the buckets

At Lab08, we have steered away from ALBs in favor of using AWS NLB (Network Load Balancer). You might know thatS3 operates by storing objects into buckets (for those not familiar with S3, you can think of the bucket as a folder), so that was what we were targeting next.

Here was the first significant brainstorming in the team. What should we cover? Should we go full cover or go slowly if needed and not overcomplicate the service? After some discussion (and some beer), we went with the more minimal approach and implemented only the most used commands for creating, listing, tagging, and updating some configurations. (You can see the complete list here: https://labo8.github.io/nestjs-s3/api/classes/BucketsService and https://labo8.github.io/nestjs-s3/buckets-service

After we patted ourselves on the back, we had a cool new service for managing buckets. One we could simply inject anywhere we needed by:

import { Injectable } from '@nestjs/common';
import { BucketsService } from '@lab08/nestjs-s3';

@Injectable()
export class MyService {
 public constructor(private readonly bucketService: BucketService) {}
} 

After that, we decided that we needed a way to manage objects. So, we created the service of the object (https://labo8.github.io/nestjs-s3/api/classes/ObjectsService and https://labo8.github.io/nestjs-s3/objects-service), which allowed us to quickly deal with mundane tasks related to our things – a piece of cake. One just needs to inject the service, and you are done:

import { Injectable } from '@nestjs/common';
import { ObjectsService } from '@lab08/nestjs-s3';

@Injectable()
export class MyService {
 public constructor(private readonly objectsService: ObjectsService) {}
}

And now, we could easily list objects:

const result = await 
this.objectsService.listObjects('bucket-name',
options);

Or put new ones:

const result = await this.objectsService.putObject('bucket-name',
buffer, remote, options);

Or, if we were even lazier – instead of Buffer, we could just pass a local path

const result = await
this.objectsService.putObjectFromPath('bucket-name',
 '/path-to-object'/, remote, options);

We dealt with the Signed URLs in the same way: (https://labo8.github.io/nestjs-s3/signed-url-service and https://labo8.github.io/nestjs-s3/api/classes/SignedUrlService), auto remote prefixing (based on the module configuration) and because we are not fans of boilerplate, we even made a download helper ( https://labo8.github.io/nestjs-s3/download%20helper).

On the fifth day, there was the demo application

After finishing all our services, we reached the fun part – seeing our creation in action. 

We created a simple console app that will let us manage buckets and upload files directly to AWS. Since we had covered everything with tests, it all worked like a charm, and you can find it here:

https://github.com/LabO8/nestjs-s3/tree/master/demo

On the sixth day, our worst nightmare – documentation

All was good and fun until we had to write the documentation, and boy, oh boy, we programmers ‘love’ writing documentation. After some time discussing the need for (or lack thereof) documentation, we created an instance of Docusaurus and started writing.

If you need something to read before you go to bed, feel free to check it out: https://labo8.github.io/nestjs-s3/

On the seventh day, we were ready to launch

After a long and tiresome week of coding, designing, discussions, and dreadful documentation, our baby was ready to be shown to the world. With fingers crossed, we wrote the magic words git push origin master, and our idea became a reality.

You can almost hear the champagne popping as you read this, right?

Martin Andreev

Tech Lead
Linkedin

Martin Andreev is a Technical Lead at Lab08. He handles the technical design and development of complex software solutions, coaching and supporting developers, and conducting technical interviews.

Be sure to follow us on social media to receive updates about other similar content!

Back to current openings Back to current openings

Have questions? I’m here to help.

Diliana Beeva

Talent Community Manager

Linkedin
diliana@lab08.com

By Martin Andreev

OTHER ARTICLES