Kuros.in

Real-time Data Streaming with Kafka Connect, ksqlDB, and MySQL using Docker Compose

2023-04-10T00:00:00+05:30

Kafka is a popular distributed streaming platform that can be used for building real-time streaming data pipelines and applications. Kafka Connect is a tool that makes it easy to integrate Kafka with other systems. In this blog post, we’ll show you how to use Kafka Connect to connect Kafka with MySQL.

Setup

Before we can connect Kafka with MySQL, we need to set up the required services. We’ll use Docker and Docker Compose to do this. Create a file called docker-compose.yaml with the following contents:

version: '3'

services:
  zookeeper:
    image: confluentinc/cp-zookeeper:7.0.0
    hostname: zookeeper
    container_name: zookeeper
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181

  kafka:
    image: confluentinc/cp-kafka:7.0.0
    hostname: kafka
    container_name: kafka
    ports:
      - "9092:9092"
    depends_on:
      - zookeeper
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

  schema-registry:
    image: confluentinc/cp-schema-registry:7.0.0
    hostname: schema-registry
    container_name: schema-registry
    ports:
      - "8081:8081"
    depends_on:
      - kafka
    environment:
      SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL: zookeeper:2181
      SCHEMA_REGISTRY_HOST_NAME: schema-registry
      SCHEMA_REGISTRY_LISTENERS: http://0.0.0.0:8081
      SCHEMA_REGISTRY_DEBUG: "true"

  mysql:
    image: mysql:5.7
    hostname: mysql
    container_name: mysql
    ports:
      - "3306:3306"
    environment:
      MYSQL_ROOT_PASSWORD: root
      MYSQL_DATABASE: my_database

This will create four services:

zookeeper: A ZooKeeper service that Kafka depends on.
kafka: A Kafka service that will be used to send and receive messages.
schema-registry: A schema registry service that will be used to manage Avro schemas.
mysql: A MySQL service that will be used to store data.

Run the following command to start the services:

docker-compose up -d

Configure Kafka Connect

Now that we have set up the required services using Docker, let’s move on to configuring Kafka Connect to connect Kafka with MySQL.

Create a file called mysql.properties with the following contents:

name=mysql-source
connector.class=io.confluent.connect.jdbc.JdbcSourceConnector
tasks.max=1
connection.url=jdbc:mysql://mysql:3306/my_database
connection.user=root
connection.password=root
topic.prefix=mysql-
mode=timestamp+incrementing
incrementing.column.name=id
timestamp.column.name=updated_at
validate.non.null=false

In this file, we have defined the following properties:

name: The name of the connector.
connector.class: The class of the connector to be used.
tasks.max: The number of tasks to be used by the connector.
connection.url: The JDBC URL for connecting to the MySQL instance running in our Docker container.
connection.user: The username to connect to the database.
connection.password: The password to connect to the database.
topic.prefix: The prefix to be used for the topics that will be created.
mode: The mode of the connector. In this case, we’re using the timestamp+incrementing mode, which tracks both the timestamp and an incrementing column to detect changes in the database.
incrementing.column.name: The name of the column used to detect changes in the database.
timestamp.column.name: The name of the column used to track the timestamp of the changes.
validate.non.null: Should validate non-null columns in the database.

Start the Kafka Connect service

With the configuration file ready, let’s start the Kafka Connect service. To do this, run the following command:

docker run -d \
    --name=kafka-connect \
    --network=kafka-connect-network \
    -e CONNECT_BOOTSTRAP_SERVERS=kafka:9092 \
    -e CONNECT_REST_ADVERTISED_HOST_NAME=kafka-connect \
    -e CONNECT_REST_PORT=8083 \
    -e CONNECT_GROUP_ID=kafka-connect-group \
    -e CONNECT_CONFIG_STORAGE_TOPIC=kafka-connect-configs \
    -e CONNECT_OFFSET_STORAGE_TOPIC=kafka-connect-offsets \
    -e CONNECT_STATUS_STORAGE_TOPIC=kafka-connect-status \
    -e CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR=1 \
    -e CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR=1 \
    -e CONNECT_STATUS_STORAGE_REPLICATION_FACTOR=1 \
    -e CONNECT_PLUGIN_PATH=/usr/share/java \
    -v /path/to/mysql-connector-java-8.0.23.jar:/usr/share/java/kafka-connect-jdbc/mysql-connector-java-8.0.23.jar \
    confluentinc/cp-kafka-connect:7.0.0

This command will start the Kafka Connect service and configure it to use the Kafka service we set up earlier. We’ve also mounted the MySQL Connector/J JAR file to the plugin path so that Kafka Connect can use it.

Create a Kafka topic

Before we can start streaming data from MySQL to Kafka, we need to create a Kafka topic. We can do this using the Kafka command line tool. Run the following command to create a topic called mysql-employees:

docker exec kafka \
    kafka-topics \
    --create \
    --topic mysql-employees \
    --partitions 1 \
    --replication-factor 1 \
    --if-not-exists \
    --zookeeper zookeeper:2181

This will create a new topic called mysql-employees with one partition and a replication factor of one.

Start streaming data

Now that everything is set up, we can start streaming data from MySQL to Kafka. Run the following command to start the MySQL connector:

curl -X POST \
    -H "Content-Type: application/json" \
    --data '{ "name": "mysql-source", "config": { "connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector", "tasks.max": 1, "connection.url": "jdbc:mysql://mysql:3306/my_database", "connection.user": "root", "connection.password": "root", "topic.prefix": "mysql-", "mode": "timestamp+incrementing", "incrementing.column.name": "id", "timestamp.column.name": "updated_at", "validate.non.null": false } }' \
    http://localhost:8083/connectors

This will start the MySQL connector and begin streaming data from the MySQL employees table to the Kafka topic we created earlier.

Verify data is streaming

To verify that data is streaming from MySQL to Kafka, let’s consume the data from the mysql-employees topic. Run the following command to consume the data:

docker run --rm \
    --network=kafka-connect-network \
    confluentinc/cp-kafkacat \
    kafkacat -b kafka:9092 \
    -t mysql-employees \
    -C -q \
    -o beginning

This command will start consuming data from the mysql-employees topic from the beginning. If there is data in the table, you should see it streaming in the console.

ksqlDB

ksqlDB is a powerful tool that allows you to process and analyze Kafka streams in real-time.

Install

To install ksqlDB, we’ll need to create a Docker Compose file that includes the ksqlDB service. Create a file called docker-compose.yml with the following contents:

version: '3'
services:
  ksqldb-server:
    image: confluentinc/ksqldb-server:0.21.0
    hostname: ksqldb-server
    container_name: ksqldb-server
    network_mode: host
    environment:
      KSQL_CONFIG_DIR: "/etc/ksql"
      KSQL_LOG4J_OPTS: "-Dlog4j.configuration=file:/etc/ksql/log4j-rolling.properties"
    volumes:
      - ./ksql-server:/etc/ksql
    depends_on:
      - kafka
    ports:
      - 8088:8088
  ksqldb-cli:
    image: confluentinc/ksqldb-cli:0.21.0
    container_name: ksqldb-cli
    network_mode: host
    tty: true
    depends_on:
      - ksqldb-server

This compose file defines two services: ksqldb-server and ksqldb-cli. ksqldb-server is the ksqlDB service, and ksqldb-cli is the command-line interface for interacting with the ksqlDB service.

Note that we’re using the network_mode: host setting to allow the services to communicate directly with the Kafka service without any additional configuration.

Start ksqlDB

With the docker-compose.yml file ready, let’s start the ksqlDB service. Run the following command to start the services:

docker-compose up -d

This command will start the ksqldb-server service and the ksqldb-cli service. Once the services are running, you can interact with the ksqlDB service using the ksqldb-cli container.

Connect ksqlDB to Kafka

Before we can use ksqlDB to process our Kafka streams, we need to connect it to the Kafka service. To do this, we’ll need to create a ksql.properties file that defines the Kafka bootstrap servers. Create a file called ksql.properties with the following contents:

bootstrap.servers=kafka:9092

This file specifies that ksqlDB should connect to the Kafka service at kafka:9092.

Create a ksqlDB stream

With ksqlDB connected to Kafka, we can now create a stream that represents the data in our MySQL table. Run the following command to create a ksqlDB stream:

CREATE STREAM employees_stream (
  id INT,
  name VARCHAR,
  age INT,
  salary FLOAT,
  updated_at BIGINT
) WITH (
  KAFKA_TOPIC='mysql-employees',
  VALUE_FORMAT='AVRO'
);

With this command, we’re creating a new stream called employees_stream with five fields: id, name, age, salary, and updated_at. We’re using the AVRO format, and we’re defining the schema for the stream using the employees_stream.avro file we created earlier.

Query the ksqlDB stream

Now that we have a ksqlDB stream representing the data in our MySQL table, we can use ksqlDB to query and process the data in real-time. Run the following command to query the employees_stream:

SELECT name, age, salary FROM employees_stream;

This command will select the name, age, and salary fields from the employees_stream. If there is data in the stream, you should see it streaming in the console.

Conclusion

In this tutorial, we’ve seen how to use Kafka Connect to stream data from MySQL to Kafka, and how to use ksqlDB to process and analyze the data in real-time. Kafka Connect provides an easy way to connect data sources to Kafka, and ksqlDB provides a powerful tool for processing and analyzing Kafka streams.

By leveraging these tools, you can build powerful real-time data pipelines that allow you to process and analyze data as it streams into Kafka. Whether you’re building real-time analytics dashboards, or processing large volumes of data in real-time, Kafka Connect and ksqlDB provide the building blocks you need to get started.

React hooks in typescripts

2023-04-02T00:00:00+05:30

React Hooks are a recent addition to React that allows developers to add state and other React features to functional components. Typescript, a statically typed superset of JavaScript, adds more power and reliability to the development process by providing type checking during compile time.

In this blog post, we’ll go over some common React Hooks and explore how they can be used with Typescript.

useState Hook

The useState Hook is used to add state to functional components. It returns an array containing the current state value and a function to update the state. The type of the state value can be inferred from the initial value passed to useState or explicitly declared using Typescript.

Suppose you have a simple form with two input fields, a name and an email. You can use the useState Hook to manage the state of the input fields, and update them as the user types.

import React, { useState } from "react";

const Form: React.FC = () => {
  const [name, setName] = useState("");
  const [email, setEmail] = useState("");

  const handleNameChange = (event: React.ChangeEvent<HTMLInputElement>): void => {
    setName(event.target.value);
  };

  const handleEmailChange = (event: React.ChangeEvent<HTMLInputElement>): void => {
    setEmail(event.target.value);
  };

  const handleSubmit = (event: React.FormEvent<HTMLFormElement>): void => {
    event.preventDefault();
    console.log(`Name: ${name}, Email: ${email}`);
  };

  return (
    <form onSubmit={handleSubmit}>
      <label htmlFor="name">Name:</label>
      <input type="text" id="name" value={name} onChange={handleNameChange} />

      <label htmlFor="email">Email:</label>
      <input type="email" id="email" value={email} onChange={handleEmailChange} />

      <button type="submit">Submit</button>
    </form>
  );
};

export default Form;

In the example above, the useState Hook is used to handle the state of the name and email input fields, and we assign an initial empty string value to each state variable.

Two functions are created to update the state variables based on user input in the input fields, which are then integrated into the onChange handlers of the input elements.

Finally, the handleSubmit function is implemented to print the current values of the name and email state variables when the form is submitted.

By using the useState Hook, we are able to manage and update the simple state in our components as the user interacts with it.

useEffect Hook

The useEffect Hook is used to handle side effects in functional components. It’s similar to the componentDidMount, componentDidUpdate, and componentWillUnmount lifecycle methods in class components. The useEffect Hook takes two arguments: a function that contains the side effect logic and an array of dependencies.

import React, { useState, useEffect } from "react";

const Timer: React.FC = () => {
  const [seconds, setSeconds] = useState(0);

  useEffect(() => {
    const intervalId = setInterval(() => {
      setSeconds((prevSeconds) => prevSeconds + 1);
    }, 1000);

    return () => clearInterval(intervalId);
  }, []);

  return (
    <div>
      <h1>Timer: {seconds}s</h1>
    </div>
  );
};

export default Timer;

In this example, the useEffect Hook is used to create a timer that updates the number of seconds displayed in the component every second.

The useEffect Hook requires two arguments: a function to execute after the component has rendered, and an optional array of dependencies.

The first argument is an anonymous function that sets up a timer using the setInterval function, and updates the seconds state variable every second using the setSeconds function.

The second argument is an empty array, indicating that this effect should only be executed once when the component mounts and not again unless the component is unmounted and mounted again.

In addition, the useEffect Hook provides a cleanup function that clears the interval to prevent memory leaks when the component is unmounted.

Lastly, the current value of the seconds state variable is displayed in the component.

Using the useEffect Hook allows for performing actions after rendering, such as setting up timers, fetching data, or subscribing to events.

useContext Hook

The useContext Hook is used to access data from the React context in functional components. It takes a context object created by the React.createContext method and returns the current context value.

Suppose you have a theme that you want to apply to your entire application. You can create a ThemeContext using the createContext method and provide it with a default theme. Then, you can use the useContext Hook to access the current theme value in your components.

import React, { useContext } from "react";

interface Theme {
  backgroundColor: string;
  color: string;
}

const defaultTheme: Theme = {
  backgroundColor: "white",
  color: "black"
};

const ThemeContext = React.createContext(defaultTheme);

const App: React.FC = () => {
  return (
    <ThemeContext.Provider value={defaultTheme}>
      <div>
        <Header />
        <Main />
        <Footer />
      </div>
    </ThemeContext.Provider>
  );
};

const Header: React.FC = () => {
  const theme = useContext(ThemeContext);

  return (
    <header style=>
      <h1>My App</h1>
    </header>
  );
};

const Main: React.FC = () => {
  const theme = useContext(ThemeContext);

  return (
    <main style=>
      <p>Welcome to my app!</p>
    </main>
  );
};

const Footer: React.FC = () => {
  const theme = useContext(ThemeContext);

  return (
    <footer style=>
      <p>Copyright © 2023</p>
    </footer>
  );
};

export default App;

In the above example, a ThemeContext is established using the createContext function with a default theme provided. The useContext Hook is then utilized in the Header, Main, and Footer components to retrieve the current theme value. By doing so, the same theme can be applied to all components without the need to pass it down via props.

useReducer Hook

The useReducer Hook is used to manage state in more complex components. It takes a reducer function and an initial state value as arguments and returns an array containing the current state value and a dispatch function to update the state.

Suppose you have a simple counter that you want to increment or decrement by clicking buttons. You can use the useReducer Hook to manage the state of the counter, and handle different actions like incrementing or decrementing the counter value.

import React, { useReducer } from "react";

interface CounterState {
  count: number;
}

interface CounterAction {
  type: string;
  payload?: any;
}

const initialState: CounterState = {
  count: 0
};

const reducer = (state: CounterState, action: CounterAction): CounterState => {
  switch (action.type) {
    case "INCREMENT":
      return { ...state, count: state.count + 1 };
    case "DECREMENT":
      return { ...state, count: state.count - 1 };
    case "RESET":
      return initialState;
    default:
      return state;
  }
};

const Counter: React.FC = () => {
  const [state, dispatch] = useReducer(reducer, initialState);

  const handleIncrement = (): void => {
    dispatch({ type: "INCREMENT" });
  };

  const handleDecrement = (): void => {
    dispatch({ type: "DECREMENT" });
  };

  const handleReset = (): void => {
    dispatch({ type: "RESET" });
  };

  return (
    <div>
      <h2>Count: {state.count}</h2>
      <button onClick={handleIncrement}>Increment</button>
      <button onClick={handleDecrement}>Decrement</button>
      <button onClick={handleReset}>Reset</button>
    </div>
  );
};

export default Counter;

In the example shown above, a reducer function is created to handle various actions such as incrementing, decrementing, or resetting the counter value. To manage the counter state, the useReducer Hook is used along with the reducer function and the initial state.

Three functions are also created that dispatch different actions to the reducer function, which are then used in the onClick handlers of three buttons. Finally, the current count value is displayed using the state variable returned by useReducer.

This approach enables managing more complex state in components using the useReducer Hook and handling different actions in a type-safe manner.

Conclusion

React Hooks offer an intuitive and powerful way to incorporate state and other React features into functional components. When paired with Typescript, they provide even greater reliability and type safety to the development process.

This blog post has explored several common React Hooks and demonstrated how they can be used effectively with Typescript. We have demonstrated how the useState Hook can be utilized to add state to a component, how the useEffect Hook can be utilized to manage side effects, how the useContext Hook can be utilized to access data from the context, and how the useReducer Hook can be utilized to handle more complex state. By utilizing these tools, developers can create more efficient, reliable, and type-safe React applications.

Using Amazon Cognito with NestJS for User Authentication and Management

2023-04-02T00:00:00+05:30

Amazon Cognito is a user authentication and management service that makes it easy to add sign-up, sign-in, and access control to your web and mobile applications. In this blog post, we will explore how to use Amazon Cognito with NestJS to manage user authentication and authorization.

Prerequisites

Basic knowledge of NestJS framework.
An AWS account with permissions to create a Cognito User Pool and App Client.

Setup

Set up a Cognito User Pool

To use Amazon Cognito with NestJS, we need to first set up a user pool in the AWS console. A user pool is a user directory in Amazon Cognito. It allows users to sign up and sign in to our application.

To create a user pool, navigate to the Amazon Cognito console, and click on Manage User Pools. Then click on Create a user pool button. Fill in the details like pool name, policies, attributes, and other configuration as required. Finally, create the user pool.

Create an App Client

Now that we have created a user pool, we need to create an app client to enable our NestJS application to interact with the user pool. To create an app client, navigate to the App clients tab in the user pool settings, and click on Add an app client button. Fill in the details like app client name, app client settings, and other configuration as required. Finally, create the app client.

Install Required Libraries

To use the Cognito service in our NestJS application, we need to install the following packages:

amazon-cognito-identity-js
@aws-sdk/client-cognito-identity-provider Run the following command to install the required packages:

npm install amazon-cognito-identity-js @aws-sdk/client-cognito-identity-provider

Configure AWS Cognito in NestJS

We will now create a service to manage user authentication and authorization using Cognito in our NestJS application. Create a new file named aws-cognito-auth.service.ts and add the following code to it:

import { BadRequestException, HttpException, HttpStatus, Injectable } from '@nestjs/common';
import { AwsCognitoAuthConfig } from '../../aws-cognito-auth.config';
import {
    AuthenticationDetails,
    CognitoUser,
    CognitoUserAttribute,
    CognitoUserPool,
    CognitoUserSession,
    ISignUpResult,
} from 'amazon-cognito-identity-js';

import { RegisterRequest } from '../../model/register.request';
import { LoginRequest } from '../../model/login.request';
import {
    AdminDeleteUserCommand,
    CognitoIdentityProviderClient,
    GetUserCommand,
    ChangePasswordCommand,
} from '@aws-sdk/client-cognito-identity-provider';
import { Optional } from '@connect/common-utils';

@Injectable()
export class AwsCognitoAuthService {
    private readonly userPool: CognitoUserPool;
    private readonly cognitoIdentityServiceProvider: CognitoUserSession
    private readonly providerClient: CognitoIdentityProviderClient;
    constructor(private readonly authConfig: AwsCognitoAuthConfig) {
        this.userPool = new CognitoUserPool({
            UserPoolId: this.authConfig.userPoolId,
            ClientId: this.authConfig.clientId,
        });

        this.providerClient = new CognitoIdentityProviderClient({
            region: this.authConfig.region
        });
    }

    // Add methods to manage user authentication, registration, and authorization here.
}

In the code above, we have created an AwsCognitoAuthService service.

Register User

Now that we have our service skeleton ready, its time to add method to register user:

To register a new user we want the user to use their email to as login & we also need to verify the email address.

Add User

First we will add a user, cognito will send a mail to users with the verification code (this is as per settings in cognito).

    registerUser(registerRequest: RegisterRequest): Promise<ISignUpResult> {
        const { email, password } = registerRequest;
        return new Promise((resolve, reject) => {
            return this.userPool.signUp(
                email,
                password,
                [new CognitoUserAttribute({ Name: 'email', Value: email })],
                null,
                (err, result) => {
                    if (err) {
                        reject(err);
                    } else {
                        resolve(result);
                    }
                },
            );
        });
    }

The registerUser function is responsible for registering a user with the Cognito User Pool. It receives a RegisterRequest object containing the email and password of the user.

To accomplish this task, the function utilizes the userPool object, which is created during the initialization of the constructor. It then invokes the signUp method by passing in the email and password provided, along with a new instance of the CognitoUserAttribute object. The attribute object specifies the email attribute to be passed to the Cognito service during registration.

Upon successful registration, the signUp method returns a promise that resolves with an ISignUpResult object, containing information about the newly registered user. If the registration fails, the promise is rejected with an error object.

The function uses a new Promise object to wrap the signUp method, allowing for convenient error handling using the reject and resolve methods. If the signUp method resolves successfully, the resolve method is called with the ISignUpResult object. In the case of an error, the reject method is called with the error object.

Verify user using email

The verifyUser function is responsible for verifying the registration of a user with the provided verification code. It takes in two parameters: the user’s email and the verification code.

    verifyUser(email, verificationCode) {
        return new Promise((resolve, reject) => {
            return new CognitoUser({ Username: email, Pool: this.userPool})
                .confirmRegistration(verificationCode,
                    true,
                    (err, result) => {
                        if(err) {
                            reject(err);
                        } else {
                            resolve(result);
                        }
                })
        });
    }

To handle the asynchronous nature of the confirmation process, the function returns a new Promise object that resolves with a successful confirmation or rejects with an error.

The function creates a new instance of the CognitoUser object using the provided email and the userPool object created during the constructor initialization. It then calls the confirmRegistration method of the CognitoUser object, passing in the verification code and a boolean flag to force alias creation if necessary. The callback function takes two parameters, err and result.

If there is an error during the confirmation process, the promise is rejected with the err object. If the confirmation process is successful, the promise is resolved with the result object.

Authenticate User

Create a function that will be used to authenticate a user with the Cognito User Pool. It takes a LoginRequest object as input, which contains the user’s email and password.

    authenticateUser(loginRequest: LoginRequest) {
        const { email, password } = loginRequest;

        const authenticationDetails = new AuthenticationDetails({
            Username: email,
            Password: password,
        });

        const userData = {
            Username: email,
            Pool: this.userPool,
        };

        const newUser = new CognitoUser(userData);

        return new Promise<CognitoUserSession>((resolve, reject) => {
            return newUser.authenticateUser(authenticationDetails, {
                onSuccess: result => {
                    resolve(result);
                },
                onFailure: err => {
                    reject(err);
                },
            });
        });
    }

The function creates an instance of the AuthenticationDetails class, which is used to store the user’s email and password. It also creates an object userData with the user’s email and a reference to the userPool object.

The function then creates a new instance of the CognitoUser class using the userData object.

The function returns a Promise object that resolves with a CognitoUserSession object if the authentication is successful, or rejects with an error if the authentication fails. The newUser object’s authenticateUser method is called with the authenticationDetails object and an object containing two properties, onSuccess and onFailure.

If the authentication is successful, the onSuccess method is called with the CognitoUserSession object, which is then resolved with the Promise object. If the authentication fails, the onFailure method is called with an error object, which is then rejected with the Promise object.

Forgot Password (reset via email)

When user forgets his password we need to provide a mechanism to reset the password. This will be done in two steps

A verification code will be sent to email.
Set the new password with the verification code sent on the email.

Send forgot password email

The forgotPassword function is used to initiate the forgotten password flow for a user in a Cognito User Pool. The function takes an email as input and creates a new CognitoUser object with the Username set to the email and a reference to the userPool object.

    forgotPassword(email) {
        return new Promise((resolve, reject) => {
            return new CognitoUser({Username: email, Pool: this.userPool}).forgotPassword(
                {
                    onSuccess: function (result) {
                        resolve(result);
                    },
                    onFailure: function (err) {
                        reject(err)
                    }
                });
        });
    }

Set new password

The setNewPassword function is used to set a new password for a user who has forgotten their password and gone through the password reset process. It takes three parameters: the user’s email, the verification code sent to the user to confirm their identity, and the new password they want to set.

    confirmPassword(email, verificationCode, newPassword) {
        return new Promise((resolve, reject) => {
            return new CognitoUser({Username: email, Pool: this.userPool}).confirmPassword(verificationCode, newPassword, {
                onSuccess: function (result) {
                    resolve(result);
                },
                onFailure: function (err) {
                    reject(err)
                }
            });
        });
    }

Inside the function, a new Promise object is created to handle the asynchronous nature of the password reset process. The function creates a new instance of the CognitoUser class using the user’s email and the userPool object.

The confirmPassword method of the CognitoUser object is then called with the verificationCode and newPassword parameters, along with an object containing two properties, onSuccess and onFailure.

If the password reset is successful, the onSuccess method is called with the result object, which is then resolved with the Promise object. If there is an error during the password reset process, the onFailure method is called with the error object, which is then rejected with the Promise object.

Delete user

The deleteUser function is used to delete a user from the Amazon Cognito User Pool. It takes a single parameter sub which is the user’s sub (subject) attribute.

    async deleteUser(sub: string) {
        const adminDeleteUserCommand = new AdminDeleteUserCommand({
            Username: sub,
            UserPoolId: this.authConfig.userPoolId
        });

        await this.providerClient.send(adminDeleteUserCommand)
    }

Inside the function, a new instance of the AdminDeleteUserCommand class is created with an object that contains the Username of the user to be deleted and the UserPoolId of the user pool where the user is registered.

The await keyword is used to wait for the send method of the providerClient object to send the adminDeleteUserCommand to the Cognito service to delete the user.

If successful, the function completes without returning anything. If there is an error, it will throw an exception that can be caught and handled by calling code.

Conclusion

In this article, we have reviewed various code snippets related to user authentication and management in AWS Cognito. We have discussed the purpose and functionality of functions such as authenticateUser, forgotPassword, confirmPassword, setNewPassword, and deleteUser. Overall, these functions provide a solid foundation for managing user authentication and access control in AWS Cognito.

Fix error for lambda function url in localstack

2022-11-05T00:00:00+05:30

Spent more than half a day, struggled with this error when I was testing an async lambda on my local.

Error

I created a lambda which had an async implementation, but when I deployed it on lambda, and used function url to access it.

It started giving an error in response

{"__type": "InternalError", "message": "exception while calling lambda with unknown operation: 'str' object has no attribute 'get'"}

Steps to recreate

Let us create a lambda with simple wait function.

export const sleep = (ms) => {
    return new Promise(resolve => setTimeout(resolve, ms));
}

exports.handler = async (event, context) => {
    await sleep(10)

    return {
        statusCode: 200,
        headers: {'Content-Type': 'application/json'},
        body: "Hello World",
    };
};

Deploying to localstack

So we have our code ready to deploy, I will use terraform to deploy the code to localstack. But first make sure, you have your localstack setup, follow the instructions if you have not.

Terraform

Now we will create provider.tf to point our aws provider to localstack.

terraform {
  required_version = "~> 1.3.3"
}

provider "aws" {
  access_key                  = "test"
  secret_key                  = "test"
  region                      = "us-east-1"
  s3_force_path_style         = false
  skip_credentials_validation = true
  skip_metadata_api_check     = true
  skip_requesting_account_id  = true

  endpoints {
    apigateway     = "http://localhost:4566"
    apigatewayv2   = "http://localhost:4566"
    cloudformation = "http://localhost:4566"
    cloudwatch     = "http://localhost:4566"
    dynamodb       = "http://localhost:4566"
    ec2            = "http://localhost:4566"
    es             = "http://localhost:4566"
    elasticache    = "http://localhost:4566"
    firehose       = "http://localhost:4566"
    iam            = "http://localhost:4566"
    kinesis        = "http://localhost:4566"
    lambda         = "http://localhost:4566"
    rds            = "http://localhost:4566"
    redshift       = "http://localhost:4566"
    route53        = "http://localhost:4566"
    s3             = "http://s3.localhost.localstack.cloud:4566"
    secretsmanager = "http://localhost:4566"
    ses            = "http://localhost:4566"
    sns            = "http://localhost:4566"
    sqs            = "http://localhost:4566"
    ssm            = "http://localhost:4566"
    stepfunctions  = "http://localhost:4566"
    sts            = "http://localhost:4566"
  }
}

As we have the provider setup, lets deploy the lambda.

resource "aws_iam_role" "iam_for_lambda_fix" {
  name = "iam_for_lambda_fix"

  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Effect": "Allow",
      "Sid": ""
    }
  ]
}
EOF
}

data "archive_file" "lambda_zip" {
  type        = "zip"
  source_file = "./index.js"
  output_path = "lambda.zip"

}

resource "aws_lambda_function" "lambda" {
  filename         = "lambda.zip"
  function_name    = "lambda_fix"
  role             = aws_iam_role.iam_for_lambda_fix.arn
  handler          = "./index.handler"
  source_code_hash = "${filebase64sha256("./index.js")}"
  runtime          = "nodejs16.x"
  timeout = 30

  depends_on = [data.archive_file.lambda_zip]
}

resource "aws_lambda_function_url" "function_url" {

  function_name = aws_lambda_function.lambda.function_name
  authorization_type = "NONE"

}

output "url" {
  value = aws_lambda_function_url.function_url
}

Here, we have created an iam policy, then archived the js file, and deploy a function.

All set, now to run this application.

$ terraform init
$ terraform plan
$ terraform apply --auto-approve

Once the commands are executed.

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

Outputs:

url = {
  "authorization_type" = "NONE"
  "cors" = tolist([])
  "function_arn" = "arn:aws:lambda:us-east-1:000000000000:function:lambda_fix"
  "function_name" = "lambda_fix"
  "function_url" = "http://cdd7e6500eb91965c99dc0eb42b1613a.lambda-url.us-east-1.localhost.localstack.cloud:4566/"
  "id" = "lambda_fix"
  "qualifier" = ""
  "timeouts" = null /* object */
  "url_id" = "cdd7e6500eb91965c99dc0eb42b1613a"
}

now if we do curl, we will get the error

$ curl http://cdd7e6500eb91965c99dc0eb42b1613a.lambda-url.us-east-1.localhost.localstack.cloud:4566/

{"__type": "InternalError", "message": "exception while calling lambda with unknown operation: the JSON object must be str, bytes or bytearray, not Response"}% 

But if you deploy the same lambda to aws , it works fine. To my surprise if I remove the async operation, the application works fine on localstack as well.

Fix

The fix is rather simple, for unknown reason, localstack is unable to convert the async/await to promise result.

So to fix the above issue lets do a simple modification.

export const sleep = (ms) => {
    return new Promise(resolve => setTimeout(resolve, ms));
}

exports.handler = async (event, context) => {
    await sleep(10)

    return Promise.resolve({
        statusCode: 200,
        headers: {'Content-Type': 'application/json'},
        body: "Hello World",
    });
};

We just add the Promise.resolve() explicitly. And that solve the issue with localstack function url

Apply our change again using terraform apply & finally do a curl.

$ curl http://cdd7e6500eb91965c99dc0eb42b1613a.lambda-url.us-east-1.localhost.localstack.cloud:4566/
Hello World

You can find the code here.

Testing lambda & sqs integration locally using localstack

2022-10-29T00:00:00+05:30

Running serverless is fun, cost-effective and scalable especially if your application is still growing. But at the same time I found testing lambdas locally to be a pain, especially if you want to test your application integration with SQS or SNS. In this article, I will share my take on how I did it to test my applications.

Problem Statement

I have created many microservices using NestJS as my core framework. Each of my microservice is designed to handle one business feature (e.g. each microservice will have GET/POST/PATCH/PUT/DELETE per business entity).

Logical grouping helps us keep our microservices small and at the same time prevents us from service explosion, thus letting us do a population control of microservices.

As a part of logical grouping, in my microservice, I am also handling SQS/SNS messages within the same lambda.

Solution

Since my aim was to test the events on my local server, here is what I will do.

Create a lambda which routes all the events to my web application. Setup SQS with this as target lambda, simple.

You can skip the initial setup and jump directly to SQS & Lambda integration

Setting up sample NestJS app

I am using a nodejs application but the idea can be leveraged across any other language. I will create a sample lambda function using NestJS, for details refer set up a nestJs application Once the NestJS application is set up, we need to add wrappers to convert it into lambda function, for which we will add serverless dependencies Then we add handlers

So now our lambda is all setup to be deployed on aws. In this article, I am not going in details on how to deploy it on aws, but you can refer this article to do via serverless framework

SQS message handler

If the lambda event contains we will be processing the received messages. For simplicity, I am just logging the message event, in real world application, you would have a factory which will handle each type of messages.

The modified handler will look something like this.

import { NestFactory } from "@nestjs/core";
import { AppModule } from "./app.module";
import serverlessExpress from "@vendia/serverless-express";
import { Callback, Context, Handler } from "aws-lambda";

let server: Handler;

async function bootstrap() {
  const app = await NestFactory.create(AppModule);
  await app.init();

  const expressApp = app.getHttpAdapter().getInstance();
  return serverlessExpress({ app: expressApp });
}

const handleMessage = (messages: any[]) => {
  messages.forEach(message => console.log(`Message received: ${message}`)) 
}

export const handler: Handler = async (event: any, context: Context, callback: Callback) => {
  if (event.Records) {
    handleMessage(event.Records);
    return 
  }
  
  server = server ?? (await bootstrap());
  return server(event, context, callback);
};

Here I have a handleMessage function which is just printing all the records. Now our application is ready to receive messages

Setup Localstack

First thing we need to install Localstack on my local, I will be using a docker image for it. So create a docker-compose.yml to initialize localstack

version: "3.7"

services:
  localstack:
    container_name: "${LOCALSTACK_DOCKER_NAME-localstack_main}"
    image: localstack/localstack:1.2.0
    network_mode: bridge
    ports:
      - "127.0.0.1:53:53"                # only required for Pro (DNS)
      - "127.0.0.1:53:53/udp"            # only required for Pro (DNS)
      - "127.0.0.1:443:443"              # only required for Pro (LocalStack HTTPS Edge Proxy)
      - "127.0.0.1:4510-4559:4510-4559"  # external service port range
      - "127.0.0.1:4566:4566"            # LocalStack Edge Proxy
    environment:
      - SERVICES=${SERVICES-}
      - DEBUG=${DEBUG-}
      - DATA_DIR=${DATA_DIR-}
      - LAMBDA_EXECUTOR=${LAMBDA_EXECUTOR-}
      - LOCALSTACK_API_KEY=${LOCALSTACK_API_KEY-}  # only required for Pro
      - HOST_TMP_FOLDER=${TMPDIR:-/tmp/}localstack
      - DOCKER_HOST=unix:///var/run/docker.sock
    volumes:
      - "${TMPDIR:-/tmp}/localstack:/tmp/localstack"
      - "/var/run/docker.sock:/var/run/docker.sock"
      - ./localstack_bootstrap:/docker-entrypoint-initaws.d/

now when do docker-compose up, this will download the localstack (v1.2.0) and start it. To verify weather the service are up and running, just to go http://localhost:4566/health in your browser, it would be something like this.

Install Terraform

I will be using Terraform to set up my infrastructure. To install follow the instructions given here

SQS & Lambda Integration

This is main section where all the magic happens.

Setup Terraform Provider

We need to setup localstack as out aws provider, to do this, we will override the endpoints and provide random secrets.

provider "aws" {
  access_key                  = "test"
  secret_key                  = "test"
  region                      = "us-east-1"
  s3_force_path_style         = false
  skip_credentials_validation = true
  skip_metadata_api_check     = true
  skip_requesting_account_id  = true

  endpoints {
    apigateway     = "http://localhost:4566"
    apigatewayv2   = "http://localhost:4566"
    cloudformation = "http://localhost:4566"
    cloudwatch     = "http://localhost:4566"
    dynamodb       = "http://localhost:4566"
    ec2            = "http://localhost:4566"
    es             = "http://localhost:4566"
    elasticache    = "http://localhost:4566"
    firehose       = "http://localhost:4566"
    iam            = "http://localhost:4566"
    kinesis        = "http://localhost:4566"
    lambda         = "http://localhost:4566"
    rds            = "http://localhost:4566"
    redshift       = "http://localhost:4566"
    route53        = "http://localhost:4566"
    s3             = "http://s3.localhost.localstack.cloud:4566"
    secretsmanager = "http://localhost:4566"
    ses            = "http://localhost:4566"
    sns            = "http://localhost:4566"
    sqs            = "http://localhost:4566"
    ssm            = "http://localhost:4566"
    stepfunctions  = "http://localhost:4566"
    sts            = "http://localhost:4566"
  }
}

Helper Lambda

We will create a lambda which will forward the event to my application. This is very simple lambda, create in file lambda/src/main.ts

import axios from 'axios';

exports.handler = async (event, context) => {
  const body = {event: event, context: context}
  try {
    const axiosResponse = await axios.post(process.env.URL, body);
    return {
      statusCode: 200,
      headers: {'Content-Type': 'application/json'},
      body: axiosResponse.data?.body || axiosResponse.data
    }
  } catch (e) {
    return {
      statusCode: 500,
      headers: {'Content-Type': 'application/json'},
      body: JSON.stringify(e.message)
    }
  }
}

We are using axios to make post request to the given url.

I created a script to compile the application scripts/build.sh

#!/bin/bash

rm -rf lambda_function.zip
pushd lambda || exit 1
npm install
npm run build
popd || exit 1

Creating lambda in localstack

I am creating lambda.tf, in this file,

we will create aws_iam_role for this lambda.
We will use null_resource to execute the above script created, it will be triggered if main.ts gets modified
Then we create a zip file for our lambda using archive_file
Finally, we create lambda by the name of helper lambda aws_lambda_function. We are also passing the environment variable here.

resource "aws_iam_role" "iam_for_lambda" {
  name = "iam_for_lambda"

  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Effect": "Allow",
      "Sid": ""
    }
  ]
}
EOF
}

resource "null_resource" "build" {
  triggers = {
    versions = filesha256("./lambda/src/main.ts")
  }

  provisioner "local-exec" {
    command = "/bin/bash ./scripts/build.sh"
  }
}

data "archive_file" "lambda_zip" {
  type        = "zip"
  source_dir = "./lambda"
  output_path = "lambda_function.zip"
  excludes = ["./lambda/src"]

  depends_on = [null_resource.build]
}

resource "aws_lambda_function" "lambda" {
  filename         = "lambda_function.zip"
  function_name    = "helper-lambda"
  role             = aws_iam_role.iam_for_lambda.arn
  handler          = "dist/main.handler"
  source_code_hash = filebase64sha256("lambda_function.zip")
  runtime          = "nodejs16.x"
  environment {
    variables = {
      URL = var.url
    }
  }

  depends_on = [data.archive_file.lambda_zip]
}

SQS

Next we will create the queue and the helper-lambda as its trigger, so that when ever the message is received by SQS it would trigger the lambda with that message.

aws_sqs_queue creates a queue with name ‘test-queue’, we need to keep visibility_timeout_seconds high since we need to make sure that the message doesn’t get replayed.
aws_lambda_event_source_mapping sets up the lambda function as a target to our sqs queue.

resource "aws_sqs_queue" "test-queue" {
  name = "test-queue"
  visibility_timeout_seconds = 600
}


resource "aws_lambda_event_source_mapping" "test_queue_lambda_mapping" {
  batch_size        = 10
  event_source_arn  = aws_sqs_queue.test-queue.arn
  enabled           = true
  function_name     = aws_lambda_function.lambda.function_name
}

variable.tf

Finally, we create a variable.tf to store variables, in this case I want to create variable named url which I am using in my lambda environment variable.

variable "url" {
  type = string
  description = "Url for forwarding the event"
  default = "http://host.docker.internal:3001/lambda-handler"
}

Notice I am using host.docker.internal instead of ~~localhost~~, it’s because the lambda would be running inside my docker, and to access my host ip from within docker container I use ‘host.docker.internal’.

So, our terraform script are ready.

now we will do:

terraform init, it will download module dependencies for aws.

terraform plan, this will generate details of the resources to be created. In this case it will be like this.

$ terraform plan
  
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create
 <= read (data resources)
  
Terraform will perform the following actions:
  
  # data.archive_file.lambda_zip will be read during apply
  # (depends on a resource or a module with changes pending)
 <= data "archive_file" "lambda_zip" {
      + excludes            = [
          + "./lambda/src",
        ]
      + id                  = (known after apply)
      + output_base64sha256 = (known after apply)
      + output_md5          = (known after apply)
      + output_path         = "lambda_function.zip"
      + output_sha          = (known after apply)
      + output_size         = (known after apply)
      + source_dir          = "./lambda"
      + type                = "zip"
    }
  
  # aws_iam_role.iam_for_lambda_tf will be created
  + resource "aws_iam_role" "iam_for_lambda" {
      + arn                   = (known after apply)
      + assume_role_policy    = jsonencode(
            {
              + Statement = [
                  + {
                      + Action    = "sts:AssumeRole"
                      + Effect    = "Allow"
                      + Principal = {
                          + Service = "lambda.amazonaws.com"
                        }
                      + Sid       = ""
                    },
                ]
              + Version   = "2012-10-17"
            }
        )
      + create_date           = (known after apply)
      + force_detach_policies = false
      + id                    = (known after apply)
      + managed_policy_arns   = (known after apply)
      + max_session_duration  = 3600
      + name                  = "iam_for_lambda"
      + name_prefix           = (known after apply)
      + path                  = "/"
      + tags_all              = (known after apply)
      + unique_id             = (known after apply)
  
      + inline_policy {
          + name   = (known after apply)
          + policy = (known after apply)
        }
    }
  
  # aws_lambda_event_source_mapping.test_queue_lambda_mapping will be created
  + resource "aws_lambda_event_source_mapping" "test_queue_lambda_mapping" {
      + batch_size                    = 10
      + enabled                       = true
      + event_source_arn              = (known after apply)
      + function_arn                  = (known after apply)
      + function_name                 = "helper-lambda"
      + id                            = (known after apply)
      + last_modified                 = (known after apply)
      + last_processing_result        = (known after apply)
      + maximum_record_age_in_seconds = (known after apply)
      + maximum_retry_attempts        = (known after apply)
      + parallelization_factor        = (known after apply)
      + state                         = (known after apply)
      + state_transition_reason       = (known after apply)
      + uuid                          = (known after apply)
  
      + amazon_managed_kafka_event_source_config {
          + consumer_group_id = (known after apply)
        }
  
      + self_managed_kafka_event_source_config {
          + consumer_group_id = (known after apply)
        }
    }
  
  # aws_lambda_function.lambda will be created
  + resource "aws_lambda_function" "lambda" {
      + architectures                  = (known after apply)
      + arn                            = (known after apply)
      + filename                       = "lambda_function.zip"
      + function_name                  = "helper-lambda"
      + handler                        = "dist/main.handler"
      + id                             = (known after apply)
      + invoke_arn                     = (known after apply)
      + last_modified                  = (known after apply)
      + memory_size                    = 128
      + package_type                   = "Zip"
      + publish                        = false
      + qualified_arn                  = (known after apply)
      + qualified_invoke_arn           = (known after apply)
      + reserved_concurrent_executions = -1
      + role                           = (known after apply)
      + runtime                        = "nodejs16.x"
      + signing_job_arn                = (known after apply)
      + signing_profile_version_arn    = (known after apply)
      + source_code_hash               = "1"
      + source_code_size               = (known after apply)
      + tags_all                       = (known after apply)
      + timeout                        = 3
      + version                        = (known after apply)
  
      + environment {
          + variables = {
              + "URL" = "http://localhost:3001/lambda-handler"
            }
        }
  
      + ephemeral_storage {
          + size = (known after apply)
        }
  
      + tracing_config {
          + mode = (known after apply)
        }
    }
  
  # aws_sqs_queue.test-queue will be created
  + resource "aws_sqs_queue" "test-queue" {
      + arn                               = (known after apply)
      + content_based_deduplication       = false
      + deduplication_scope               = (known after apply)
      + delay_seconds                     = 0
      + fifo_queue                        = false
      + fifo_throughput_limit             = (known after apply)
      + id                                = (known after apply)
      + kms_data_key_reuse_period_seconds = (known after apply)
      + max_message_size                  = 262144
      + message_retention_seconds         = 345600
      + name                              = "test-queue"
      + name_prefix                       = (known after apply)
      + policy                            = (known after apply)
      + receive_wait_time_seconds         = 0
      + redrive_allow_policy              = (known after apply)
      + redrive_policy                    = (known after apply)
      + sqs_managed_sse_enabled           = (known after apply)
      + tags_all                          = (known after apply)
      + url                               = (known after apply)
      + visibility_timeout_seconds        = 300
    }
  
  # null_resource.build will be created
  + resource "null_resource" "build" {
      + id       = (known after apply)
      + triggers = {
          + "versions" = "ffef874951cd9c637c7d1699b1604f6b6c5f957a14ca3bc9686bb44731fff0e9"
        }
    }
  
Plan: 5 to add, 0 to change, 0 to destroy.
  
Changes to Outputs:
  + lambda     = (known after apply)
  + test-queue = (known after apply)
╷
│ Warning: Argument is deprecated
│ 
│   with provider["registry.terraform.io/hashicorp/aws"],
│   on provider.tf line 5, in provider "aws":
│    5:   s3_force_path_style         = false
│ 
│ Use s3_use_path_style instead.
│ 
│ (and one more similar warning elsewhere)
╵
╷
│ Warning: Attribute Deprecated
│ 
│   with provider["registry.terraform.io/hashicorp/aws"],
│   on provider.tf line 5, in provider "aws":
│    5:   s3_force_path_style         = false
│ 
│ Use s3_use_path_style instead.
│ 
│ (and one more similar warning elsewhere)
╵
  
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
  
Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these actions if you run "terraform apply" now.

terraform apply --auto-approve to finally create the resources.

null_resource.build: Creating...
null_resource.build: Provisioning with 'local-exec'...
null_resource.build (local-exec): Executing: ["/bin/sh" "-c" "/bin/bash ./scripts/build.sh"]
null_resource.build (local-exec): ~/Documents/github/blog-code/terraform/sqs-lambda-integration-localstack/lambda ~/Documents/github/blog-code/terraform/sqs-lambda-integration-localstack
aws_iam_role.iam_for_lambda: Creating...
aws_sqs_queue.test-queue: Creating...
aws_iam_role.iam_for_lambda: Creation complete after 0s [id=iam_for_lambda]
  
null_resource.build (local-exec): up to date, audited 322 packages in 2s
  
null_resource.build (local-exec): 41 packages are looking for funding
null_resource.build (local-exec):   run `npm fund` for details
  
null_resource.build (local-exec): found 0 vulnerabilities
  
null_resource.build (local-exec): > lamda@0.0.1 prebuild
null_resource.build (local-exec): > rimraf dist
  
  
null_resource.build (local-exec): > lamda@0.0.1 build
null_resource.build (local-exec): > npm run tsc
  
  
null_resource.build (local-exec): > lamda@0.0.1 tsc
null_resource.build (local-exec): > tsc -p tsconfig.json && tsc -p tsconfig.build.json
  
null_resource.build (local-exec): ~/Documents/github/blog-code/terraform/sqs-lambda-integration-localstack
null_resource.build: Creation complete after 8s [id=5577006791947779410]
data.archive_file.lambda_zip: Reading...
aws_sqs_queue.test-queue: Still creating... [10s elapsed]
data.archive_file.lambda_zip: Read complete after 8s [id=245dcc34cbcf937d81b543bffd855dcda246f144]
aws_lambda_function.lambda: Creating...
aws_sqs_queue.test-queue: Still creating... [20s elapsed]
aws_lambda_function.lambda: Creation complete after 8s [id=helper-lambda]
aws_sqs_queue.test-queue: Creation complete after 25s [id=http://localhost:4566/000000000000/test-queue]
aws_lambda_event_source_mapping.test_queue_lambda_mapping: Creating...
aws_lambda_event_source_mapping.test_queue_lambda_mapping: Creation complete after 0s [id=d6634787-8cc0-4366-aedc-cfd18ab75c9c]
╷
│ Warning: Argument is deprecated
│ 
│   with provider["registry.terraform.io/hashicorp/aws"],
│   on provider.tf line 5, in provider "aws":
│    5:   s3_force_path_style         = false
│ 
│ Use s3_use_path_style instead.
│ 
│ (and 2 more similar warnings elsewhere)
╵
╷
│ Warning: Attribute Deprecated
│ 
│   with provider["registry.terraform.io/hashicorp/aws"],
│   on provider.tf line 5, in provider "aws":
│    5:   s3_force_path_style         = false
│ 
│ Use s3_use_path_style instead.
│ 
│ (and 2 more similar warnings elsewhere)
╵
  
Apply complete! Resources: 5 added, 0 changed, 0 destroyed.
  
Outputs:
  
lambda = "arn:aws:apigateway:us-east-1:lambda:path/2015-03-31/functions/arn:aws:lambda:us-east-1:000000000000:function:helper-lambda/invocations"
test-queue = "http://localhost:4566/000000000000/test-queue"

And you have your lambda & sqs integration created.

Lambda handler in main app

Before, we start our test, we need to create a post controller in our main application, which will be triggered by our deployed lambda.

import {Controller, Post} from '@nestjs/common';
import {handler} from './lambda';

@Controller()
export class AppController {

  @Post('/lambda-handler')
  async lambdaHandler(@Body() reqeust: any) {
    return await handler(reqeust.event, reqeust.context);
  }
}

Conclusion

Time to test our integration, I am using aws cli to send the message to SQS.

$ aws sqs send-message --endpoint http://localhost:4566  --queue-url http://localhost:4566/000000000000/test-queue --message-body "hello"

# generated output
{
    "MD5OfMessageBody": "5d41402abc4b2a76b9719d911017c592",
    "MessageId": "2d0fc032-64ca-49a6-9f98-868854013942"
}

After a moment later, we can see the output in our main application’s log:

Message received: {"body":"hello","receiptHandle":"YjhjNDBlZGMtYjAxNi00YTNhLWFmZjEtZWU4MjI5YmE4NDZiIGFybjphd3M6c3FzOnVzLWVhc3QtMTowMDAwMDAwMDAwMDA6dGVzdC1xdWV1ZSAyZDBmYzAzMi02NGNhLTQ5YTYtOWY5OC04Njg4NTQwMTM5NDIgMTY2NzExOTA4My43NjI1NA==","md5OfBody":"5d41402abc4b2a76b9719d911017c592","eventSourceARN":"arn:aws:sqs:us-east-1:000000000000:test-queue","eventSource":"aws:sqs","awsRegion":"us-east-1","messageId":"2d0fc032-64ca-49a6-9f98-868854013942","attributes":{"SenderId":"000000000000","SentTimestamp":"1667119082819","ApproximateReceiveCount":"1","ApproximateFirstReceiveTimestamp":"1667119083762"},"messageAttributes":{}}

For me this architecture has reduced my efforts in testing SQS events integration with lambda(I am running more than 20 copies of this helper lambda for all my microservices endpoint)

I would surely love to listen to you as well, how are you testing SQS & Lambda in your local environment. You can find terraform & lambda code on github

Create Docker Container to Run Jekyll

2022-06-26T00:00:00+05:30

I always faced issues when running jekyll locally, my blog uses jekyll to generate static pages, but every time my os is upgraded or I change my laptop, I always get into trouble of installing/managing ruby version, bundles etc. To my rescue comes the docker, I have created a simple a docker file and script to serve my jekyll pages locally, and it now works like a charm.

I always have been admirer of docker, I tend to use docker more often if have to install any dev app, databases etc or package but still want to keep my machine clean. In this case as well I wanted a consistent mechanism to run my blogs locally.

Dockerfile

The first thing I do is create a simple dockerfile in my Jekyll app. I copy my Gemfile and Gemfile.lock to install the bundles.

FROM jekyll/jekyll:3.8

COPY Gemfile* ./

RUN gem install bundler:2.2.24 && bundle install

ENTRYPOINT [ "jekyll" ]

CMD [ "build" ]

The reason I copied Gemfile is because sometimes the bundle version is different, so now I install the required bundler (in my case its 2.2.24), and install all the dependencies.

Execution Script

Next I created a simple script local_serve.sh to build and serve my code, so that i don’t keep on typing the commands every time.

#!/usr/bin/env bash

set -e

case $1 in
    'prod-build')
        echo "Building docker image of Jekyll"
        docker build -t jekyll .

        echo "Building Jekyll site"    
        docker run \
            --volume="$PWD:/srv/jekyll:Z" \
            -e JEKYLL_ENV=prod \
            -t jekyll \
            build
    ;;
    'build')
        echo "Building docker image of Jekyll"
        docker build -t jekyll .

        echo "Building Jekyll site"    
        docker run \
            --volume="$PWD:/srv/jekyll:Z" \
            -t jekyll
    ;;
    'serve')
    docker run \
            --volume="$PWD:/srv/jekyll:Z" \
            -p 4000:4000 \
            -it jekyll serve
    ;;
esac

In the build command, I am creating the jekyll image. I can mapping the current directory to /srv/jekyll and running my jekyll container to build the site.

In the serve command, I using the same directory to serve my pages at port 4000 and using interactive mode to explicitly invoke serve command.

Run the Jekyll container

Before we proceed do not forget to give executable permission to local_service.sh

$ chmod a+x local_serve.sh

So, first thing to do is to build our Jekyll app so that all the dependencies are downloaded. We will just run

$ ./local_serve.sh build

Once the dependencies are downloaded, now lets start our Jekyll server

$ ./local_serve.sh serve

And we can see the server started

ruby 2.6.3p62 (2019-04-16 revision 67580) [x86_64-linux-musl]
Configuration file: /srv/jekyll/_config.yml
            Source: /srv/jekyll
       Destination: /srv/jekyll/_site
 Incremental build: disabled. Enable with --incremental
      Generating... 
       Jekyll Feed: Generating feed for posts
   GitHub Metadata: No GitHub API authentication could be found. Some fields may be missing or have incorrect data.
                    done in 17.778 seconds.
 Auto-regeneration: enabled for '/srv/jekyll'
    Server address: http://0.0.0.0:4000/
  Server running... press ctrl-c to stop.

Enjoy running Jekyll locally without installing ruby or jekyll directly on your system.

Deploy NestJS application on AWS Lambda

2021-12-28T00:00:00+05:30

Lets go serverless, in todays world deploying an application might just cost you cents, you can run a full fledged application for less than $1 a year if you have less traffic on your website.

When you create a simplest lambda can be created by creating a function which takes event, context, & callback params like this which returns a Hello World!.

exports.handler = (event, context, callback) => {
  callback(null, "Hello World!");
};

But it real world, each service might be having multiple endpoints. So we are going to convert our simple nest application to be aws compatible, so in this article, we will focus on deploying a nestJS app on aws lambda.

Application

To start we will create a nest application.

$ npm i -g @nestjs/cli
$ nest new nestjs-aws-lambda

Now we can run and test if our application is working.

$ cd nestjs-aws-lambda
$ npm start

Our default application is running, so lets get back to our primary goal, i.e. to convert this app to serverless.

Serverless Dependencies

There are multiple package already available to convert a node applications to aws compatible. Let install the required packages

$ npm i -S @vendia/serverless-express aws-lambda serverless
$ npm i -D @types/aws-lambda serverless-offline

Creating Handler

We will create a new file serverless.ts in our source directory.

import { NestFactory } from "@nestjs/core";
import { AppModule } from "./app.module";
import serverlessExpress from "@vendia/serverless-express";
import { Callback, Context, Handler } from "aws-lambda";

let server: Handler;

async function bootstrap() {
  const app = await NestFactory.create(AppModule);
  await app.init();

  const expressApp = app.getHttpAdapter().getInstance();
  return serverlessExpress({ app: expressApp });
}

export const handler: Handler = async (
  event: any,
  context: Context,
  callback: Callback
) => {
  server = server ?? (await bootstrap());
  return server(event, context, callback);
};

The yaml file

Now that we have created a handler function, we need to tell the lambda to find the handler method in serverless.ts file. To do this we will create a serverless.yaml

service: nest-auth

plugins:
  - serverless-offline

provider:
  name: aws
  runtime: nodejs12.x
  region: eu-west-1
  profile: default
  memorySize: 256 # optional, in MB, default is 1024
  # stage: dev

functions:
  main:
    handler: dist/serverless.handler
    events:
      - http:
          method: ANY
          path: /
      - http:
          method: ANY
          path: "{proxy+}"

custom:
  serverless-offline:
    noPrependStageInUrl: true

Running the lambda locally

There are two ways of running the application in lambda locally

You can install serverless globally by npm install -g serverless and then run sls offline.

Other way is use locally installed serverless module. To do that:

execute command node ./node_modules/serverless/bin/serverless.js offline
I like to include the above command in package.json scripts.

...
...
"scripts": {
 "prebuild": "rimraf dist",
 "build": "nest build",
 "format": "prettier --write \"src/**/*.ts\" \"test/**/*.ts\"",
 "start": "nest start",
 "start:dev": "nest start --watch",
 "start:debug": "nest start --debug --watch",
 "start:prod": "node dist/main",
 "lint": "eslint \"{src,apps,libs,test}/**/*.ts\" --fix",
 "test": "jest",
 "test:watch": "jest --watch",
 "test:cov": "jest --coverage",
 "test:debug": "node --inspect-brk -r tsconfig-paths/register -r ts-node/register node_modules/.bin/jest --runInBand",
 "test:e2e": "jest --config ./test/jest-e2e.json",
 "deploy:local": "node ./node_modules/serverless/bin/serverless.js offline"
}

...
...

And now we can simply do npm run deploy:local

You might need to add flag to enable interop, in your tsconfig.ts, if you get an error function not found.

"esModuleInterop": true

Verify

Once the application is ready, you will see your service started locally

Now as we see server is listening to http://localhost:3000

Let’s do a curl to see if we can get a response from our lambda.

$ curl -i http://localhost:3000

HTTP/1.1 200 OK
x-powered-by: Express
content-type: text/html; charset=utf-8
content-length: 12
etag: W/"c-Lve95gjOVATpfV8EL5X4nxwjKHE"
cache-control: no-cache
accept-ranges: bytes
Date: Sat, 12 Feb 2022 08:57:38 GMT
Connection: keep-alive
Keep-Alive: timeout=5

Hello World!%

Looking at the logs we can see as soon as request is sent nest application starting up and after it completes we get a request id a duration and amount of time it will be built for once deployed to production.

Deploying

Now we can run sls deploy –-stage prod and this will package our application and upload to s3 and create all the necessary resources for us.

Once it completes we will get a api url that we can test out and use in production.

After our application is deployed we get some information and we get the endpoints here that we can use to make requests to this lambda in production.

Now if we open postman and replace the url with the production url and we get a response after we created a user which is great.

And here we have nestjs serverless application running on lambda. You can find the code here.

Role based authentication in NestJS using Guards

2021-12-28T00:00:00+05:30

In this article, we will focus on Role Based Access Control (RBAC) using the Guards in a nestJS application.

Guard

A guard is a class annotated with the @Injectable() decorator. Guards should implement the CanActivate interface.

Guards have a single responsibility. They determine whether a given request will be handled by the route handler or not, depending on certain conditions (like permissions, roles, ACLs, etc.) present at run-time. This is often referred to as authorization. Authorization (and its cousin, authentication, with which it usually collaborates) has typically been handled by middleware in traditional Express applications.

Authorization guard

Authorization is a great use case for Guards because specific routes should be available only when the caller (usually a specific authenticated user) has sufficient permissions.

Role Based Authentication

Role-based access control (RBAC) restricts network access based on a person’s role within an organization and has become one of the main methods for advanced access control.

The roles in RBAC refer to the levels of access that employees have to the network. Employees are only allowed to access the information necessary to effectively perform their job duties. Access can be based on several factors, such as authority, responsibility, and job competency.

As a result, lower-level employees usually do not have access to sensitive data if they do not need it to fulfill their responsibilities.

This is especially helpful if you have many employees and use third-parties and contractors that make it difficult to closely monitor network access. Using RBAC helps in securing your company’s sensitive data and important applications.

Creating application

Lets create a new application from scratch using : nest new nest-authorization

Creating initial boilerplate for our app:

To do authorization we need to know who the user is and what roles do they have.

To do this we also need some kind of example api to allow some and protect some depending on user roles.

Lets create a new resource users using : nest g resource users

Defining user entities and roles enum:

To get started our user needs to have roles for us to be able to do role based authorization. We need to have a representation of different roles. We can start with a simple enum file.

export enum Role {
  USER = "user",
  ADMIN = "admin",
}

Testing the initial build

Lets first run the application and see how it behaves originally using npm run start:dev

Now we have sort of a mock api which is open to everyone right now.

Implementation and integration

Now we need to check what role does a user have and are they allowd to access these requests.

We can do this by decorating the methods in users.controllers.ts by using setMetadata() Decorator.

There is also another way that is by using a custom decorator that represents the same thing that setMetadata( ) can do.

In users folder create a roles.decorator.ts

import { SetMetadata } from "@nestjs/common";
import { Role } from "./entities/role.enum";

export const Roles = (...roles: Role[]) => SetMetadata("roles", roles);

Now in users.controller.ts add this decorator .

import {
  Controller,
  Get,
  Post,
  Body,
  Patch,
  Param,
  Delete,
  SetMetadata,
} from "@nestjs/common";
import { UsersService } from "./users.service";
import { CreateUserDto } from "./dto/create-user.dto";
import { UpdateUserDto } from "./dto/update-user.dto";
import { Role } from "./entities/role.enum";
import { Roles } from "./roles.decorator";
@Controller("users")
export class UsersController {
  constructor(private readonly usersService: UsersService) {}

  @Post()
  @Roles(Role.ADMIN)
  create(@Body() createUserDto: CreateUserDto) {
    return this.usersService.create(createUserDto);
  }

  @Get()
  findAll() {
    return this.usersService.findAll();
  }

  @Get(":id")
  findOne(@Param("id") id: string) {
    return this.usersService.findOne(+id);
  }

  @Patch(":id")
  @Roles(Role.ADMIN)
  update(@Param("id") id: string, @Body() updateUserDto: UpdateUserDto) {
    return this.usersService.update(+id, updateUserDto);
  }

  @Delete(":id")
  @Roles(Role.ADMIN)
  remove(@Param("id") id: string) {
    return this.usersService.remove(+id);
  }
}

This doesn’t protect the method yet we are just providing metadata.

Now lets add the piece that adds the actual security i.e. lets create a guard.

Creating a guard

Inside users create a new Roles guard roles.guard.ts

Here we need to check if the user has the required roles to allow the user.

Because we do not have authentication in place which you should to identify the user here we will keep it really simple and mock a user considering we did the authentication.

import { Injectable, CanActivate, ExecutionContext } from "@nestjs/common";
import { Reflector } from "@nestjs/core";
import { Observable } from "rxjs";
import { Role } from "./entities/role.enum";
import { User } from "./entities/user.entity";
@Injectable()
export class RolesGuard implements CanActivate {
  constructor(private reflector: Reflector) {}
  canActivate(context: ExecutionContext): boolean {
    const requireRoles = this.reflector.getAllAndOverride<Role[]>("roles", [
      context.getHandler(),
      context.getClass(),
    ]);

    if (!requireRoles) {
      return true;
    }
    //const {user}=context.switchToHttp().getRequest();
    const user: User = {
      name: "Nishant",
      roles: [Role.USER],
    };
    return requireRoles.some((role) => user.roles.includes(role));
  }
}

Testing

Generic User

If a user is a generic. Then

It will only be able to perform get request. Because here generic user is only allowed get request by the guard.

For a post request by the user it will give a 403 forbidden error.

Admin User

If a user has admin role.

it will be able to perform get request, post, Patch, delete.

References

https://docs.nestjs.com/guards
https://digitalguardian.com/blog/what-role-based-access-control-rbac-examples-benefits-and-more

This was all about using guards in a nestJS application to authorize. You can find the code example here

NestJs Application with MongoDB

2021-12-28T00:00:00+05:30

This article focuses on integrating mongoDB to a Nest Application.

Installation

Nodejs

You will need nodejs to run the application, if you don’t have already installed,you can install node.js from https://nodejs.org/en/download/

NestJS

Now install nest.js using command npm i -g @nestjs/cli

if you are using a MAC device and it throws npm WARN checkPermissions Missing write access to /usr/local/lib/node_modules this error, run this script

$ sudo chown -R $(whoami) $(npm config get prefix)/{lib/node_modules,bin,share}

Adding mongodb to nestjs application

Now you can create a new project using nest new New-proj command

Now install the mongoose dependency using npm install @nestjs/mongoose mongoose command

Integration

Initialize connection in AppModule

Now in app.module.ts import mongoose module and give it the connection string to your database.

Now our mongoose connection to mongodb is done.

Create a schema

Schema the definition of data that we are going to store in database.

We Create different folders for diff domains.
Create a users folder inside src folder which will deal with users related logic.
Inside that create a schemas folder coz we can have more than 1 schemas.
In this eg we will only have one.
Create a file and lets call it user.schema.ts

This is a demo schema. Now we need to tell nestjs about the new schema we have created.

Module

Inside users module create a new file and lets call it users.module.ts

Repository

Now in order to use mongoose and to save or fetch data from database we will create a repository file namely users.repository.ts

Service

In order to test the new repository I have created the users.service to test things out.

Controller

After the service, we will also need a controller to expose an api, for this we will create users.controller.ts

Before you test the api make sure the application is running using yarn start:dev

Testing

To see this in action we can make post request to the backend using postman.

You can now view the database collection. I am using MongoDBCompass.

If you are not familiar with services I have uploaded this code to github

And now we have a fully working nestjs app with mongoDb.

NestJs Application to upload/download file from GCP cloud storage

2021-10-23T00:00:00+05:30

In this article, we will upload a file to cloud storage and download it to serve it from our nestJS api.

Prerequisites:

Create a bucket

To create a bucket:

Open a terminal window. Use the gsutil mb command and a unique name to create a bucket:

$ gsutil mb -b on -l us-east1 gs://my-awesome-bucket/

This uses a bucket named “my-awesome-bucket.” You must choose your own, globally-unique, bucket name. For more information refer docs

Setting up service account

The first thing, we need to set up our service account, so that github has required permission to access the cloud storage.

You need to add Storage Admin permission for publishing artifacts (optionally you can provide Cloud Run Service Agent for cloud run deployments)

Then generate & download key in json format

Creating nestJs application

We will create a nestJS application, create a module to upload to upload and download files. And then we will plugin it to the controller.

We will be using nest cli to scaffold the project.

$ nest new gcp-storage

You will be asked to choose the package manager as per your linking, I prefer pnpm as its faster and takes up less space compared to npm or yarn.

Storage Module

NestJS provides an elegant way to organize your code into modules. Each module can comprise of providers, controller, services etc, you can also import/export modules/services/providers etc.

To create a module, we will navigate to gcp-storage/src.

$ nest g module storage

This command would generate nest module under src/storage.

We now create a config file storage-config.ts

const StorageConfig = {
  projectId: process.env.PROJECT_ID,
  private_key: process.env.PRIVATE_KEY,
  client_email: process.env.CLIENT_EMAIL,
  mediaBucket: process.env.STORAGE_MEDIA_BUCKET,
};

export default StorageConfig;

Also lets create a storage-file.ts to hold the file data.

export class StorageFile {
  buffer: Buffer;
  metadata: Map<string, string>;
  contentType: string;
}

Storage Service

Let us create a service which will be responsible for uploading and downloading file from storage bucket. To do this we will again use nest cli at location src.

$ nest g service storage

This command will create a storage.service.ts at src/storage. I will also annotate the class with @Injectable() and will add reference to ServiceModule. Additionally, we will also export the StorageService from module, as we will be using StorageService in other modules.

So our StorageModule class will look like this.

import { Module } from "@nestjs/common";
import { StorageService } from "./storage.service";

@Module({
  providers: [StorageService],
  exports: [StorageService],
})
export class StorageModule {}

Let’s get back to our storage service. In the constructor we will initialize Storage object with projectId & credentials.

import { StorageFile } from "./storage-file";
import { DownloadResponse, Storage } from "@google-cloud/storage";
import { Injectable } from "@nestjs/common";
import StorageConfig from "./storage-config";

@Injectable()
export class StorageService {
  private storage: Storage;
  private bucket: string;

  constructor() {
    this.storage = new Storage({
      projectId: StorageConfig.projectId,
      credentials: {
        client_email: StorageConfig.client_email,
        private_key: StorageConfig.private_key,
      },
    });

    this.bucket = StorageConfig.mediaBucket;
  }

  async save(
    path: string,
    contentType: string,
    media: Buffer,
    metadata: { [key: string]: string }[]
  ) {
    const object = metadata.reduce((obj, item) => Object.assign(obj, item), {});
    const file = this.storage.bucket(this.bucket).file(path);
    const stream = file.createWriteStream();
    stream.on("finish", async () => {
      return await file.setMetadata({
        metadata: object,
      });
    });
    stream.end(media);
  }

  async delete(path: string) {
    await this.storage
      .bucket(this.bucket)
      .file(path)
      .delete({ ignoreNotFound: true });
  }

  async get(path: string): Promise<StorageFile> {
    const fileResponse: DownloadResponse = await this.storage
      .bucket(this.bucket)
      .file(path)
      .download();
    const [buffer] = fileResponse;
    const storageFile = new StorageFile();
    storageFile.buffer = buffer;
    storageFile.metadata = new Map<string, string>();
    return storageFile;
  }

  async getWithMetaData(path: string): Promise<StorageFile> {
    const [metadata] = await this.storage
      .bucket(this.bucket)
      .file(path)
      .getMetadata();
    const fileResponse: DownloadResponse = await this.storage
      .bucket(this.bucket)
      .file(path)
      .download();
    const [buffer] = fileResponse;

    const storageFile = new StorageFile();
    storageFile.buffer = buffer;
    storageFile.metadata = new Map<string, string>(
      Object.entries(metadata || {})
    );
    storageFile.contentType = storageFile.metadata.get("contentType");
    return storageFile;
  }
}

In the save method, we are creating the writeStream to upload the file to the bucket. On upload we are also updating metadata information.

There are two different implementation of get method, with & without metadata information.

Now that we have our service ready lets add a controller in the main module.

Media Module

I will create a separate media module to which I will inject the storage service we created. Again to generate module, I will goto src folder.

$ nest g module media

CREATE media/media.module.ts (82 bytes)
UPDATE app.module.ts (385 bytes)

To expose the endpoint, I will also need a nestjs controller, so we will again use the nest cli to generate controller.

$ nest g controller media

CREATE media/media.controller.spec.ts (485 bytes)
CREATE media/media.controller.ts (99 bytes)
UPDATE media/media.module.ts (170 bytes)

This command will generate a controller and also update the media module.

Importing Storage module

In order to use StorageService within MediaModule we will have to import the StorageModule. So let’s update our media/media.module.ts

import { Module } from "@nestjs/common";
import { StorageModule } from "src/storage/storage.module";
import { MediaController } from "./media.controller";

@Module({
  imports: [StorageModule],
  controllers: [MediaController],
})
export class MediaModule {}

Also, remember we had already exported the StorageService in StorageModule, so now once we import the module all the exported providers will now be accessible in the current module.

Media Controller

To handle file uploading, Nest provides a built-in module based on the multer middleware package for Express. Multer handles data posted in the multipart/form-data format, which is primarily used for uploading files via an HTTP POST request. This module is fully configurable and you can adjust its behavior to your application requirements.

For better type safety, let’s install Multer typings package

$ pnpm i -D @types/multer

With this package installed, we can now use the Express.Multer.File type (you can import this type as follows: import { Express } from ‘express’).

Now we create the upload method in controller.

import {
  Body,
  Controller,
  Get,
  NotFoundException,
  Param,
  Post,
  Res,
  ServiceUnavailableException,
  UploadedFile,
  UseInterceptors,
} from "@nestjs/common";
import { FileInterceptor } from "@nestjs/platform-express";
import { Response } from "express";
import { StorageFile } from "src/storage/storage-file";
import { StorageService } from "src/storage/storage.service";

@Controller("media")
export class MediaController {
  constructor(private storageService: StorageService) {}

  @Post()
  @UseInterceptors(
    FileInterceptor("file", {
      limits: {
        files: 1,
        fileSize: 1024 * 1024,
      },
    })
  )
  async uploadMedia(
    @UploadedFile() file: Express.Multer.File,
    @Body("mediaId") mediaId: string
  ) {
    await this.storageService.save(
      "media/" + mediaId,
      file.mimetype,
      file.buffer,
      [{ mediaId: mediaId }]
    );
  }

  @Get("/:mediaId")
  async downloadMedia(@Param("mediaId") mediaId: string, @Res() res: Response) {
    let storageFile: StorageFile;
    try {
      storageFile = await this.storageService.get("media/" + mediaId);
    } catch (e) {
      if (e.message.toString().includes("No such object")) {
        throw new NotFoundException("image not found");
      } else {
        throw new ServiceUnavailableException("internal error");
      }
    }
    res.setHeader("Content-Type", storageFile.contentType);
    res.setHeader("Cache-Control", "max-age=60d");
    res.end(storageFile.buffer);
  }
}

In the upload method we are using FileInterceptors to restrict the file size and number of files to be uploaded.

Working with .env files

Only one thing remaining to set the environment variables which is being used by StorageConfig.

We will first add env-cmd & dotenv packages.

$ pnpm install -D env-cmd dotenv

Now we change the package.json file and modify the start:dev script command.

"start:dev": "env-cmd -f .env nest start --watch",

Finally we create .env file in the project folder.

PROJECT_ID = <YOUR_PROJECT_ID>
PRIVATE_KEY = <YOUR_SERVICE_ACCOUNT_PRIVATE_KEY>
CLIENT_EMAIL = <YOUR_SERVICE_ACCOUNT_EMAIL>
STORAGE_MEDIA_BUCKET = <YOUR_BUCKET_NAME>

And there you have it. A working nest application to upload and download files. You can download the source code from github.