The api package

This is often called the "backend". It consists of database-oriented features, plus the API implementation proper, and the independent jobs process.

The database

LiteFarm uses PostgreSQL, a relational database management system, for permanent storage of most data. LiteFarm code uses the Knex query builder to interact with Postgres. When additions or changes to the database schema are needed, we rely on Knex's migrations mechanism. Migrations are JavaScript code that use Knex to modify or extend the database schema. (That is, migrations are mostly concerned with the definition of tables, columns, and other "structural" elements. Less commonly, migrations can deal with data contents, but this is usually limited to "reference" data such as lists of countries, etc.)

Migrations are stored in the packages/api/db/migration folder. A new migration is created with the command knex migrate:make descriptive_name --knexfile=./.knex/knexfile.js, which creates a new template file named YYYYMMDDHHMM_descriptive_name.js , where the prefix indicates the date and time.

Database tables for LiteFarm's notifications features were initially created by the migrations in 20220207164158_notification.js. From the requirements it was clear that we would need two tables. One table (notification) holds the title or subject line, message body, and other characteristics of the message itself. The second table (notification_user) tracks the status of a notification with respect to an individual user-- have they read it, etc. Here is an excerpt from the original migration file.

exports.up = async function (knex) {
  await knex.schema.createTable('notification', function (table) {
    table.uuid('notification_id').primary();
    table.string('title').notNullable();
    table.text('body').notNullable();
    table.enum('ref_table', ['task', 'location', 'users', 'farm', 'document', 'export', ]);
    table.enum('ref_subtable', [/* ... */]);
    table.string('ref_pk');
    table.uuid('farm_id').references('farm_id').inTable('farm');
    table.boolean('deleted').defaultTo(false);
    table.string('created_by_user_id').references('user_id').inTable('users');
    table.string('updated_by_user_id').references('user_id').inTable('users');
    table.dateTime('created_at').notNullable();
    table.dateTime('updated_at').notNullable();
  });

  await knex.schema.createTable('notification_user', function (table) {
    table.primary(['notification_id', 'user_id']);
    table.uuid('notification_id').references('notification_id').inTable('notification');
    table.string('user_id').references('user_id').inTable('users');
    table.boolean('alert').defaultTo(true).notNullable();
    table.enum('status', ['Unread', 'Read', 'Archived']).defaultTo('Unread').notNullable();
    table.boolean('deleted').defaultTo(false);
    table.string('created_by_user_id').references('user_id').inTable('users');
    table.string('updated_by_user_id').references('user_id').inTable('users');
    table.dateTime('created_at').notNullable();
    table.dateTime('updated_at').notNullable();
  });
}

exports.down = async function (knex) {
  await knex.schema.dropTable('notification_user');
  await knex.schema.dropTable('notification');
}

Once a migration has been committed to the repository, it should not be modified. Instead, we write additional migrations as needed to evolve the database schema. For example, the notification table should have been defined to generate the universal identifiers for its records, but this was overlooked. So migration 20220221162055_generate_notification_id.js resolved this.

exports.up = function (knex) {
  return Promise.all([
    knex.raw(
      'ALTER TABLE notification ALTER COLUMN notification_id SET DEFAULT uuid_generate_v4();',
    ),
  ]);
};

exports.down = function (knex) {
  return Promise.all([
    knex.raw('ALTER TABLE notification ALTER COLUMN notification_id DROP DEFAULT;'),
  ]);
};

As work on the notifications features progressed, it became clear that the structure of the notifications table needed to be rethought, and it was modified to its current form by migration 20220325020707_refactor-notification.js.

const newFields = [
  { name: 'translation_key', defn: 'character varying(255) COLLATE pg_catalog."default"' },
  { name: 'variables', defn: 'jsonb' },
  { name: 'entity_id', defn: 'character varying(255) COLLATE pg_catalog."default"' },
  { name: 'entity_type', defn: 'character varying(255) COLLATE pg_catalog."default"' },
  { name: 'context', defn: 'jsonb' },
];

const oldFields = [
  { name: 'title', defn: 'character varying(255) COLLATE pg_catalog."default"' },
  { name: 'body', defn: 'text COLLATE pg_catalog."default"' },
  { name: 'ref_table', defn: 'text COLLATE pg_catalog."default"' },
  { name: 'ref_subtable', defn: 'text COLLATE pg_catalog."default"' },
  { name: 'ref_pk', defn: 'character varying(255) COLLATE pg_catalog."default"' },
];

exports.up = async function (knex) {
  for (const field of oldFields) {
    console.log(`ALTER TABLE notification DROP COLUMN ${field.name};`);
    await knex.raw(`ALTER TABLE notification DROP COLUMN ${field.name};`);
  }
  for (const field of newFields) {
    console.log(`ALTER TABLE notification ADD COLUMN ${field.name} ${field.defn};`);
    await knex.raw(`ALTER TABLE notification ADD COLUMN ${field.name} ${field.defn};`);
  }
};

exports.down = async function (knex) {
  for (const field of newFields) {
    await knex.raw(`ALTER TABLE notification DROP COLUMN ${field.name};`);
  }
  for (const field of oldFields) {
    await knex.raw(`ALTER TABLE notification ADD COLUMN ${field.name} ${field.defn};`);
  }
};

Migration code is run whenever the system is deployed. For example, when Docker containers are created as a new deployment of LiteFarm, all of the migrations will be run, in chronological order. This means the database schema quickly evolves from its initial definition, up through all changes to date.

The migration mechanism uses a special table in the database to track which individual migrations have been applied to the database. When a long-lived deployment (such as production) is updated, only the new migrations are applied. This modifies the database code as needed to be compatible with the new code being deployed.

Migrations can be "undone" in reverse chronological order, because each migration defines a function up for making the changes, and a function down for reverting the changes. In practice, LiteFarm rarely uses this feature.

In a LiteFarm deployment, the database runs as litefarm-db, a Docker container built from the official Postgres image.

The API code

An Application Programming Interface (API) is an interface that a software product provides for use by other software, rather than by human users.

The code for the LiteFarm API is in the packages/api/src folder. In a LiteFarm deployment, this Node.js server runs in a Docker container named litefarm-api. The container is built to put the API code in a standard Node image, and also uses a standard Nginx image to create a reverse proxy server.

The API borrows the Model and Controller concepts from the Model-View-Controller (MVC) architecture pattern.

Models

In the MVC pattern, a model is a dynamic data structure that directly manages the data, logic and rules of the application. Often, a model corresponds to a table in relational databases.

LiteFarm uses Objection.js to define the models. Objection extends the Knex query builder.

Code for the model that corresponds to the notification_user database table is found in packages/api/src/models/NotificationUserModel.js. A portion of the code is shown below.

Each Objection model is a JavaScript class. Most extend a class named BaseModel, which implements what are called the "base properties" (explained below).

The code above shows this model's implementation of several static property accessors defined by the Objection API.

tableName() tells Objection the database table name for this model
idColumn() indicates the table columns needed to uniquely identify a table row
jsonSchema() supports validating the data contents of class instances before database modifications are made. Objection uses the embedded information to determine the appropriate data types, which values are required to be present, etc.
relationMappings() describes how this model is related to others. Each instance of this class belongs to one instance of the Notification model class.
templateMappingSchema() (not shown) this is a custom function created by LiteFarm to define how one would treat properties and relations of the models graph in a duplication context (example: repeat management plans)

In jsonSchema(), the spread syntax expression ...this.baseProperties is used to include the base model's "base properties" in validating instances of class NotificationUser. These properties are "bookkeeping" columns found in most LiteFarm database tables. They record who created the row, and when; who last updated the row, and when; and whether the row is marked as "deleted". (In most cases, LiteFarm performs "soft deletes"; rather than actually deleting a database row, the row is marked as deleted, and excluded from further use.)

Controllers

In the MVC pattern, a controller acts as a kind of translator between the model(s) on one hand, and a client on the other. In general terms, a controller accepts input from the UI, converts it to commands to the model(s), transforms the results and responds to the UI.

In terms specific to LiteFarm, a controller is a file of functions that handle all requests related to a particular type of entity. For example, packages/api/src/controllers/notificationUserController.js contains all the functions that handle requests involving the current user's notifications.

These functions are built on the Express.js framework, which we will explore in greater detail below. For now, it is enough to understand that a controller function always receives (at least) two parameters. These are references to objects that represent the HTTP request received from the client, and the HTTP response that we are currently putting together. Here is an example extracted from the file named in the previous paragraph.

When the client requests a list of notifications to display to the user, the Express framework calls this function. The call passes in req and res which represent the HTTP request and response, respectively.

The function trys a call to the NotificationUser model's method, getNotificationsForFarmUser(). (Some context: a registered LiteFarm user is always logged into the context of one particular farm. The request we are handling here means approximately "show me my notifications related to the farm I'm now logged into".) To fetch the necessary data, the model needs to know two things: who is the user, and what is their current farm? These values are part of the request (req.user.user_id, and req.headers.farm_id respectively) and the controller passes the values in its call to the model's method.

If the model successfully returns results, the controller responds to the client, with an HTTP success code (200) and the notifications data from the model. (The Express framework will send that data as JSON.) If there is a problem, the catch block has the controller log the error, and responds to the client with a HTTP "Bad Request" code (400) and JSON information about the specific error.

Here is another example from the same controller.

When the client requests to update the user's notifications, Express calls this function. The controller unpacks the request to transform its details into the form needed by the model. Specifically, it creates a new object as a copy of the request's body and deletes the notification_id property from that copy. It then calls the model's update() method, passing the user_id, the array of notification IDs, and the modified request body.

Take a moment to consider the separation of concerns between the controller and the model. The controller sticks to its job: it translates a specific form of incoming HTTP request into commands that the model knows how to handle. The controller does not and should not know anything about database table names, columns, or even that we are using a database; it just knows how to talk to the model through a set of method calls that keep concerns separated.

Meanwhile, the model is given the information it needs to do its job: managing the storage and retrieval of data while applying appropriate logic and rules. It is not passed the entire request object; that is an HTTP concept and HTTP is outside the model's concerns. Accordingly, the model does not return an HTTP status code-- it doesn't know them. It either returns the data that the controller asked for, or it throws an error to indicate why it cannot.

Routes and middleware

The previous section stated that the Express framework calls notificationUserController.getNotifications() method when the client requests a list of the users notification, but did not explain how the client makes that request, or how Express knows what code to call.

The explanation lies in Express routing. When the API server starts, the setup routines include the statements const notificationUserRoute = require('./routes/notificationUserRoute'); and, much later, app.use('/notification_user', notificationUserRoute). This configures the application to route all requests to URLs that start with /notification_user to the code exported from packages/api/src/routes/notificationUserRoute.js. An excerpt from that file is shown below.

Each of the last two statements uses an Express router object to define the application's behavior for an endpoint-- an (HTTP request method, URL) pair. For example, the first endpoint is an HTTP GET request for the URL /notification_user/. (The HTTP request method is specified by calling the router's get() method; the URL is specified by appending the argument / to the base path where the router was mounted. This base path was the first argument in the app.use('/notification_user', notificationUserRoute) statement.) When a GET request for that URL is received, Express will first call the function checkUserFarmStatus, and then call the NotificationUserController.getNotifications method already discussed.

The last statement's endpoint is an HTTP PUT request sent to the same URL as the previous endpoint. In response, Express calls checkUserFarmStatus and then NotificationUserController.patchNotifications.

In addition to get and put, The Express router defines "route methods" for post and other HTTP request methods. These methods take the URL suffix (to be appended to the base path) as their first argument, followed by a variable number of functions. Generally, the last of these functions will be the main route handling code that responds to requests as we have seen in our examples.

When one or more functions are listed before the final route handler function, these functions are called "middleware". The middleware function checkUserFarmStatus is called for both of the endpoints defined above.

Like a route handler, middleware functions receive Express's req and res objects as arguments. They also receive an argument named next, which represents the next function listed for the endpoint. A middleware function can choose to call that next function or not; this means it can permit control to continue through the list of functions, or interrupt the flow.

For example, the middleware function above queries the Objection userFarmModel class to determine if the user making this request is an authorized user for the farm identified in the request. If so, the middleware calls next() which activates the route handler that we have already discussed. But if the user is not authorized for the farm, the middleware does not use next to activate the route handler. Instead, it handles the response itself by sending an HTTP status code 403 ("Forbidden") and a description of the problem.

LiteFarm generally follows the REST conventions for endpoints and status codes used in responses.

Backend tests

The packages/api/tests folder contains tests of the API built with Jest. You can read more about this here.