This is often called the "backend". It consists of database-oriented features, plus the API implementation proper, and the independent jobs process.
The database
LiteFarm uses PostgreSQL, a relational database management system, for permanent storage of most data. LiteFarm code uses the Knex query builder to interact with Postgres. When additions or changes to the database schema are needed, we rely on Knex's migrations mechanism. Migrations are JavaScript code that use Knex to modify or extend the database schema. (That is, migrations are mostly concerned with the definition of tables, columns, and other "structural" elements. Less commonly, migrations can deal with data contents, but this is usually limited to "reference" data such as lists of countries, etc.)
Migrations are stored in the packages/api/db/migration folder. A new migration is created with the command knex migrate:make descriptive_name, which creates a new template file named YYYYMMDDHHMM_descriptive_name.js , where the prefix indicates the date and time.
Database tables for LiteFarm's notifications features were initially created by the migrations in 20220207164158_notification.js. From the requirements it was clear that we would need two tables. One table (notification) holds the title or subject line, message body, and other characteristics of the message itself. The second table (notification_user) tracks the status of a notification with respect to an individual user-- have they read it, etc. Here is an excerpt from the original migration file.
exports.up = async function (knex) {
await knex.schema.createTable('notification', function (table) {
table.uuid('notification_id').primary();
table.string('title').notNullable();
table.text('body').notNullable();
table.enum('ref_table', ['task', 'location', 'users', 'farm', 'document', 'export', ]);
table.enum('ref_subtable', [/* ... */]);
table.string('ref_pk');
table.uuid('farm_id').references('farm_id').inTable('farm');
table.boolean('deleted').defaultTo(false);
table.string('created_by_user_id').references('user_id').inTable('users');
table.string('updated_by_user_id').references('user_id').inTable('users');
table.dateTime('created_at').notNullable();
table.dateTime('updated_at').notNullable();
});
await knex.schema.createTable('notification_user', function (table) {
table.primary(['notification_id', 'user_id']);
table.uuid('notification_id').references('notification_id').inTable('notification');
table.string('user_id').references('user_id').inTable('users');
table.boolean('alert').defaultTo(true).notNullable();
table.enum('status', ['Unread', 'Read', 'Archived']).defaultTo('Unread').notNullable();
table.boolean('deleted').defaultTo(false);
table.string('created_by_user_id').references('user_id').inTable('users');
table.string('updated_by_user_id').references('user_id').inTable('users');
table.dateTime('created_at').notNullable();
table.dateTime('updated_at').notNullable();
});
}
exports.down = async function (knex) {
await knex.schema.dropTable('notification_user');
await knex.schema.dropTable('notification');
}
Once a migration has been committed to the repository, it should not be modified. Instead, we write additional migrations as needed to evolve the database schema. For example, the notification table should have been defined to generate the universal identifiers for its records, but this was overlooked. So migration 20220221162055_generate_notification_id.js resolved this.
exports.up = function (knex) {
return Promise.all([
knex.raw(
'ALTER TABLE notification ALTER COLUMN notification_id SET DEFAULT uuid_generate_v4();',
),
]);
};
exports.down = function (knex) {
return Promise.all([
knex.raw('ALTER TABLE notification ALTER COLUMN notification_id DROP DEFAULT;'),
]);
};
As work on the notifications features progressed, it became clear that the structure of the notifications table needed to be rethought, and it was modified to its current form by migration 20220325020707_refactor-notification.js.
const newFields = [
{ name: 'translation_key', defn: 'character varying(255) COLLATE pg_catalog."default"' },
{ name: 'variables', defn: 'jsonb' },
{ name: 'entity_id', defn: 'character varying(255) COLLATE pg_catalog."default"' },
{ name: 'entity_type', defn: 'character varying(255) COLLATE pg_catalog."default"' },
{ name: 'context', defn: 'jsonb' },
];
const oldFields = [
{ name: 'title', defn: 'character varying(255) COLLATE pg_catalog."default"' },
{ name: 'body', defn: 'text COLLATE pg_catalog."default"' },
{ name: 'ref_table', defn: 'text COLLATE pg_catalog."default"' },
{ name: 'ref_subtable', defn: 'text COLLATE pg_catalog."default"' },
{ name: 'ref_pk', defn: 'character varying(255) COLLATE pg_catalog."default"' },
];
exports.up = async function (knex) {
for (const field of oldFields) {
console.log(`ALTER TABLE notification DROP COLUMN ${field.name};`);
await knex.raw(`ALTER TABLE notification DROP COLUMN ${field.name};`);
}
for (const field of newFields) {
console.log(`ALTER TABLE notification ADD COLUMN ${field.name} ${field.defn};`);
await knex.raw(`ALTER TABLE notification ADD COLUMN ${field.name} ${field.defn};`);
}
};
exports.down = async function (knex) {
for (const field of newFields) {
await knex.raw(`ALTER TABLE notification DROP COLUMN ${field.name};`);
}
for (const field of oldFields) {
await knex.raw(`ALTER TABLE notification ADD COLUMN ${field.name} ${field.defn};`);
}
};
Migration code is run whenever the system is deployed. For example, when Docker containers are created as a new deployment of LiteFarm, all of the migrations will be run, in chronological order. This means the database schema quickly evolves from its initial definition, up through all changes to date.
The migration mechanism uses a special table in the database to track which individual migrations have been applied to the database. When a long-lived deployment (such as production) is updated, only the new migrations are applied. This modifies the database code as needed to be compatible with the new code being deployed.
Migrations can be "undone" in reverse chronological order, because each migration defines a function up for making the changes, and a function down for reverting the changes. In practice, LiteFarm rarely uses this feature.
In a LiteFarm deployment, the database runs as litefarm-db, a Docker container built from the official Postgres image.
The API code
An Application Programming Interface (API) is an interface that a software product provides for use by other software, rather than by human users.
The code for the LiteFarm API is in the packages/api/src folder. In a LiteFarm deployment, this Node.js server runs in a Docker container named litefarm-api. The container is built to put the API code in a standard Node image, and also uses a standard Nginx image to create a reverse proxy server.
The API borrows the Model and Controller concepts from the Model-View-Controller (MVC) architecture pattern.
Models
In the MVC pattern, a model is a dynamic data structure that directly manages the data, logic and rules of the application. Often, a model corresponds to a table in relational databases.
LiteFarm uses Objection.js to define the models. Objection extends the Knex query builder.
Code for the model that corresponds to the notification_user database table is found in packages/api/src/models/NotificationUserModel.js. A portion of the code is shown below.
/**
* Models data persistence for users' notifications.
*/
class NotificationUser extends baseModel {
/**
* Tracks open subscription channels for server-sent events. To support multiple sessions by the same user,
* keys are user IDs; values are Maps with timestamp keys and HTTP response object values.
* @member {Map}
* @static
*/
static subscriptions = new Map();
/**
* Identifies the database table for this Model.
* @static
* @returns {string} Names of the database table.
*/
static get tableName() {
return 'notification_user';
}
/**
* Identifies the primary key fields for this Model.
* @static
* @returns {string[]} Names of the primary key fields.
*/
static get idColumn() {
return ['notification_id', 'user_id'];
}
/**
* Supports validating instances of this Model class.
* @static
* @returns {Object} A description of valid instances.
*/
static get jsonSchema() {
return {
type: 'object',
required: ['user_id'],
properties: {
notification_id: { type: 'string' },
user_id: { type: 'string' },
alert: { type: 'boolean' },
status: {
type: 'string',
/**
* @name userNotificationStatusType
* @desc Enumerated type for user notification status.
* @enum
* */
enum: ['Unread', 'Read', 'Archived'],
},
...this.baseProperties,
},
additionalProperties: false,
};
}
/**
* Defines this Model's associations with other Models.
* @static
* @returns {Object} A description of Model associations.
*/
static get relationMappings() {
return {
notification: {
relation: Model.BelongsToOneRelation,
modelClass: Notification,
join: {
from: 'notification_user.notification_id',
to: 'notification.notification_id',
},
},
};
}
// custom methods not shown here.
}
Each Objection model is a JavaScript class. Most extend a class named BaseModel, which implements what are called the "base properties" (explained below).
The code above shows this model's implementation of several static property accessors defined by the Objection API.
tableName()tells Objection the database table name for this modelidColumn()indicates the table columns needed to uniquely identify a table rowjsonSchema()supports validating the data contents of class instances before database modifications are made. Objection uses the embedded information to determine the appropriate data types, which values are required to be present, etc.relationMappings()describes how this model is related to others. Each instance of this class belongs to one instance of theNotificationmodel class.
In jsonSchema(), the spread syntax expression ...this.baseProperties is used to include the base model's "base properties" in validating instances of class NotificationUser. These properties are "bookkeeping" columns found in most LiteFarm database tables. They record who created the row, and when; who last updated the row, and when; and whether the row is marked as "deleted". (In most cases, LiteFarm performs "soft deletes"; rather than actually deleting a database row, the row is marked as deleted, and excluded from further use.)
Controllers
In the MVC pattern, a controller acts as a kind of translator between the model(s) on one hand, and a client on the other. In general terms, a controller accepts input from the UI, converts it to commands to the model(s), transforms the results and responds to the UI.
In terms specific to LiteFarm, a controller is a file of functions that handle all requests related to a particular type of entity. For example, packages/api/src/controllers/notificationUserController.js contains all the functions that handle requests involving the current user's notifications.
These functions are built on the Express.js framework, which we will explore in greater detail below. For now, it is enough to understand that a controller function always receives (at least) two parameters. These are references to objects that represent the HTTP request received from the client, and the HTTP response that we are currently putting together. Here is an example extracted from the file named in the previous paragraph.
/**
* Responds with the user's notifications regarding their current farm.
* @param {Request} req - The HTTP request object.
* @param {Response} res - The HTTP response object.
* @async
*/
async getNotifications(req, res) {
try {
const notifications = await NotificationUser.getNotificationsForFarmUser(
req.headers.farm_id,
req.user.user_id,
);
res.status(200).send(notifications);
} catch (error) {
console.log(error);
res.status(400).json({ error });
}
},
When the client requests a list of notifications to display to the user, the Express framework calls this function. The call passes in req and res which represent the HTTP request and response, respectively.
The function trys a call to the NotificationUser model's method, getNotificationsForFarmUser(). (Some context: a registered LiteFarm user is always logged into the context of one particular farm. The request we are handling here means approximately "show me my notifications related to the farm I'm now logged into".) To fetch the necessary data, the model needs to know two things: who is the user, and what is their current farm? These values are part of the request (req.user.user_id, and req.headers.farm_id respectively) and the controller passes the values in its call to the model's method.
If the model successfully returns results, the controller responds to the client, with an HTTP success code (200) and the notifications data from the model. (The Express framework will send that data as JSON.) If there is a problem, the catch block has the controller log the error, and responds to the client with a HTTP "Bad Request" code (400) and JSON information about the specific error.
Here is another example from the same controller.
/**
* Handles requests to update user notifications.
* @param {Request} req - The HTTP request object.
* @param {Response} res - The HTTP response object.
* @async
*/
async patchNotifications(req, res) {
const payload = { ...req.body };
delete payload.notification_ids;
try {
await NotificationUser.update(req.user.user_id, req.body.notification_ids, payload);
res.sendStatus(200);
} catch (error) {
console.log(error);
res.status(400).json({ error });
}
},
When the client requests to update the user's notifications, Express calls this function. The controller unpacks the request to transform its details into the form needed by the model. Specifically, it creates a new object as a copy of the request's body and deletes the notification_id property from that copy. It then calls the model's update() method, passing the user_id, the array of notification IDs, and the modified request body.
Take a moment to consider the separation of concerns between the controller and the model. The controller sticks to its job: it translates a specific form of incoming HTTP request into commands that the model knows how to handle. The controller does not and should not know anything about database table names, columns, or even that we are using a database; it just knows how to talk to the model through a set of method calls that keep concerns separated.
Meanwhile, the model is given the information it needs to do its job: managing the storage and retrieval of data while applying appropriate logic and rules. It is not passed the entire request object; that is an HTTP concept and HTTP is outside the model's concerns. Accordingly, the model does not return an HTTP status code-- it doesn't know them. It either returns the data that the controller asked for, or it throws an error to indicate why it cannot.
Routes and middleware
The previous section stated that the Express framework calls notificationUserController.getNotifications() method when the client requests a list of the users notification, but did not explain how the client makes that request, or how Express knows what code to call.
The explanation lies in Express routing. When the API server starts, the setup routines include the statements const notificationUserRoute = require('./routes/notificationUserRoute'); and, much later, app.use('/notification_user', notificationUserRoute). This configures the application to route all requests to URLs that start with /notification_user to the code exported from packages/api/src/routes/notificationUserRoute.js. An excerpt from that file is shown below.
const NotificationUserController = require('../controllers/notificationUserController');
const checkUserFarmStatus = require('../middleware/acl/checkUserFarmStatus');
const express = require('express');
const router = express.Router();
router.get('/', checkUserFarmStatus(), NotificationUserController.getNotifications);
router.patch('/', checkUserFarmStatus(), NotificationUserController.patchNotifications);
Each of the last two statements uses an Express router object to define the application's behavior for an endpoint-- an (HTTP request method, URL) pair. For example, the first endpoint is an HTTP GET request for the URL /notification_user/. (The HTTP request method is specified by calling the router's get() method; the URL is specified by appending the argument / to the base path where the router was mounted. This base path was the first argument in the app.use('/notification_user', notificationUserRoute) statement.) When a GET request for that URL is received, Express will first call the function checkUserFarmStatus, and then call the NotificationUserController.getNotifications method already discussed.
The last statement's endpoint is an HTTP PUT request sent to the same URL as the previous endpoint. In response, Express calls checkUserFarmStatus and then NotificationUserController.patchNotifications.
In addition to get and put, The Express router defines "route methods" for post and other HTTP request methods. These methods take the URL suffix (to be appended to the base path) as their first argument, followed by a variable number of functions. Generally, the last of these functions will be the main route handling code that responds to requests as we have seen in our examples.
When one or more functions are listed before the final route handler function, these functions are called "middleware". The middleware function checkUserFarmStatus is called for both of the endpoints defined above.
const userFarmModel = require('../../models/userFarmModel');
const checkUserFarmStatus = (status = 'Active') => async (req, res, next) => {
const { user_id, farm_id } = req.headers;
const userFarm = await userFarmModel.query().where({ user_id, farm_id }).first();
return userFarm && userFarm.status === status ? next() : res.status(403).send('Do not have access to this farm');
}
module.exports = checkUserFarmStatus;
Like a route handler, middleware functions receive Express's req and res objects as arguments. They also receive an argument named next, which represents the next function listed for the endpoint. A middleware function can choose to call that next function or not; this means it can permit control to continue through the list of functions, or interrupt the flow.
For example, the middleware function above queries the Objection userFarmModel class to determine if the user making this request is an authorized user for the farm identified in the request. If so, the middleware calls next() which activates the route handler that we have already discussed. But if the user is not authorized for the farm, the middleware does not use next to activate the route handler. Instead, it handles the response itself by sending an HTTP status code 403 ("Forbidden") and a description of the problem.
LiteFarm generally follows the REST conventions for endpoints and status codes used in responses.
Backend tests
The packages/api/tests folder contains tests of the API built with Jest. From a terminal in the packages/api folder, you can use the command npm test to run all these tests. Or, npx jest --runInBand tests/notificationUser.test.js will run a single file of tests-- in this case, the tests for the code we have explored. Here is an excerpt.
const mocks = require('./mock.factories');
describe('Notification tests', () => {
function getRequest(url, { user_id = user.user_id, farm_id = farm.farm_id }, callback) {
chai.request(server).get(url).set('user_id', user_id).set('farm_id', farm_id).end(callback);
}
let user;
let farm;
let userFarm;
beforeEach(async () => {
[user] = await mocks.usersFactory();
[farm] = await mocks.farmFactory();
[userFarm] = await mocks.userFarmFactory({ promisedUser: [user], promisedFarm: [farm] });
const middleware = require('../src/middleware/acl/checkJwt');
middleware.mockImplementation((req, res, next) => {
req.user = {};
req.user.user_id = req.get('user_id');
next();
});
});
describe('GET user notifications', () => {
test('Users should get their notifications scoped for their current farm', async (done) => {
const [notification] = await mocks.notification_userFactory({
promisedUserFarm: [userFarm],
});
getRequest('/notification_user', {}, (err, res) => {
expect(err).toBe(null);
expect(res.status).toBe(200);
expect(res.body.length).toBe(1);
expect(res.body[0].user_id).toBe(user.user_id);
expect(res.body[0].notification_id).toBe(notification.notification_id);
done();
});
});
beforeEach configures the Jest framework to run the setup needed before each individual test. describe is used to organize groups of tests. test defines an individual test with an English description and a function that receives done. The argument is a callback function that the test must call when it has finished.
The test shown relies on database contents created by beforeEach. The userFarm object contains details of these database contents, and the test passes it to notification_userFactory to create additional database content specific to this test. This establishes known database contents as a basis for testing data retrieval by the API route "under test". (The mocks collection defines "factory" functions for most LiteFarm database record types. These support testing by creating database records as needed.)
The test then uses a helper function to make a GET request to the URL for retrieving the user's notifications for their current farm. This executes the code under test, which we expect to retrieve known data values. When the request completes, Jest runs the anonymous function that receives two arguments representing any error, and the API response.
expect calls are the actual testing; they describe the expected results. Conditional logic is embedded within expect so that any unsatisfied expectations will cause the test to fail. If all expectations are met and the code calls done(), then the test passes.
The output below includes the English provided to the describe and test calls above. The second green check mark indicates that the test shown in the excerpt has passed.
