header img

About

Liberate your forms with libreForms-flask, a web-based implementation of the libreForms form manager API specification.

Competing browser-based form managers give form administrators little control over form fields, the resulting data, or the underlying web application. Likewise, few proprietary options provide both self-hosting support and a viable licensing model.

The libreForms project, first and foremost, defines a simple but highly extensible abstraction layer that matches form fields to data structures. On top of this specification, libreForms-flask provides a highly configurable browser-based application and a document-oriented database. In short, the platform allows you to do anything, host anywhere, and control everything about your internal form management system.

abstraction layer

Use Cases

The libreForms project is designed to meet the needs of enterprises of all sizes, for example:

You are a small enterprise that has been using Google Forms for your organization’s internal forms because it is low-cost, but you dislike the restricted features and lack of direct control over your data.
You are a medium-sized enterprise that wants a simple, low-cost tool to manage their internal forms. You don’t mind self-hosting the application, and you have staff with rudimentary experience using Python to deploy and maintain the system.
You are a large enterprise with significant technical staff that routinely host and maintain applications for use on your organization’s intranet. You periodically rely on physical or digitized forms, reports, and questionnaires. You have assessed options for form managers on the market and determined that proprietary services provide little direct control over the application source code, or otherwise fail to provide a viable licensing model.

While the libreForms abstraction layer does not proscribe a system for classifying forms, it remains aware of various purposes that enterprises may employ it, for example:

Forms: a standard submission subject to a routing and/or approval process
Reports: a periodic, standard submission used to track status or updates on a process
Ticket: a flexible submission initiating a process and subject to a routing and/or assignment process
Surveys: a standard submission used to aggregate sample and/or population data

Architecture

libreForms-flask is meant to run within an enterprise’s intranet behind a reverse proxy. It does not currently support high availability, but does spawn multiple workers on the system upon which it is deployed. See this discussion about accounting for enterprise requirements.

While intended primarily for internal use, libreForms-flask provides out-of-the-box support for external-facing, anonymous forms. These forms employ signed URLs, rather than local authentication, to control access to forms. These are intended to also run behind a reverse proxy that points to whatever external network (eg. the internet or another organization’s network) you’d like to provide access.

Here is an example diagram for such a deployment:

example architecture

Features

implements the libreForms spec using Python dictionaries
a flask web application that will work well behind most standard reverse-proxies
plotly dashboards for data visualization
a document-oriented database to store form data
basic local authentication
user groups and routing lists for form review, approval, and notifications
database lookups in form fields
automated reports on form submissions

Installation

In most cases, the following commands must be run with root privileges.

Docker

Currently, Docker installations are not supported but are under development.

RHEL 8

install dependencies

# see https://www.mongodb.com/docs/manual/tutorial/install-mongodb-on-red-hat/
echo "[mongodb-org-6.0]
name=MongoDB Repository
baseurl=https://repo.mongodb.org/yum/redhat/$releasever/mongodb-org/6.0/x86_64/
gpgcheck=1
enabled=1
gpgkey=https://www.mongodb.org/static/pgp/server-6.0.asc" | tee /etc/yum.repos.d/mongodb-org-6.0.repo
yum update -y
yum install python3.8 mongodb-org -y
systemctl enable --now mongod

You also need to install rabbitmq-server, see https://www.rabbitmq.com/install-rpm.html.

Download this repository into the opt directory.

Either download a stable release of the application.

cd /opt
wget https://github.com/libreForms/libreForms-flask/archive/refs/tags/X.X.X.tar.gz
tar -xvf *.*.*.tar.gz
mv libreForms-flask-*.*.*/ libreForms/

Or install the cutting-edge version of the application using Git.

cd /opt
git clone https://github.com/libreForms/libreForms-flask.git
mv libreForms-flask/ libreForms/

install Python virtual environment and initialize flask

cd /opt/libreForms
python3.8 -m venv venv
source venv/bin/activate
pip install -r requirements/app.txt

libreforms user

useradd --no-create-home --system libreforms
chown -R libreforms:libreforms /opt/libreForms

systemd service

cp /opt/libreForms/gunicorn/*.service /etc/systemd/system
systemctl daemon-reload
systemctl enable --now libreforms-gunicorn
systemctl enable --now libreforms-celery
systemctl enable --now libreforms-celerybeat

Amazon Linux 2

install dependencies

echo "[mongodb-org-6.0]
name=MongoDB Repository
baseurl=https://repo.mongodb.org/yum/amazon/2/mongodb-org/6.0/x86_64/
gpgcheck=1
enabled=1
gpgkey=https://www.mongodb.org/static/pgp/server-6.0.asc" | tee /etc/yum.repos.d/mongodb-org-6.0.repo 
yum update
yum install mongodb-org
amazon-linux-extras install python3.8
systemctl enable --now mongod

You also need to install rabbitmq-server, see https://www.rabbitmq.com/install-rpm.html.

Download this repository into the opt directory.

Either download a stable release of the application.

cd /opt
wget https://github.com/libreForms/libreForms-flask/archive/refs/tags/X.X.X.tar.gz
tar -xvf *.*.*.tar.gz
mv libreForms-flask-*.*.*/ libreForms/

Or install the cutting-edge version of the application using Git.

cd /opt
git clone https://github.com/libreForms/libreForms-flask.git
mv libreForms-flask/ libreForms/

install Python virtual environment and initialize flask

cd /opt/libreForms
python3.8 -m venv venv
source venv/bin/activate
pip install -r requirements/app.txt

libreforms user

useradd --no-create-home --system libreforms
chown -R libreforms:libreforms /opt/libreForms

systemd service

cp /opt/libreForms/gunicorn/*.service /etc/systemd/system
systemctl daemon-reload
systemctl enable --now libreforms-gunicorn
systemctl enable --now libreforms-celery
systemctl enable --now libreforms-celerybeat

Ubuntu 20.04

install dependencies

apt update -y && apt upgrade -y
apt install -y mongodb python3-pip python3-venv rabbitmq-server # for the most up to date version of mongodb, see https://www.mongodb.com/docs/manual/tutorial/install-mongodb-on-ubuntu/
systemctl enable --now mongodb

Download this repository into the opt directory:

Either download a stable release of the application.

cd /opt
wget https://github.com/libreForms/libreForms-flask/archive/refs/tags/X.X.X.tar.gz
tar -xvf *.*.*.tar.gz
mv libreForms-flask-*.*.*/ libreForms/

Or install the cutting-edge version of the application using Git.

cd /opt
git clone https://github.com/libreForms/libreForms-flask.git
mv libreForms-flask/ libreForms/

install Python virtual environment and initialize flask

cd /opt/libreForms
python3 -m venv venv
source venv/bin/activate
pip install -r requirements/app.txt

libreforms user

useradd --no-create-home --system libreforms
chown -R libreforms:libreforms /opt/libreForms

systemd service

cp /opt/libreForms/gunicorn/*.service /etc/systemd/system
systemctl daemon-reload
systemctl enable --now libreforms-gunicorn
systemctl enable --now libreforms-celery
systemctl enable --now libreforms-celerybeat

hCaptcha

If you’d like to install hCaptcha dependencies, run the following:

pip install -r requirements/hcaptcha.txt

Flower

If you’d like to install Flower dependencies, run the following:

pip install -r requirements/flower.txt

Then, you can start flower by enabling and starting the systemd service.

systemctl enable --now libreforms-flower

Elasticsearch

If you want to run Elasticsearch, you can install it on Ubuntu 20.04 as follows:

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list
apt update
apt install elasticsearch
systemctl enable --now elasticsearch.service

MongoDB

If you’d like to set up a mongodb user, run mongosh --port 27017 and enter the following configuration:

use libreforms
db.createUser(
    {
        user: "libre",
        pwd: passwordPrompt(), // or cleartext password
        roles: [ 
            { role: "dbAdmin", db: "libreforms" },
            { role: "readWrite", db: "libreforms" } 
        ]
    }
)

Next, add the following configuration to /etc/mongod.conf:

security:
  authorization: enabled

And finally, run echo YOUR_PASSWORD > /opt/libreForms/mongodb_creds, replacing YOUR_PASSWORD with the password you set for mongodb. Ensure this file is owned by the libreforms user by running chown -R libreforms:libreforms /opt/libreForms again.

You should double check that this, and any other credential / key file like smtp_creds or secret_key are modified to have limited (at most 600) permissions by running chmod 600 mongodb_creds secret_key smtp_creds.

Common Issues

Failure to start Systemd Unit: if you experience a failure when you check systemctl status libreforms, then try chowning the program files and restarting the application.

chown -R libreforms:libreforms /opt/libreForms
systemctl restart libreforms

pymongo.errors.AutoReconnect: connection pool paused: if you receive this error, try restarting the application by running systemctl stop libreforms; systemctl start libreforms. If this doesn’t work, try the following:

chown -R mongodb:mongodb /var/lib/mongodb # often the permissions aren't set up correctly
chown mongodb:mongodb /tmp/mongodb-27017.sock # borrowed from https://stackoverflow.com/a/64810086
systemctl restart mongod

Abstraction Layer

libreForms constitutes a set of rules, or an abstraction layer, that can be used to build browser-based forms in Python. To accomplish this, the specification describes a configuration file that stores all the information needed to generate forms and parse form data into a cohesive data structure while accounting for additional features an organization might want to add.

The libreForms specification is defined in libreforms/__init__.py and, when used in conjuction with the web application, it expects administrators will overwrite the default form used to illustrate the rules by adding a file called libreforms/form_config.py. At this time, the specification can handle the “text”, “password”, “radio”, “select”, “checkbox”, “date”, “hidden”, and “number” input types, and can write to Python’s str, float, int, and list data types.

At a high level, the abstraction layer breaks down individual forms into fields and configurations. A field must have a unique name, which must employ underscores instead of spaces (“My Form Field” would not work, but “My_Form_Field” is a correct field name). Configuration names are preceded by an underscore (eg. “_dashboard” or “_allow_repeats”) and allow form administrators to define unique, custom form behavior. All built-in configurations default to a value of False.

Here is an overview of the abstraction layer in versions <1.0.0. Specifically, the following snippet defines a single form called sample-form with a handful of fields like Text_Field, Pass_Field, etc. and a dashboard view for the form using the _dashboard configuration.

forms = {
    "sample-form": {
        "Text_Field": {
            "input_field": {"type": "text", "content": ["NA"]},
            "output_data": {"type": "str", "required": False, "validators": [lambda p: len(p) >= 6]},
        },
        "Pass_Field": {
            "input_field": {"type": "password", "content": [""]},
            "output_data": {"type": "str", "required": False, "validators": []},
        },
        "Radio_Field": {
            "input_field": {"type": "radio", "content": ["Pick", "An", "Option"]},
            "output_data": {"type": "str", "required": False, "validators": []},
        },
        "Check_Field": {
            "input_field": {"type": "checkbox", "content": ["Pick", "An", "Option"]},
            "output_data": {"type": "list", "required": False, "validators": []},
        },
        "Date_Field": {
            "input_field": {"type": "date", "content": [datetime.datetime.today().strftime("%Y-%m-%d")]},
            # "input_field": {"type": "date", "content": []},
            "output_data": {"type": "str", "required": False, "validators": []},
        },
        "Hidden_Field": {
            "input_field": {"type": "hidden", "content": ["This field is hidden"]},
            "output_data": {"type": "str", "required": False, "validators": []},
        },
        "Float_Field": {
            "input_field": {"type": "number", "content": [0]},
            "output_data": {"type": "float", "required": False, "validators": []},
        }, 
        "Int_Field": {
            "input_field": {"type": "number", "content": [0]},
            "output_data": {"type": "int", "required": False, "validators": []},
        }, 
        "_dashboard": {             # defaults to False
            "type": "scatter",      # this is a highly powerful feature but requires
            "fields": {             # some knowledge of plotly dashboards; currently
                "x": "Timestamp",   # only line charts with limited features supported
                "y": "Int_Field", 
                "color": "Text_Field"
            }
        },
        "_allow_repeat": False, # defaults to False
        "_allow_uploads": True, # defaults to False
        "_allow_csv_templates": True, # defaults to False
        "_suppress_default_values": False, # defaults to False
    },
}

Versions above 2.0.0 will introduce compatibility-breaking changes that are intended to simplify the abstraction layer. For more information, see the discussion about these changes.

Database Lookups

You can configure the database lookups in the abstraction layer with some relatively straightforward hacking. For example, if you wanted the possible values of some field X in form Y to be drawn from the stored values for field A in form B, then you could add the following code to your libreforms/form_config.py file.

import os
import pandas as pd
from app import mongo

if os.path.exists ("mongodb_creds"):
    with open("mongodb_creds", "r") as f:
        mongodb_creds = f.read().strip()
else:  
    mongodb_creds=None

mongodb = mongo.MongoDB(dbpw=mongodb_creds)

def _db_lookup(collection, *args, combine=False):
    df = pd.DataFrame(list(mongodb.read_documents_from_collection(collection)))
    new_df = pd.DataFrame()
    if args:
        for a in args:
            new_df[a] = df[a]
        if combine:
            new_df['combine'] = ""
            for a in args:
                new_df['combine'] = new_df['combine'] + df[a] + " "
        return new_df
    else:
        return df

Then, when you’re constructing the form field described above, you can define is as follows.

forms = {
    "Form_Y": {
        "Field_X": {
            "input_field": {"type": "select", "content": [r'{}'.format(x) for x in _db_lookup("Form_B")['Field_A']]},
            "output_data": {"type": "str", "required": True, "validators": [], "description": "Select one of the available options",},
        },
    }
}

Web Application

In this repository, a web application written in Flask sits atop the libreForms abstraction layer defined above. This application includes basic authentication, provides access to forms, tabular views of form responses, and dashboards when these have been defined in the abstraction layer.

Views

The application provides table and dashboard views for form data. Right now, line graphs are the only supported form of Plotly dashboard but, in the future, the project plans to allow arbitrary dashboard configurations using kwargs in the _dashboard configuration in the abstraction layer. The application allows users to tailor the data in their dashboards and tables using GET variabes as selectors. For example, when you define a dashboard for a given form, you need to set a dependent variable. However, this can be overridden by passing the ?y=field_name GET variable in the browser. Likewise, you can tailor tabular data by passing the ?FIELD_NAME=VALUE GET variable in the browser. Put another way, if a table has a field called Sub-Unit and another called Fiscal_Year, and you would like to tailor the table to only show data for the Finance sub-unit in the 2021 fiscal year, then you could pass the following GET variables: ?Sub-Unit=Finance&Fiscal_Year=2021 to select only this data.

In addition, the default site display options can be overridden by adding a file called .py to the app/ directory. This file should contain a dictionary object with key-value attributes that you want to override.

display = {
    'site_name':"My-Site",
    'homepage_msg': "Welcome to My-Site. Select a view from above to get started.",
    'warning_banner':"Please be mindful of the data you post on this system.",
    'favicon':"my_new_favicon.ico",
    'default_org':"My Organization's Name",
}

Auth

By default, this application employs an sqlite user database. The default login credentials are

User: libreforms
Pass: libreforms

In addition to username and password fields, the application ships by default with phone number, organization, and email fields. These can be modified by changing the fields defined in app/schema.sql and app/auth.py. Currently, there is no group-based permission sets defined, but these will be added per https://github.com/libreForms/libreForms-flask/issues/16.

Anonymous registration is enabled by default but can be modified by setting config[‘allow_anonymous_registration’] to False. Bulk registration can be enabled by seeting config[‘allow_bulk_registration’] to True, which then allows logged-in users to upload a CSV of users to add at /auth/register/bulk.

REST

There is a RESTful API that allows users to complete read and, perhaps in the future, write operations against the MongoDB database, see discussion here. This requires API keys. You can create a file in the application home directory called api_keys structured as:

api_keys
YOUR_KEY_HERE

But if you don’t, the default test key will be t32HDBcKAAIVBBPbjBADCbCh. In the future, we may choose to manage API keys, along with signed URLs, using a database, see here.

Database

When data is written from the web application to the database backend, it appends the following fields:

- _id: unique id for each db write
- Reporter: the username of the reporting user
- Timestamp: the timestamp that the form was submitted

If you elect to password protect your database, which is recommended, you should drop a file in the application home directory named mongodb_creds and ensure that the libreforms user has read access to this file.

Mail

The web application will authenticate a mail server when you add a file called smtp_creds to the working directory. This file should conform to the following format:

mail_server,port,username,password,from_address
smtp-server.example.com,587,USERNAME,PASSWORD,libreForms <libreforms@example.com>

Web Server

You can set up a web server on the same system running libreForms, or externalize it. We provide an example Nginx config that you can adapt to meet your needs. After installing Nginx using your system package manager, you can run the following to set up a web server with SSL/TLS.

cp /opt/libreforms/gunicorn/example-nginx.libreforms.conf /etc/nginx/sites-available/libreforms.conf
sudo ln -s /etc/nginx/sites-available/libreforms.conf /etc/nginx/sites-enabled/libreforms.conf
mkdir /opt/libreforms/certificates
openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /opt/libreforms/certificates/libreforms.example.com.key -out /opt/libreforms/certificates/libreforms.example.com.pem

If you’d like to install Let’s Encrypt certificates, follow your distribution’s instructions for installing eg. using certbot.

Reporting

[Placeholder]

Dependencies

The flask application has a few dependencies that, in its current form, may be prone to obsolescence; there is an issue in the backlog to test for, among other things, obsolete and vulnerable dependencies. In addition to the standard requirements, like MongoDB, Python3, Python3-Pip, Python3-Venv, and Rabbit-mq (or Redis), here is a list of dependencies that ship with the application under the static/ directory:

bootstrap-darkly-5.1.3.min.css
bootstrap-5.1.3.bundle.min.js
plotly-v1.58.5.min.js

As well as the following python requirements and their dependencies:

celery==5.2.7
cryptography==38.0.4
Flask==2.2.2
Flask-Login==0.6.2
Flask-SQLAlchemy==3.0.2
gunicorn==20.1.0
pandas==1.5.2
plotly==5.11.0
pymongo==4.3.3
reportlab==3.6.12
webargs==8.2.0

In the tests requirements file, we add the following requirements

coverage==6.4.1
pytest==7.1.2

There are also optional requirements files for hCaptcha and Flower.

Copyright

libreForms is an open form manager API
Copyright (C) 2022 Sig Janoska-Bedi

This program is free software: you can redistribute it and/or modify
it under the terms of the GNU Affero General Public License as
published by the Free Software Foundation, either version 3 of the
License, or (at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU Affero General Public License for more details.

You should have received a copy of the GNU Affero General Public License
along with this program.  If not, see <https://www.gnu.org/licenses/>.