Flindt without Docker

Introduction

This is a testcase aka. Proof of Concept to run the Spindle feedback-tool “Flindt” without permanent Docker-containers. The setup was done on macOS Sierra using Vagrant 1.9.7. for VirtualBox, using Vagrant Box debian/contrib-jessie64.

Vagrant plugins used:

  • vagrant-cachier
  • vagrant-hostsupdater
  • vagrant-share
  • vagrant-vbguest

Adding the Vagrantbox can be done with: vagrant box add debian/contrib-jessie64; installing a specific Vagrant-plugin with vagrant plugin install vagrant-cachier.

It is advised to use the Vagrantfile from the appendix below.

Note

Use your own DNS-names for cust.flindt.org (frontend) and api.cust.flindt.org (backend)!

Warning

You will also need user-accounts at GlassFrog, Google Developer and Slack to make full use of this holacracy feedback-tool!

Frontend

Start by building the static content use by the frontend.

Edit file frontend/src/constants/constants.js to contain the right value for your GOOGLE_CLIENT_ID.

Option 1

The easy way, still using Docker though…

Build with ./build-frontend.sh (see in appendix) outside of the Vagrant VM. For this, you need to have Docker installed on your host. The resulting folder uploads/ can be uploaded to a webserver. It contains static content (img / js / css).

FYI Vagrant will mount the whole Flindt-directory, including the generated content in upload, under /srv/flindt.

Option 2

Note

Installing NodeJS in the VM as described below, is not needed when you build the frontend one time with Docker and serve the generated static files from the upload-folder.

Install NodeJS in the VM

curl -sL https://deb.nodesource.com/setup_8.x | sudo -E bash -
apt-get install -y nodejs

Install node modules

npm install
export NODE_PATH=/srv/flindt/frontend/node_modules
npm install -g gulp-cli webpack@1.14.0

Vagrant / Linux

Run vagrant up.

Once the box is up and running, do vagrant ssh followed by sudo -i. You are now root in the virtual machine. Next install some essential packages. So as user root:

apt-get install python3.4-dev python3-pip virtualenvwrapper python3-virtualenv \
 libjpeg-dev curl zlib1g-dev mysql-server libmysqlclient-dev nginx-full

While installing, you will be prompted to set a password for the MySQL admin-user. Take note of this for later usage. You’ll need it to create a database in the next step and in the .env-file for the backend.

Add export WORKON_HOME="/srv/virtualenvs" to file /root/.bashrc. It’s needed for the backend’s virtualenv.

Setup the “virtualenvs”, based on Python 3.4:

mkdir -p /srv/virtualenvs
export WORKON_HOME="/srv/virtualenvs"
mkvirtualenv -a /srv/flindt -p /usr/bin/python3.4 flindt

Log out and back in as user root to activate the just installed virtualenv-functions.

(More on Python Virtual Environments.)

Backend

First create a MySQL-database name “flindt”:

# mysql -u root -p
Enter password:

mysql> create database flindt CHARACTER SET utf8 COLLATE utf8_general_ci;
Query OK, 1 row affected (0.00 sec)

mysql> exit

Run workon flindt to activate the virtualenv. The working directory will change to /srv/flindt.

Enter the backend-directory, upgrade pip and install the Python-packages as stated in the requirements-files:

cd backend
pip install -U pip
pip install -r requirements-dev.txt

In the file .env (in directory backend), adapt values as needed, e.g. the DATABASE_URL, MySQL-password and the server-names:

SOCIAL_AUTH_GOOGLE_PLUS_KEY=SomeSecretKey.apps.googleusercontent.com
CORS_ORIGIN_WHITELIST=localhost:8005,cust.flindt.org,api.cust.flindt.org
ALLOWED_HOSTS=localhost,127.0.0.1,cust.flindt.org,api.cust.flindt.org
DATABASE_URL=mysql://root:password@localhost/flindt
FRONTEND_HOSTNAME=cust.flindt.org
DEBUG=1

Now the application can be setup:

python manage.py migrate
python manage.py createsuperuser
python manage.py collectstatic

nginx

Web server “nginx” will be used to server the static content from the frontend in upload and as a reverse proxy to the uWSGI-server, listening on port 8005. Used config files can be found below in the appendix.

cd /etc/nginx/sites-available
cp *.flindt.org . # Copy the nginx vhosts-files from the appendix
cd ../sites-enabled
ln -s /etc/nginx/sites-available/* .
rm default
nginx -t # configuration test

uWSGI

Start uWSGI with uwsgi --daemonize2 uwsgi.log --ini flindt_backend.ini.

Again the used config file flindt_backend.ini can be found in the appendix.

Finally you can login with the superuser-account at http://api.cust.flindt.org/admin/ and add the application as described in flindt.readthedocs.io.

Connecting to the Flindt-frontend should result in a page that says Welcome to Flindt!

Google API

Some Google related errors encountered along the way, accompanied by what fixed it.

Not a valid origin

"Not a valid origin for the client: http://cust.flindt.org has not been whitelisted for \
client ID 197265621337-blahblahblah.apps.googleusercontent.com. Please go to \
https://console.developers.google.com/ and whitelist this origin for your project's client ID."

FIX: Domain verification from Google to DNS-provider.

Added domain “flindt.org” to the whitelist.

Not allowed access

XMLHttpRequest cannot load https://api.flindt.wearespindle.com/api-social-auth/convert-token/. \
Response to preflight request doesn't pass access control check: \
No 'Access-Control-Allow-Origin' header is present on the requested resource. \
Origin 'http://cust.flindt.org' is therefore not allowed access.

FIX: Recompile (and upload) the frontend with the right name / URL in file constants.js!

Can’t reach Google API / error 502

XMLHttpRequest cannot load http://api.cust.flindt.org/api-social-auth/convert-token/. \
Response to preflight request doesn't pass access control check: \
No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin \
'http://cust.flindt.org' is therefore not allowed access. The response had HTTP status code 502.

FIX: Added wide-open CORS-config to nginx vhost-config cust.flindt.org.

See CORS on Nginx.

Appendix

Configuration files

Configuration files not found in the default Flindt repository.

Vagrant

Note

Entry cust.flindt.org is added by Vagrant, but you’ll need to add a line 192.168.33.10 api.cust.flindt.org to file /etc/hosts manually!

File Vagrantfile:

# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure("2") do |config|
  config.vm.provider :virtualbox do |vb|

    # change the network card hardware for better performance
    vb.customize ["modifyvm", :id, "--nictype1", "virtio" ]
    vb.customize ["modifyvm", :id, "--nictype2", "virtio" ]

    # suggested fix for slow network performance
    # see https://github.com/mitchellh/vagrant/issues/1807
    vb.customize ["modifyvm", :id, "--natdnshostresolver1", "on"]
    vb.customize ["modifyvm", :id, "--natdnsproxy1", "on"]
  end

  config.vm.box = "debian/contrib-jessie64"
  config.ssh.insert_key = false

  config.vbguest.auto_update = false

  # To speed up installation cache packages
  # on the local drive
  if Vagrant.has_plugin?("vagrant-cachier")
    config.cache.scope = :box
    config.cache.synced_folder_opts = {
      type: :nfs,
      mount_options: ['rw', 'vers=3', 'tcp', 'nolock']
    }
  end

  config.vm.synced_folder "./", "/srv/flindt", type: "nfs"

  config.vm.network "private_network", ip: "192.168.33.10"
  config.vm.hostname = 'cust.flindt.org'
  # Add 'api.cust.flindt.org' to /etc/hosts manually !!
  config.vm.network "forwarded_port", guest: 443, host: 443
  config.vm.network "forwarded_port", guest: 80, host: 80

  config.vm.provider "virtualbox" do |vb|
    vb.memory = "2048"
  end

end

nginx vhosts

The frontend-server.

File cust.flindt.org:

server {

  listen 80 default;
  server_name cust.flindt.org;

  root /srv/flindt/upload;

# Wide-open CORS config for nginx
# https://enable-cors.org/server_nginx.html
location / {
     if ($request_method = 'OPTIONS') {
        add_header 'Access-Control-Allow-Origin' '*';
        add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
        #
        # Custom headers and headers various browsers *should* be OK with but aren't
        #
        add_header 'Access-Control-Allow-Headers' 'DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Content-Range,Range';
        #
        # Tell client that this pre-flight info is valid for 20 days
        #
        add_header 'Access-Control-Max-Age' 1728000;
        add_header 'Content-Type' 'text/plain charset=UTF-8';
        add_header 'Content-Length' 0;
        return 204;
     }
     if ($request_method = 'POST') {
        add_header 'Access-Control-Allow-Origin' '*';
        add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
        add_header 'Access-Control-Allow-Headers' 'DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Content-Range,Range';
        add_header 'Access-Control-Expose-Headers' 'DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Content-Range,Range';
     }
     if ($request_method = 'GET') {
        add_header 'Access-Control-Allow-Origin' '*';
        add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
        add_header 'Access-Control-Allow-Headers' 'DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Content-Range,Range';
        add_header 'Access-Control-Expose-Headers' 'DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Content-Range,Range';
     }
}

}

The backend-server, which serves the API.

File api.cust.flindt.org:

server {

  listen 80;
  server_name api.cust.flindt.org;

  location /static/ {
    alias /srv/flindt/backend/flindt/static/;
  }

  location / {
    #if ($http_x_forwarded_proto != "https") {
    #  rewrite ^(.*)$ https://$http_host$1 permanent;
    #}

    proxy_pass http://127.0.0.1:8005;
  }

}

uWSGI

File flindt_backend.ini:

[uwsgi]
virtualenv = /srv/virtualenvs/flindt
chdir = /srv/flindt/backend
wsgi-file = /srv/flindt/backend/flindt/wsgi.py

module = flindt.wsgi:application
env DJANGO_SETTINGS_MODULE = flindt.settings
master = true
pidfile = /tmp/project-master.pid
http = 0.0.0.0:8005
processes = 5
harakiri = 1000
max-requests = 5000

TODO

  • HTTPS / certificates for CUSTOMER.flindt.org and api.CUSTOMER.flindt.org, preferably with Let’s Encrypt and Caddyserver.