diff --git a/README.md b/README.md index 660fe1f4f0..6c2cc38101 100644 --- a/README.md +++ b/README.md @@ -1,249 +1,122 @@ # Polis -Polis is an AI powered sentiment gathering platform. More organic than surveys and less effort than focus groups. +> :warning: We are migrating to [New Implementation](https://github.com/MaanasArora/convergent/) -> [Chorus](https://github.com/choruslabs/chorus) and archiving this repo! -For a detailed methods paper, see [Polis: Scaling Deliberation by Mapping High Dimensional Opinion Spaces][methods-paper]. - - [methods-paper]: https://www.e-revistes.uji.es/index.php/recerca/article/view/5516/6558 +Polis is an AI-powered sentiment gathering platform. More organic than surveys and less effort than focus groups. [![Docker Image Builds](https://github.com/CivicTechTO/polis/workflows/Docker%20image%20builds/badge.svg)][docker-image-builds] [![E2E Tests](https://github.com/CivicTechTO/polis/workflows/E2E%20Tests/badge.svg)][e2e-tests] - [docker-image-builds]: https://hub.docker.com/u/newjerseystyle [e2e-tests]: https://github.com/CivicTechTO/polis/actions?query=workflow%3A%22E2E+Tests%22 ---- - -## 🎈 đŸĒ Start here! đŸĒ 🎈 - -If you're interested in using or contributing to Polis, please see the following: - -- [📚 **knowledge base**][knowledge-base]: for a comprehensive wiki to help you understand and use the system -- [🌐 **main deployment**](https://pol.is): the main deployment of Polis is at , and is - free to use for nonprofits and government -- [đŸ’Ŧ **discussions**][discussions]: for questions (QA) and discussion -- [✔ī¸ **issues**][issues]: for well-defined technical issues -- [🏗ī¸ **project board**][board]: somewhat incomplete, but still useful; We stopped around the time that Projects Beta came out, and we have a [Projects Beta Board][beta-board] that we'll eventually be migrating to -- [✉ī¸ reach out][hello]: if you are applying Polis in a high-impact context, and need more help than you're able to get through the public channels above - - [knowledge-base]: https://compdemocracy.org/Welcome - [issues]: https://github.com/compdemocracy/polis/issues - [board]: https://github.com/compdemocracy/polis/projects/1 - [beta-board]: https://github.com/compdemocracy/polis/projects/1 - [discussions]: https://github.com/compdemocracy/polis/discussions - [hello]: mailto:hello@compdemocracy.org - -If you're trying to set up a Polis deployment or development environment, then please read the rest of this document 👇 âŦ‡ī¸ 👇 - ---- - -## ⚡ Running Polis - -Polis comes with Docker infrastructure for running a complete system, whether for a [production deployment](#-production-deployment) or a [development environment](#-development-tooling) (details for each can be found in later sections of this document). -As a consequence, the only prerequisite to running Polis is that you install a recent `docker` (and Docker Desktop if you are on Mac or Windows). -If you aren't able to use Docker for some reason, the various Dockerfiles found in subdirectories (`math`, `server`, `*-client`) of this repository _can_ be used as a reference for how you'd set up a system manually. -If you're interested in doing the legwork to support alternative infrastructure, please [let us know in an issue](https://github.com/compdemocracy.org/issues). +## What is Polis? -### Quick Start +Polis is a platform for gathering, analyzing, and understanding sentiment at scale. It uses AI to help facilitate large-scale, open-ended discussions. -```sh -cp example.env .env -make start -``` +* **For a detailed methods paper**, see [Polis: Scaling Deliberation by Mapping High Dimensional Opinion Spaces][methods-paper]. -That should run docker compose with the development overlay (see below) and default configuration values. + [methods-paper]: https://www.e-revistes.uji.es/index.php/recerca/article/view/5516/6558 -Visit `localhost:80/createuser` and get started. +## Quick Links đŸĒ 🎈 -### Docker & Docker Compose +* **Main Deployment:** [https://pol.is](https://pol.is) (Free for nonprofits and governments) +* **CivicTech Deployment (This Fork):** [https://civictech-polis.azurewebsites.net/](https://civictech-polis.azurewebsites.net/) +* **Knowledge Base (Wiki):** [https://compdemocracy.org/Welcome](https://compdemocracy.org/Welcome) (Comprehensive user guide) +* **Discussions (Q&A):** [https://github.com/compdemocracy/polis/discussions](https://github.com/compdemocracy/polis/discussions) +* **Technical Issues:** [https://github.com/compdemocracy/polis/issues](https://github.com/compdemocracy/polis/issues) +* **High-Impact Support:** (For significant deployments requiring more support) +* **Community Support:** [https://github.com/CivicTechTO/polis/issues](https://github.com/CivicTechTO/polis/issues) +* **Talk to a person / Polis 101** : [https://civictech.ca](https://civictech.ca) (Join CivicTech Hacknight - Ask for a Polis 101) +* **CivicTech Slack:** [https://civictech.ca](https://civictech.ca) (Join the `#polis` channel) -Newer versions of `docker` have `docker compose` built in as a subcommand. -If you are using an older version (and don't want to upgrade), you'll need to separately install `docker-compose`, and use that instead in the instructions that follow. -Note however that the newer `docker compose` command is required to [take advantage of Docker Swarm](/docs/scaling#docker-compose-over-docker-swarm) as a scaling option. -Many convenient commands are found in the Makefile. Run `make help` for a list of available commands. +## Getting Started (For Developers & Deployers) -### Building and running the containers +This section is for those who want to run their own instance of the CivicTech fork of Polis or contribute to development. -First clone the repository, then navigate via command line to the root directory and run the following command to build and run the docker containers. +Polis comes with Docker infrastructure for running a complete system, whether for a [production deployment](#docs/deployment.md#-production-deployment) or a [development environment](#-development-mode). -Copy the example.env file and modify as needed (although it should just work as is for development and testing purposes). +### Quick Start (Development) +Open terminal and enter: ```sh cp example.env .env -``` - - -```sh -docker compose --profile postgres up --build -``` - -If you get a permission error, try running this command with `sudo`. -If this fixes the problem, sudo will be necessary for all other commands as well. -To avoid having to use `sudo` in the future (on a Linux or Windows machine with WSL), [you can follow setup instructions here.](https://docs.docker.com/engine/install/linux-postinstall/) - -Once you've built the docker images, you can run without `--build`, which may be faster. Run - -```sh -docker compose --profile postgres up -``` - -or simply - -```sh make start ``` -Any time you want to _rebuild_ the images, just reaffix `--build` when you run. Another way to -easily rebuild and start your containers is with `make start-rebuild`. - -If you have only changed configuration values in .env, you can recreate your containers without -fully rebuilding them with `--force-recreate`. For example: - -```sh -docker compose --profile postgres down -docker compose --profile postgres up --force-recreate -``` - -To see what the environment of your containers is going to look like, run: - -```sh -docker compose --profile postgres convert -``` - -#### Using a local or remote (non-docker) database - -Omit the `--profile postgres` flag to use a local or remote database. You will need to set the `DATABASE_URL` environment variable in your `.env` file to point to your database. - -When using `make` commands, setting POSTGRES_DOCKER to `true` or `false` will determine whether to automatically include `--profile postgres` when it calls out to `docker compose`. - -#### Production Mode Shortcuts - -The commands in the Makefile can be prefaced with PROD. If so, the "dev overlay" configuration in `docker-compose.dev.yml` will be ignored. -Ports from services other than the HTTP proxy (80/443) will not be exposed. Containers will not mount local directories, watch for changes, -or rebuild themselves. In theory this should be one way to run Polis in a production environment. - -You need a `prod.env` file: - -`cp example.env prod.env` (and update accordingly). - -Then you can run things like: - -```sh -make PROD start - -make PROD start-rebuild -``` - -### Testing out your instance - -You can now test your setup by visiting `http://localhost:80/home`. - -Once the index page loads, you can create an account using the `/createuser` path. -You'll be logged in right away; email validation is not required. - -When you're done working, you can end the process using `Ctrl+C`, or typing `docker compose --profile postgres down` -if you are running in "detached mode". +That should run docker compose with the development overlay (see below) and default configuration values. -### Updating the system +Visit 🚀 -If you want to update the system, you may need to handle the following: +### Detailed Instructions -- [âŦ†ī¸ Run database migrations](docs/migrations.md), if there are new such -- Update docker images by running with `--build` if there have been changes to the Dockerfiles - - consider using `--no-cache` if you'd like to rebuild from scratch, but note that this will take much longer +* **Running Polis:** See [Running Polis](#running-polis) below for more details. +* **Configuration:** See [`docs/configuration.md`](docs/configuration.md) for deployment configuration. +* **Deployment:** See [`docs/deployment.md`](docs/deployment.md) for deployment bash commands. +* **Contributing:** See [CONTRIBUTING.md](https://github.com/compdemocracy/polis/blob/edge/CONTRIBUTING.md) for how to contribute upstream polis main repository. +* **Contributing:** See [our wiki](https://github.com/CivicTechTO/polis/wiki#if-you) for how to contribute to this fork. ---- +## Running Polis -## 🚀 Production deployment +Polis uses Docker and Docker Compose for easy setup. -While the commands above will get a functional Polis system up and running, additional steps must be taken to properly configure, secure and scale the system. -In particular +### Prerequisites -- [⚙ī¸ Configure the system](docs/configuration.md), esp: - - the domain name you'll be serving from - - enable and add API keys for 3rd party services (e.g. automatic comment translation, spam filtering, etc) -- [🔏 Set up SSL/HTTPS](docs/ssl.md), to keep the site secure -- [📈 Scale](docs/scaling.md) for large or many concurrent conversations +* Docker and Docker Compose installed. [Instructions for avoiding `sudo` with Docker](https://docs.docker.com/engine/install/linux-postinstall/). -### Support +### Basic Commands -We encourage you to take advantage of the public channels above for support setting up a deployment. -However, if you are deploying in a high impact context and need help, please [reach out to us][hello] +* **Start (Development):** `make start` (Equivalent to `docker compose --profile postgres up`) +* **Start (Rebuild):** `make start-rebuild` (Rebuilds Docker images) +* **Start (Production - see docs/deployment.md for complete instructions):** `make PROD start` (Requires `prod.env`) +* **Stop:** `make stop` (Equivalent to `docker compose --profile postgres down`) +* **Help:** `make help` (Lists available commands) +* **View Configuration:** `docker compose --profile postgres convert` +* **Full Rebuild (Use with caution - wipes database):** `make start-FULL-REBUILD` ---- +### Using a local or remote (non-docker) database +See [Using a local or remote (non-docker) database](#using-a-local-or-remote-non-docker-database) below -## đŸ’ģ Development tooling +### đŸ’ģ Development Mode -Once you've gotten [Polis running (as described above)](#-running-polis), you can enable developer conveniences by running +For development, use the following command to enable live code reloading, debugging, and other conveniences: ```bash -docker compose -f docker-compose.yml -f docker-compose.dev.yml --profile postgres up +docker compose -f docker-compose.yml -f docker-compose.dev.yml --profile postgres up --build ``` -(run with `--build` if this is your first time running, or if you need to rebuild containers) - -This enables: - +**This enables:** - Live code reloading and static type checking of the server code - A nREPL connection port open for connecting to the running math process - Ports open for connecting directly to the database container -- Live code reloading for the client repos (in process) -- etc. - -This command takes advantage of the `docker-compose.dev.yml` _overlay_ file, which layers the developer conveniences describe above into the base system, as described in the `docker-compose.yml` file. -You can specify these `-f docker-compose.yml -f docker-compose.dev.yml` arguments for any `docker` command which you need to take advantage of these features (not just `docker compose --profile postgres up`). - -You can create your own `docker-compose.x.yml` file as an overlay and add or modify any values you need to differ -from the defaults found in the `docker-compose.yml` file and pass it as the second argument to the `docker compose -f` command above. - -### Testing - -We use Cypress for automated, end-to-end browser testing for PRs on GitHub (see badge above). -Please see [`e2e/README.md`](/e2e/README.md) for more information on running these tests locally. - -### Miscellaneous & troubleshooting - -#### Docker Problems -A lot of issues might be resolved by killing all docker containers and/or restarting docker entirely. If that doesn't -work, this will wipe all of your polis containers and volumes (**INCLUDING THE DATABASE VOLUME, so don't use this in prod!**) and completely rebuild them: +**About docker compose overlay file:** +* `docker-compose.dev.yml` in above command is an _overlay_ file, which layers the developer conveniences describe above into the base system, as described in the `docker-compose.yml` file. +* You can create your own `docker-compose.x.yml` file as an overlay and add or modify any values you need to differ from the defaults found in the `docker-compose.yml` file and pass it as the second argument to the `docker compose -f` command above. +* You can specify these `-f docker-compose.yml -f docker-compose.dev.yml` arguments for any `docker` command which you need to take advantage of these features (not just `docker compose --profile postgres up`). -`make start-FULL-REBUILD` +## Using a local or remote (non-docker) database -see also `make help` for additional commands that might be useful. - -#### Git Configuration - -Due to past file re-organizations, you may find the following git configuration helpful for looking at history: - -```sh -git config --local include.path ../.gitconfig -``` - -#### Running as a background process +Omit the `--profile postgres` flag to use a local or remote database. You will need to set the `DATABASE_URL` environment variable in your `.env` file to point to your database. -If you would like to run docker compose as a background process, run the `up` commands with the `--detach` flag, and use `docker compose --profile postgres down` to stop. +When using `make` commands, setting POSTGRES_DOCKER to `true` or `false` will determine whether to automatically include `--profile postgres` when it calls out to `docker compose`. -#### Using Docker Machine as your development environment -If your development machine is having trouble handling all of the docker containers, look into [using Docker Machine](/docs/docker-machine.md). +## Testing -#### Resolving problems with npm not finding libraries +* **End-to-End (E2E) Tests:** See [`e2e/README.md`](e2e/README.md) for instructions on running automated browser tests. -Sometimes npm/docker get in a weird state, especially with native libs, and fail to recover gracefully. -You may get a message like `Error: Cannot find module .... bcrypt`. -If this happens to you, try -[following the instructions here.](https://github.com/compdemocracy/polis/issues/1391) +## Troubleshooting -#### Issues with Apple Silicon (M1 & M2) chips +* **Docker Issues:** Try restarting Docker, killing all containers, or using `make start-FULL-REBUILD` (WARNING: This wipes the database). +* **NPM Issues:** See [this issue](https://github.com/compdemocracy/polis/issues/1391) for potential solutions. +* **Apple Silicon (M1/M2) Issues:** You may need to use a [Rosetta terminal](https://support.apple.com/en-us/HT211861) for some dependencies. -You may find it necessary to install some dependencies, namely nodejs and postgres stuff, in a [Rosetta terminal](https://support.apple.com/en-us/HT211861). Create an issue or reach out if you are having strange build issues on Apple computers. -## Šī¸ License +## License [AGPLv3 with additional permission under section 7](/LICENSE) diff --git a/docs/deployment.md b/docs/deployment.md new file mode 100644 index 0000000000..7326ee5a74 --- /dev/null +++ b/docs/deployment.md @@ -0,0 +1,97 @@ +# Polis Deployment Guide + +## 🚀 Production deployment + +While the commands above will get a functional Polis system up and running, additional steps must be taken to properly configure, secure and scale the system. +In particular + +- [⚙ī¸ Configure the system](configuration.md), esp: + - the domain name you'll be serving from + - enable and add API keys for 3rd party services (e.g. automatic comment translation, spam filtering, etc) +- [🔏 Set up SSL/HTTPS](ssl.md), to keep the site secure +- [📈 Scale](scaling.md) for large or many concurrent conversations + +### Updating the system + +If you want to update the system, you may need to handle the following: + +- [âŦ†ī¸ Run database migrations](migrations.md), if there are new such +- Update docker images by running with `--build` if there have been changes to the Dockerfiles + - consider using `--no-cache` if you'd like to rebuild from scratch, but note that this will take much longer + +### Backup the system +**Manual Backup Script (Example - for Docker Postgres):** +```bash +#!/bin/bash +docker exec -t pg_dump -U -d > polis_backup_$(date +%Y-%m-%d).sql +# (Optional) Upload the backup to Azure Blob Storage or another storage service. +``` + +**Restore (Example - for Docker Postgres):** +```bash +docker exec -i psql -U -d < polis_backup_YYYY-MM-DD.sql +``` +**Important**: Replace placeholders like ``, ``, and `` with your actual values. Test your backup and restore procedures regularly! + +--- + +## Support + +We encourage you to take advantage of the public channels above for support setting up a deployment. +However, if you are deploying in a high impact context and need help, please [reach out to compdemocracy](mailto:hello@compdemocracy.org) + +--- + +# CivicTech Fork deployment + +This document provides instructions for deploying the CivicTech fork of Polis to a production environment. This guide specifically details the deployment process used for [https://civictech-polis.azurewebsites.net/](https://civictech-polis.azurewebsites.net/), which utilizes Azure App Service. + +## 1. Server Setup (Azure App Service) + +The CivicTech deployment of Polis uses Azure App Service for hosting. Here's [a step-by-step guide to setting up the environment](https://medium.com/@jyotikhetan2/creating-an-azure-app-service-using-azure-cli-a-step-by-step-guide-e1e477ff7c19). + +1. **Create an Azure Account:** If you don't already have one, create an Azure account at [https://azure.microsoft.com/](https://azure.microsoft.com/). + +2. **Create an App Service Plan:** + * In the Azure portal, search for and select "App Service plans". + * Click "Create". + * Select your subscription, resource group (or create a new one), name, operating system (Linux), region, and pricing tier. The pricing tier should be chosen based on expected load (see [Scaling Considerations](#4-scaling-considerations)). For initial setup, a "Standard" tier is often sufficient. + * Click "Review + create", then "Create". + +3. **Configure Docker:** + * In the Azure App Service, `docker-compose.yml` must be smaller than 4000 bytes or Container App service should be used. + * I embedded the `prod.env` variables into my `docker-compose.yml` using `environment` section to ensure deployemnt. + +4. **Create a Web App:** + * In the Azure portal, open a cloud shell. You can also use portal (GUI) to do the same. + ```bash + wget + az webapp create --resource-group --plan --name civictech-polis --multicontainer-config-type compose --multicontainer-config-file docker-compose.yml + ``` + +## 2. `prod.env` and Production Mode Shortcuts + +The `prod.env` file contains environment variables required for production deployments. + +1. **Create `prod.env`:** Copy the `example.env` file: `cp example.env prod.env`. + +2. **Edit `prod.env`:** Modify the values in `prod.env` to match your production environment. Crucially: + * `NODE_ENV=production` + * `DOMAIN=your-custom-domain.com` (or `your-app-name.azurewebsites.net`) + * `SESSION_SECRET` (Generate a strong, random secret) + * `DATABASE_URL` (If not using the built-in Postgres container, set this to your external database connection string.) + * `GOOGLE_CLIENT_ID` and `GOOGLE_CLIENT_SECRET` (If using Google authentication) + * `TWITTER_CONSUMER_KEY` and `TWITTER_CONSUMER_SECRET` (If using Twitter authentication) + * Any other relevant API keys (translation, spam filtering, etc.) + +--- + +## Scaling Considerations (Azure) + +* **App Service Plan:** The App Service Plan you choose determines the resources available to your application. You can scale up (more powerful instances) or scale out (more instances) as needed. +* **Database:** If you're using the built-in Postgres container for development, consider using a managed Azure Database for PostgreSQL instance for production. This provides better performance, scalability, and manageability. + +## Backup and Restore Procedures + +* **Azure App Service Backups:** Azure App Service provides built-in backup functionality. You can configure automatic backups in the "Backups" section of your web app settings. +* **Database Backups:** If you're using Azure Database for PostgreSQL, backups are managed automatically. You can configure the retention period and perform point-in-time restores. If using the built-in docker Postgres, you'll need to implement a manual backup strategy (e.g., using `pg_dump` inside the container and storing the backups in Azure Blob Storage).