Welcome to the "Open source" category in PoeticMetric Analytics documentation. In this section, we will explore the technical architecture and principles that make PoeticMetric Analytics a robust and transparent web analytics tool.



At PoeticMetric, we believe in the power of open source software and its ability to foster collaboration, security, and privacy. We are committed to providing a free as in freedom solution that empowers website owners to analyze their data while maintaining control over their information.

PoeticMetric Analytics is a fully free and open source software and all the PoeticMetric's source code is published on https://github.com/th0th/poeticmetric with the [GNU Affero General Public License](https://www.gnu.org/licenses/agpl-3.0.en.html).

Introduction

Welcome to the docs for **PoeticMetric**, a free as in freedom, open source, privacy-first, regulation-compliant, and blazingly fast alternative to Google Analytics!



PoeticMetric is **privacy-first**, **transparent**, **efficient** and **sustainable**. If you would like to read more about PoeticMetric's vision, you can check out its manifesto: [Building a Better Web: PoeticMetric's Vision](/manifesto).

This documentation is here to help you make the best use of PoeticMetric. If you think there is something missing, you can always reach out to us via [e-mail](mailto:support@poeticmetric.com?subject=I%20can't%20find%20something%20on%20docs).

Foreword

Getting started

**PoeticMetric** allows you to track the performance of your website. It requires a javascript snippet to be added to the pages that will be tracked. This injected javascript snippets sends the required information to PoeticMetric's REST API when a visitor visits your website, and the collected data is processed and aggregated to generate the reports. 



You can track unlimited number of websites, however, the number of page views (or events) you can use changes according to your subscription plan. See [pricing](/pricing) for details.

To start tracking your website's traffic, you need to update the HTML code of your website and add the PoeticMetric tracking script code to the header section. Put the script between the `<head>` and `</head>` tags.



<Alert variant="primary">
    If you are not the person managing the website's code, and there is someone else who deals with it, you can share the link to this page.
</Alert>

## PoeticMetric tracking script

The PoeticMetric tracking script code is this:

```html
<script async src="https://www.poeticmetric.com/pm.js"></script>
```

Copy this code, and add it to the `<head>...</head>` section of the every page you want to track.

Adding the script to your website

**Site** has these settings:

* [Domain](#domain)
* [Name](#name)
* [Safe query parameters](#safe-query-parameters)
* [Public reports](#public-reports)



![site settings](/docs-files/websites/site-settings/site-settings.png "Site settings")

## Domain

**Domain** (or hostname) of the website. This is the most important setting, because **it needs to match the domain of the website that has the [PoeticMetric tracking script](/docs/websites/adding-the-script-to-your-website)**. It can be a [domain](https://en.wikipedia.org/wiki/Domain_name) or a [subdomain](https://en.wikipedia.org/wiki/Subdomain), but it should include protocol (e.g. `https` or `http`), or path information (e.g. `/about`).

## Name

**Name** is the displayed name or title of the site. It is used only for displaying purposes. When there are multiple sites, sites are listed alphabetically on the sites page.

## Safe query parameters

PoeticMetric doesn't store any query parameters since they might include personal data or sensitive information. However, you can add query parameter names that doesn't include any such data, and you want to track. For more details about the query parameters please see [Query parameters](/docs/websites/query-parameters) article.

## Public reports

By default, reports for the website are accessible only to the people with a team member account within your PoeticMetric organization. If you enable this option, reports for the site will be publicly accessible on `https://www.poeticmetric.com/s?d=www.yoursite.com`. This setting come into effect immediately, even if you enabled it earlier, the moment you uncheck this option and click **Save**, public access to website's reports will be disabled.

For example, you can visit PoeticMetric's own reports on [https://www.poeticmetric.com/s?d=www.poeticmetric.com](https://www.poeticmetric.com/s?d=www.poeticmetric.com).

Site settings

**Query parameters** (also known as "query string" or "URL parameters") are the part of the **URL** (uniform resource locator) that comes after the question mark **(?)**.



Query parameters consist of a key and a value, separated by an equal sign (=). And if there are multiple parameters, each is separated by an ampersand (&).

![query parameters in the address bar](/docs-files/websites/query-parameters/address-bar.jpg "Query parameters in the address bar")

According to [RFC 3986](https://www.rfc-editor.org/rfc/rfc3986#section-3.4);

> the query component contains non-hierarchical data that, along with data in the path component, serves to identify a resource within the scope of the URI's scheme and naming authority (if any)

## Query parameters on PoeticMetric

Query parameters often include personal or sensitive information. For example, consider a form that autofills the e-mail address field from query parameters. The full URL would look like something like this

`https://www.yoursite.com/?email=gordon@blackmesa.tld`

And sometimes, one page has multiple kinds of content, and the query parameters are used for identifying what to display.

`https://www.yoursite.com/?tab=profile`

Since there is no easy way to differentiate personal data containing query parameters from those not, by default, we strip all the query parameters, except UTM-related ones. However, you can set the safe query parameters in your site's settings, and PoeticMetric will collect those as a part of the page's URL.

### Setting safe query parameters

To set safe query parameters for your website on PoeticMetric:

1. Go to **Sites**, and click the **Edit** button below the website you would like to set safe query parameters for.

   ![edit site](/docs-files/websites/query-parameters/edit-site.png "Edit site")

2. Under the **Safe query parameters** section, type the query parameter you want to add and press enter.

   ![set query parameters in PoeticMetric site settings](/docs-files/websites/query-parameters/update-safe-query-parameters.png "Set safe query parameters in PoeticMetric site settings")

3. Repeat the process for each query parameter you want to add.

4. Click on the **Save** button.

Query parameters

How much time a visitor spends on a page is one of the critical engagement metrics of website analytics. However, it is tricky to measure in an accurate and privacy-friendly manner.



## Google Analytics (and most of the other tools)

**Google Analytics** measures time spent on the page like this: Visitor lands on your website at a particular time. When this visitor navigates to another page on your site, Google Analytics considers all the time between these two events as the "time on page". ** This approach requires tracking the visitor by a unique identifier, IP address, or a cookie.** In addition to that, the evaluation by this method is inaccurate and inadequate, too. Consider these cases:

### Case#1: Visitor views a single page

If the visitor views a single page on your site and then leaves the site closing the tab, the duration is not measured at all. In this case, the duration data is totally lost.

### Case#2: Visitor switches tabs on their browser

If the visitor lands on a page on your website, switches tab on their browser for some time, and then returns to the tab that your site is on, the actual duration shouldn't include the time they were on the other tab, right?

## PoeticMetric

PoeticMetric measures the page view duration as accurately as possible.

* The duration data is saved just before the visitor leaves the page (or the tab). So, even if the visitor doesn't navigate to a second page on your site, the duration is safely measured and saved.
* Also, if the visitor has your site open on a browser tab, the time they spend viewing another site on another tab is properly deducted and not considered as the page view duration.

### Sacrificed data point: Session duration

As stated in the previous section, tracking a visitor between pages (i.e. along a session) requires attaching a certain identifier to that specific visitor. [**This is what we definitely don't do at PoeticMetric.**](/manifesto#1-being-privacy-first) So, we had to sacrifice the **session duration** data point in order to stay privacy-focused and compliant with the personal data privacy regulations.

Page view durations

By default, once you add PoeticMetric script on your website, PoeticMetric tracks every visitor. But you might want to exclude yourself and your team from affecting the statistics.



You might be familiar with this process and look for a way to ignore some IP addresses on the PoeticMetric interface. However, since we don't collect, encrypt or process IP addresses as a privacy feature, IP address filtering is not available on PoeticMetric. Instead, you will need to set a `localStorage` flag to let the tracker script know, and it will exclude you and stop event collection.

## tl;dr, to exclude yourself:

1. Go to your website, which you have the PoeticMetric script installed.

2. Open your browser's javascript console.

    * For **Google Chrome**, you can select **View > Developer > JavaScript Console** ([screenshot](/docs-files/websites/excluding-yourself-from-analytics/google-chrome.png)). Or press `Command + Option + J` on **MacOS**, `Control + Shift + J` on **Windows, Linux and Chrome OS**.

    * For **Safari**, first, you [need to enable **Developer tools**](https://support.apple.com/guide/safari/use-the-developer-tools-in-the-develop-menu-sfri20948/mac). Then you can select **Develop > Show JavaScript Console** ([screenshot](/docs-files/websites/excluding-yourself-from-analytics/safari.png)). Or press `Command + Option + C`.

    * For **Mozilla Firefox**, you can select **Menu > More tools > Browser console** ([screenshot](/docs-files/websites/excluding-yourself-from-analytics/mozilla-firefox.png)). Or press `Command + Shift + J` on **MacOS**, `Control + Shift + J` on **Windows**.

3. Then paste this and press enter:

   ```javascript
   window.localStorage.pmIgnore=1
   ```

Well done! From now on, as soon as you have this flag in your `localStorage` your events from this browser will be ignored. If you use multiple browsers, don't forget to do this in each one.

Excluding yourself from analytics

Google Search Console provides some Google related statistics about your website, like what people, who see your website in the search results, use as the search term, and how many of them actually click your site on the results.



HTTP (The Hypertext Transfer Protocol) has a specific way of handling the passing of the referrer information on [RFC 9110 - HTTP Semantics](https://httpwg.org/specs/rfc9110.html#field.referer). According to this specification, when a link is clicked, URL of the page which has the link is passed via the [`Referer`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Referer) header to the new page. However, since this behavior is [considered to cause undesirable consequences for user security and privacy](https://developer.mozilla.org/en-US/docs/Web/Security/Referer_header:_privacy_and_security_concerns), many sites including Google started handling referrers differently. That's why PoeticMetric can't see which search term your visitor used on Google, and need to use Google Search Console integration to obtain this data from Google.

![google search terms report details](/docs-files/websites/google-search-console-integration/google-search-terms-report-details.png "Google search terms report details")

With this integration, PoeticMetric enables you to see

* the **search terms** that list your website as a result,
* **your website's position** for search terms,
* **CTR** (click-through rate),
* number of **impressions** and
* and number of **clicks**.

## Adding your site to Google Search Console

To get your website's search statistics from Google, you need to add your site to Google Search Console. To do so, go to https://search.google.com/search-console and follow the steps. If you need help at some point, you can consult to [Google's guide](https://support.google.com/webmasters/answer/34592).

![google search console - add property](/docs-files/websites/google-search-console-integration/google-search-console-add-property.png "Google Search Console - Add property")

## Enable the integration for your site on PoeticMetric

To enable the Google Search Console Integration for your website on PoeticMetric:

1. Go to **Sites**, and click the **Edit** button below the website you would like to add the integration for.

   ![edit site](/docs-files/websites/google-search-console-integration/edit-site.png "Edit site")

2. Enable the **Show Google Search Console reports** option.

   ![enable the show google search console reports option](/docs-files/websites/google-search-console-integration/show-google-search-console-reports.png "Enable the Show Google Search Console reports option.")

3. Click the button that says **Connect with Google**, and follow the steps. This will allow PoeticMetric to access the Google Search Console data.

4. Once connecting with Google is completed, select the property for this site on the dropdown.

   ![select the google search console option](/docs-files/websites/google-search-console-integration/select-the-google-search-console-option.png "Select the Google Search Console option")

5. And then click the **Save** button.

   ![save](/docs-files/websites/google-search-console-integration/save.png "Save")

After you enable the integration, a new report will be visible in the **Sources** part. You can select the **Google search terms** reports from the dropdown there.

![google search terms report](/docs-files/websites/google-search-console-integration/google-search-terms-report.png "Google search terms report")

Google Search Console integration

Websites

**PoeticMetric** offers multiple team member accounts for the same organization account to ease collaboration with your colleagues.



## Inviting new members

You can invite your team by going to the [Team](/team) page, and clicking the **Invite new team member** button on top right. When you complete the form, an e-mail with a link will be sent to your teammate.

Until they accept the invite clicking the link included in the e-mail, their account will be listed on the team page with an **"invited by not activated yet"** indicator.

<img alt="invited-team-member" src="/docs-files/team/introduction/invited-team-member.png" style="max-width: 400px" />

<Alert variant="primary">
    You can read more about team member permissions on <a href="/docs/team/permissions">Permissions</a>.
</Alert>

## Permissions

The first team member, who signed up is set as the **organization owner**. Organization owner is indicated with a small lightning icon on their avatar in the team members list.



<img alt="owner" src="/docs-files/team/permissions/owner.png" style="max-width: 400px" />

The organization owner is the only team member who can

* invite new team members,
* access to the subscription and billing details,
* change the organization details in the settings page.

Other than these, there are no difference in terms of permissions between an owner and non-owner team member.

Permissions

There are profile pictures displayed next to team members' name in various places. PoeticMetric uses [Gravatar](https://gravatar.com) for fetching people's avatars.



[Gravatar](https://gravatar.com) (**G**lobally **R**ecognized **Avatar**) is a free service provided by [Automattic](https://automattic.com) (the company behind the [WordPress](https://wordpress.org)).

So, if you want to use a custom profile picture on PoeticMetric, you can go to [Gravatar](https://gravatar.com), and set an avatar with the e-mail address you use on PoeticMetric there. Your profile picture on PoeticMetric will automatically be updated.

Avatars

Team

In this article, we will explore the technical software architecture that drives PoeticMetric Analytics, highlighting the core principles of transparency and openness that are at the heart of our platform. Built upon open source technologies, PoeticMetric Analytics is designed to provide website owners with a comprehensive and privacy-first web analytics solution. By understanding the underlying technical architecture, you gain insights into how your data is collected, processed, and stored, ensuring full transparency and control over your analytics experience. Let's dive into the details of PoeticMetric's software architecture and discover how open source empowers you to make informed decisions about your data.



## An overview of PoeticMetric's topology

<a href="/docs-files/open-source/technical-architecture/topology.png" target="_blank">
    <figure>
        <img alt="Software architecture (topology)" src="/docs-files/open-source/technical-architecture/topology.png">
        <figcaption>
            PoeticMetric's architecture
        </figcaption>
    </figure>
</a>

## Components

### Data storage

#### PostgreSQL

PoeticMetric relies on [PostgreSQL](https://www.postgresql.org) as its primary, source-of-truth database, serving as a secure repository for essential data such as account details, website information, and team member records. PostgreSQL, a widely recognized and battle-tested open source database, plays a pivotal role in ensuring the reliability and robustness of PoeticMetric's data storage. With its powerful features and established reputation, PostgreSQL forms a solid foundation for the flawless functioning of PoeticMetric's analytics platform.

#### ClickHouse

PoeticMetric leverages the power of [ClickHouse](https://clickhouse.com), an open source and high-performance OLAP database management system, specifically designed to meet the demanding needs of analytics. As the storage and querying solution for analytics data, ClickHouse empowers PoeticMetric to deliver lightning-fast insights. With its exceptional speed, ClickHouse facilitates millisecond-level reporting, allowing users to obtain near-instantaneous results. By harnessing the capabilities of ClickHouse, PoeticMetric ensures that analytics data is efficiently stored and readily available, enabling swift and accurate analysis for a smooth user experience.

#### RabbitMQ

While PoeticMetric is primarily developed as a monolithic application, it incorporates multiple components resembling a microservices infrastructure, enabling seamless communication between these parts. This crucial inter-component communication is facilitated by [RabbitMQ](https://www.rabbitmq.com), a fundamental element of PoeticMetric's architecture. By serving as a reliable message broker, RabbitMQ ensures that the various components of PoeticMetric work harmoniously together. This integration of RabbitMQ within PoeticMetric's framework plays a vital role in maintaining the system's overall efficiency and effectiveness.

#### Redis

PoeticMetric leverages the power of [Redis](https://redis.io), a high-performance key-value storage system, to store transient data such as the daily changing visitor encryption salt. While this ephemeral information could be stored in any database, Redis offers the advantage of simplified expiry management. By utilizing Redis, PoeticMetric can effortlessly handle the expiration of this data as required. Redis has garnered acclaim within the open source community for its exceptional performance in key-value storage scenarios, further solidifying its suitability for PoeticMetric's needs.` 

### Backend

PoeticMetric's backend is a robust Golang application comprising three distinct components:

#### REST API

The REST API serves as the vital bridge between the frontend and the business logic in PoeticMetric, ensuring smooth connectivity by consuming data from PostgreSQL and ClickHouse databases.

It primarily:

* Connects the frontend to the business logic, facilitating smooth connectivity.
* Consumes data from PostgreSQL and ClickHouse databases.
* Enables frontend interactions with the PostgreSQL database for account-related tasks like sign-up, sign-in, and password recovery.
* Processes data collected by the tracker on customer websites and inserts it into the ClickHouse database for analysis.
* Retrieves relevant data from both PostgreSQL and ClickHouse databases to generate comprehensive site reports.
* Provides API access for third-party external applications, promoting integration and extensibility with external systems.

#### Worker

The worker component in PoeticMetric operates as a background task executor, ensuring asynchronous processing. It consumes messages from RabbitMQ queues and efficiently writes the resulting outcomes to both PostgreSQL and ClickHouse databases.

By separating long-running operations from the REST API and delegating them to the worker as a distinct process, PoeticMetric enables uninterrupted request-response cycles within the API. This approach prevents blocking and allows the REST API to promptly handle incoming requests. Moreover, it facilitates the efficient handling of a large volume of tasks, as the worker can be scaled accordingly to meet the demands of the workload.

#### Scheduler

The scheduler, a compact component within PoeticMetric's backend, serves the purpose of adding scheduled task messages to the RabbitMQ queues for consumption by the worker. It focuses solely on efficiently queuing time-based tasks, ensuring they are appropriately dispatched for processing by the worker component.

### Frontend

PoeticMetric's frontend is an impressive Next.js React application developed in TypeScript. To optimize its performance, the application is not executed as a Node.js application; instead, it is exported as static HTML, CSS, and JS files during the build process. This approach enables the frontend to be lightweight and easily served by web servers such as NGINX or Apache. The official Docker image of PoeticMetric employs NGINX for this purpose. The frontend application incorporates several exceptional open source libraries, including, but not limited to:

* [Bootstrap](https://getbootstrap.com) and [React Bootstrap](https://react-bootstrap.github.io) for streamlined and responsive UI components.
* [Lodash](https://lodash.com) for efficient data manipulation and utility functions.
* [SWR](https://swr.vercel.app) for data fetching and caching, enhancing the performance of API requests.
* [visx](https://airbnb.io/visx/) for powerful and customizable data visualization components, enabling captivating and informative graphical representations.

Technical architecture

You can get your own PoeticMetric Analytics self-hosted website analytics instance up and running in minutes following this guide. It doesn't require any software development expertise, if you are a little tech-savvy you can do it. PoeticMetric is designed to be self-hosted with Docker, and ready-to-use Docker images are published publicly.



If you are curious, there is another documentation article that explains about the parts of PoeticMetric and how these parts communicate with each other. You can check the technical architecture documentation [here](/docs/open-source/technical-architecture).

## Requirements

You will need a server with Docker installed. PoeticMetric has Docker images prebuilt for architectures `linux/amd64` and `linux/arm64`, so make sure your servers CPU is either `x86-64` or `arm64`.

PoeticMetric can run successfully on memory as low as 2GB, however, we recommend allocating at least 4GB of RAM for a smoother experience. However, keep in mind that as the traffic of your website goes up, you might need a more powerful server with a more powerful CPU and more RAM.

<Alert variant="primary">
    If you don't have a server, you can check it out [Hetzner Cloud](https://hetzner.cloud/?ref=IOqhEJnmxLGq) (referral link), we run our servers on their infrastructure, and we can recommend! 
</Alert>

## Versions

When you follow this tutorial, you will be deploying the latest stable version of PoeticMetric. However, a new version of PoeticMetric will be released with new features from time to time. Don't forget to check the section [Updating to a new version](#updating) before you leave.

PoeticMetric Analytics uses [semantic versioning](https://semver.org): `MAJOR.MINOR.PATCH`

* Updates with a `PATCH` version change (e.g. from `v1.0.0` to `v1.0.1`) include small changes, and mostly, bug fixes.
* `MINOR` version changing updates (e.g. from `v1.0.8` to `v1.1.0`) include new features, in a backward compatible manner.
* `MAJOR` version updates include big and important changes, most probably backward incompatible.

Even tough we suggest backing up your data periodically, and especially before performing an update, if you are going for a major version change, please pay extra attention to data backups.

## Running

### Downloading

Start by downloading the required Docker Compose configuration. Download the `self-hosted.zip` file for the [latest release](https://github.com/th0th/poeticmetric/releases/latest), and unzip it:

```shell
$ curl -fsSL https://github.com/th0th/poeticmetric/releases/latest/download/self-hosted.tar.gz | tar -xz
$ cd poeticmetric
```

### Set environment variables

You will find an `.env.example` file that has all the required environment variables with their default values. First, make a copy of this file with the actual file name:

```shell
$ cp .env.example .env
```

You can leave most of them as they are, however, you will need to change some of them:

#### Internal passwords

These variables will be used for setting the password the relevant component on the first run:

* `CLICKHOUSE_PASSWORD`
* `POSTGRES_PASSWORD`
* `RABBITMQ_PASSWORD`
* `REDIS_PASSWORD`

So, pick a **different**, hard to guess password for each one, and make sure you make note of these. Normally, you won't use any of these passwords while using PoeticMetric. However, if you face any issue, you will need each of them. If you lose one of these passwords, your databases may become inaccessible, and you might lose data.

#### Google Search Console integration variables

`GOOGLE_CLIENT_ID` and `GOOGLE_CLIENT_SECRET` are required only for [Google Search Console integration](https://dev.poeticmetric.com/docs/websites/google-search-console-integration). If you are planning to use the integration, you need to obtain them from [Google](https://developers.google.com/webmaster-tools). If you are not planning to use, you can safely skip and leave them as empty. You can also update these variables later.

#### URLs

PoeticMetric has two components that are accessible by HTTP:

* Frontend (`FRONTEND_BASE_URL`)
* REST API (`REST_API_BASE_URL`)

By the default `docker-compose.yaml`, frontend uses the port `8000`, and REST API `8001`. If you are not going to use a domain name, or a reverse proxy, you should set these as:

`FRONTEND_BASE_URL=http://<your_server_ip_address>:8000`

and

`REST_API_BASE_URL=http://<your_server_ip_address>:8001`

If you are going to use a reverse proxy with a domain name, you need to update the variables `FRONTEND_BASE_URL` and `REST_API_BASE_URL` accordingly, and configure two different virtual servers on your reverse proxy. One for the frontend, pointing to the port `8000`; and another one for the REST API, pointing to the port `8001`.

#### SMTP variables

PoeticMetric requires [Simple Mail Transfer Protocol (SMTP)](https://en.wikipedia.org/wiki/Simple_Mail_Transfer_Protocol) for delivering e-mails that are used for notifications, password recovery etc.

Set `SMTP_FROM`, `SMTP_HOST`, `SMTP_PASSWORD`, `SMTP_PORT` and `SMTP_USER` variables according to your e-mail service provider.

### Running services

Once you are done setting up your environment variables, you can bring PoeticMetric's services up by:

```shell
$ docker compose up -d
```

If everything goes well, you should be able to access your self-hosted website analytics instance via the frontend URL you set in the [URLs](#urls) step. Go to `http://<your_frontend_base_url>/bootstrap` to initialize the database schemas, and create your account.

## Updating

When a fresh edition of PoeticMetric becomes available, simply follow these steps: temporarily halt PoeticMetric's services, retrieve the latest docker images, and restart the services. PoeticMetric takes care of updating the databases automatically whenever necessary.

```shell
$ docker compose down --remove-orphans
$ docker compose pull
$ docker compose up -d
```

## Getting help

If you have any issues setting up your PoeticMetric Analytics instance, you can ask for community members' help on [GitHub Discussions](https://github.com/th0th/poeticmetric/discussions/new?category=self-hosted-help).