Merge pull request #437 from duyet/chore/ui-update

feat: add support for excluding monitoring users and enhance query history documentation

Author: duyet

.env.example

@@ -9,4 +9,5 @@ CLICKHOUSE_PASSWORD=
 CLICKHOUSE_NAME=
 CLICKHOUSE_TZ=
 CLICKHOUSE_MAX_EXECUTION_TIME=60
-NEXT_QUERY_CACHE_TTL=86400
+NEXT_QUERY_CACHE_TTL=86400
+CLICKHOUSE_EXCLUDE_USER_DEFAULT=

README.md

@@ -21,6 +21,15 @@ Features:
 **Documentation**:
 
 - https://duyet.github.io/clickhouse-monitoring/
+  - [Getting Started](https://duyet.github.io/clickhouse-monitoring/getting-started)
+    - [Local Development](https://duyet.github.io/clickhouse-monitoring/getting-started/local)
+    - [User Role and Profile](https://duyet.github.io/clickhouse-monitoring/getting-started/clickhouse-requirements)
+    - [Enable System Tables](https://duyet.github.io/clickhouse-monitoring/getting-started/clickhouse-enable-system-tables)
+  - [Deployments](https://duyet.github.io/clickhouse-monitoring/deploy)
+    - [Vercel](https://duyet.github.io/clickhouse-monitoring/deploy/vercel)
+    - [Docker](https://duyet.github.io/clickhouse-monitoring/deploy/docker)
+    - [Kubernetes Helm Chart](https://duyet.github.io/clickhouse-monitoring/deploy/k8s)
+  - [Advanced](https://duyet.github.io/clickhouse-monitoring/advanced)
 
 **Screenshots**:
 
@@ -33,130 +42,6 @@ Features:
 ![](.github/screenshots/screenshot_7.png)
 ![](.github/screenshots/screenshot_8.png)
 
-## Getting Started
-
-To get the project up and running on your local machine, follow these steps:
-
-1. Clone the repository
-2. Install dependencies using `npm install` or `yarn install`
-3. Create a `.env.local` file by copying the `.env.example` file and filling in the required environment variables:
-
-   - `CLICKHOUSE_HOST`: ClickHouse host(s), for example `http://localhost:8123` or `http://ch-1:8123,http://ch-2:8123`
-   - `CLICKHOUSE_NAME`: (Optional) Name of ClickHouse instance, must match the number of hosts in `CLICKHOUSE_HOST`, for example `localhost` or `ch-1,ch-2`.
-   - `CLICKHOUSE_USER`: ClickHouse user with permission to query the `system` database.
-   - `CLICKHOUSE_PASSWORD`: ClickHouse password for the specified user.
-   - `CLICKHOUSE_MAX_EXECUTION_TIME`: [`max_execution_time`](https://clickhouse.com/docs/en/operations/settings/query-complexity#max-execution-time) timeout in seconds. Default is `10`.
-   - `CLICKHOUSE_TZ`: ClickHouse server timezone. Default: `''`.
-   - `NEXT_QUERY_CACHE_TTL`: TTL of [`unstable_cache`](https://nextjs.org/docs/app/api-reference/functions/unstable_cache) - cache the results of most charts to speed up and reuse them across multiple requests. Default: `86400`.
-   - `NEXT_PUBLIC_LOGO`: (Optional) HTTP path to logo image.
-   - `EVENTS_TABLE_NAME`: The table name for storing dashboard self-tracking events. Default: `system.monitoring_events`
-
-4. Run the development server with `npm run dev` or `yarn dev`
-5. Open [http://localhost:3000](http://localhost:3000) in your browser to see the dashboard.
-
-## ClickHouse Requirements
-
-### 1. Monitoring user role
-
-Suggested role for "monitoring" user must have these privileges on `system` database:
-
-```xml
-# File: users.d/monitoring_role.xml
-<clickhouse>
-  <users>
-    <monitoring>
-      <password><!-- define password here --></password>
-      <profile>monitoring_profile</profile>
-      <networks><ip>::/0</ip></networks>
-      <grants>
-        <query>GRANT monitoring_role</query>
-      </grants>
-    </monitoring>
-  </users>
-
-  <roles>
-    <monitoring_role>
-      <grants>
-        <query>REVOKE ALL ON *.*</query>
-        <query>GRANT SELECT,SHOW,dictGet,REMOTE ON *.*</query>
-        <query>GRANT SELECT,INSERT,ALTER,CREATE,DROP,TRUNCATE,OPTIMIZE,SHOW,dictGet ON system.*</query>
-        <query>GRANT CREATE TEMPORARY TABLE ON *.*</query>
-      </grants>
-    </monitoring_role>
-  </roles>
-</clickhouse>
-```
-
-`CREATE TEMPORARY TABLE` is needed because the UI using `FROM merge(system, '^query_log')` allows retrieving all the data from old tables that were renamed during the upgrade.
-
-### 2. Monitoring user profile
-
-```xml
-# File: users.d/monitoring_profile.xml
-<clickhouse>
-  <profiles>
-    <monitoring_profile>
-      <allow_experimental_analyzer>1</allow_experimental_analyzer>
-
-      <!-- Optional: query cache to avoid hit too much queries on database -->
-      <use_query_cache>1</use_query_cache>
-      <query_cache_ttl>50</query_cache_ttl>
-      <query_cache_max_entries>0</query_cache_max_entries>
-      <query_cache_system_table_handling>save</query_cache_system_table_handling>
-      <query_cache_nondeterministic_function_handling>save</query_cache_nondeterministic_function_handling>
-    </monitoring_profile>
-  </profiles>
-</clickhouse>
-```
-
-## Deployment
-
-### 1. Vercel
-
-For easy deployment, use the [Vercel Platform](https://vercel.com/new?utm_medium=default-template&filter=next.js&utm_source=create-next-app&utm_campaign=create-next-app-readme), created by the makers of Next.js. Refer to the [Next.js deployment documentation](https://nextjs.org/docs/deployment) for more details.
-
-### 2. Docker
-
-Using the latest image here: https://github.com/duyet/clickhouse-monitoring/pkgs/container/clickhouse-monitoring
-
-```bash
-docker run -it \
-    -e CLICKHOUSE_HOST='http://localhost' \
-    -e CLICKHOUSE_USER='default' \
-    -e CLICKHOUSE_PASSWORD='' \
-    -e CLICKHOUSE_TZ='Asia/Ho_Chi_Minh' \
-    -e CLICKHOUSE_MAX_EXECUTION_TIME='15' \
-    -e NEXT_QUERY_CACHE_TTL='86400' \
-    --name clickhouse-monitoring \
-    ghcr.io/duyet/clickhouse-monitoring:main
-```
-
-### 3. Kubernetes Helm Chart
-
-Using the latest helm chart here: https://github.com/duyet/charts/tree/master/clickhouse-monitoring
-
-```bash
-helm repo add duyet https://duyet.github.io/charts
-
-cat <<EOF >> values.yaml
-env:
-  - name: CLICKHOUSE_HOST
-    value: http://localhost:8123
-  - name: CLICKHOUSE_USER
-    value: default
-  - name: CLICKHOUSE_PASSWORD
-    value: ''
-  - name: CLICKHOUSE_TZ
-    value: 'Asia/Ho_Chi_Minh'
-  - name: CLICKHOUSE_MAX_EXECUTION_TIME
-    value: '15'
-  - name: NEXT_QUERY_CACHE_TTL
-    value: '86400'
-EOF
-
-helm install -f values.yaml clickhouse-monitoring-release duyet/clickhouse-monitoring
-```
-
 ## Feedback and Contributions
 
 Feedback and contributions are welcome! Feel free to open issues or submit pull requests.

app/[host]/[query]/queries/history-queries.ts

@@ -37,6 +37,7 @@ export const historyQueriesConfig: QueryConfig = {
         AND if (notEmpty({event_time: String}), toDate(event_time) = {event_time: String}, true)
         AND if ({database: String} != '' AND {table: String} != '', has(tables, format('{}.{}', {database: String}, {table: String})), true)
         AND if ({user: String} != '', user = {user: String}, true)
+        AND if ({excluded_users: String} != '', not(has(splitByChar(',', {excluded_users: String}), user)), true)
       ORDER BY event_time DESC
       LIMIT 1000
   `,
@@ -86,6 +87,7 @@ export const historyQueriesConfig: QueryConfig = {
     database: '',
     table: '',
     user: '',
+    excluded_users: process.env.CLICKHOUSE_EXCLUDE_USER_DEFAULT || '',
   },
 
   filterParamPresets: [
@@ -114,6 +116,11 @@ export const historyQueriesConfig: QueryConfig = {
       key: 'duration_1m',
       value: '1',
     },
+    {
+      name: `user != ${process.env.CLICKHOUSE_EXCLUDE_USER_DEFAULT || ''}`,
+      key: 'excluded_users',
+      value: process.env.CLICKHOUSE_EXCLUDE_USER_DEFAULT || '',
+    },
   ],
 
   relatedCharts: [

components/data-table/data-table-faceted-filter.tsx

@@ -30,16 +30,25 @@ export function DataTableFacetedFilter({
   const pathname = usePathname()
   const { filterParamPresets = [], defaultParams = {} } = queryConfig
 
-  const selected = useMemo(
-    () => new URLSearchParams(searchParams),
-    [searchParams]
-  )
+  const selected = useMemo(() => {
+    const params = new URLSearchParams(searchParams)
+
+    // Add default params have not null value
+    Object.entries(defaultParams).forEach(([key, value]) => {
+      if (value !== '' && !params.has(key)) {
+        params.set(key, value as string)
+      }
+    })
+
+    return params
+  }, [searchParams, defaultParams])
+  console.log('selected', selected.toString())
 
   const filters = useMemo<
     NonNullable<QueryConfig['filterParamPresets']>
   >(() => {
     const filterNotFromPreset = Object.keys(defaultParams)
-      // key in URL Params
+      // Key in URL Params
       .filter((key) => selected.has(key))
       // And custom that value is not in presets
       .filter(
@@ -50,8 +59,10 @@ export function DataTableFacetedFilter({
           )
       )
       .map((key) => ({
-        name: `${key} = ${selected.get(key)} *`,
         key,
+        name: selected.get(key)
+          ? filterParamPresets.find((preset) => preset.key === key)?.name
+          : `${key} = N/A`,
         value: selected.get(key),
       })) as NonNullable<QueryConfig['filterParamPresets']>
     console.log('filterNotFromPreset', filterNotFromPreset)
@@ -90,7 +101,13 @@ export function DataTableFacetedFilter({
             name={name}
             value={value}
             isSelected={selected.get(key) === value}
-            href={getUpdatedHref(pathname, searchParams, key, value)}
+            href={getUpdatedHref(
+              pathname,
+              searchParams,
+              key,
+              value,
+              defaultParams
+            )}
             icon={preset?.icon}
             filterKey={key}
             filterValue={value}
@@ -133,9 +150,8 @@ function FilterMenuItem({
 }: FilterMenuItemProps) {
   return (
     <DropdownMenuItem>
-      <Link
+      <a
         href={href}
-        replace={true}
         data-selected={isSelected}
         className={cn(
           'flex flex-row content-between items-center gap-3',
@@ -144,7 +160,7 @@ function FilterMenuItem({
       >
         {Icon && <Icon className="mr-2 size-4 text-muted-foreground" />}
         <span>{name}</span>
-      </Link>
+      </a>
     </DropdownMenuItem>
   )
 }
@@ -154,12 +170,34 @@ function getUpdatedHref(
   pathname: string,
   searchParams: URLSearchParams,
   key: string,
-  value: string
+  value: string,
+  defaultParams: QueryConfig['defaultParams']
 ) {
   const newParams = new URLSearchParams(searchParams)
   if (newParams.get(key) === value) {
-    newParams.delete(key)
+    /**
+     * For example the query config has defaultParams = { type: 'abc' }
+     * and the URL has ?type=abc. If the user clicks on the filter to toggle it off,
+     * We should set the URL to ?type= instead of removing the key completely,
+     * as the default value is still 'abc'.
+     */
+    if (defaultParams?.[key] !== '') {
+      newParams.set(key, '')
+    } else {
+      /**
+       * If the default value is an empty string, we can just remove the key from the URL
+       * as there is no default value to fall back to.
+       *
+       * For example, if the query config has defaultParams = { type: '' }
+       * and the URL has ?type=abc. If the user clicks on the filter to toggle it off,
+       * We can set the URL to ? instead of ?type=abc.
+       */
+      newParams.delete(key)
+    }
   } else {
+    /**
+     * If the filter is not selected, we should set the URL to ?key=value
+     */
     newParams.set(key, value)
   }
   return `${pathname}?${newParams.toString()}`

docs/pages/advanced/_meta.ts

@@ -1,6 +1,8 @@
 export const meta = {
   'multiple-hosts': 'Multiple Hosts',
-  'custom-name': 'Custom ClickHouse Name',
+  'custom-name': 'Custom Host Name',
+  'queries-history': 'Queries History',
+  'self-tracking': 'Self-Tracking',
 }
 
 export default meta

docs/pages/advanced/multiple-hosts.mdx

@@ -18,18 +18,26 @@ All these hosts will share the same `CLICKHOUSE_USER` and `CLICKHOUSE_PASSWORD`.
 
 <Tabs items={['Docker (same user)', 'Docker (different user)']}>
   <Tabs.Tab>
-    ```bash docker run -it \ -e
-    CLICKHOUSE_HOST='http://ch-1:8123,http://ch-2:8123' \ -e
-    CLICKHOUSE_NAME='ch-1,ch-2' \ -e CLICKHOUSE_USER='default' \ -e
-    CLICKHOUSE_PASSWORD='' \ --name clickhouse-monitoring \
-    ghcr.io/duyet/clickhouse-monitoring:main ```
+    ```bash
+    docker run -it \
+      -e CLICKHOUSE_HOST='http://ch-1:8123,http://ch-2:8123' \
+      -e CLICKHOUSE_NAME='ch-1,ch-2' \
+      -e CLICKHOUSE_USER='default' \
+      -e CLICKHOUSE_PASSWORD='' \
+      --name clickhouse-monitoring \
+    ghcr.io/duyet/clickhouse-monitoring:main
+    ```
   </Tabs.Tab>
   <Tabs.Tab>
-    ```bash docker run -it \ -e
-    CLICKHOUSE_HOST='http://ch-1:8123,http://ch-2:8123' \ -e
-    CLICKHOUSE_NAME='ch-1,ch-2' \ -e CLICKHOUSE_USER='user1,user2' \ -e
-    CLICKHOUSE_PASSWORD='password1,password2' \ --name clickhouse-monitoring \
-    ghcr.io/duyet/clickhouse-monitoring:main ```
+    ```bash
+    docker run -it \
+      -e CLICKHOUSE_HOST='http://ch-1:8123,http://ch-2:8123' \
+      -e CLICKHOUSE_NAME='ch-1,ch-2' \
+      -e CLICKHOUSE_USER='user1,user2' \
+      -e CLICKHOUSE_PASSWORD='password1,password2' \
+      --name clickhouse-monitoring \
+    ghcr.io/duyet/clickhouse-monitoring:main
+    ```
   </Tabs.Tab>
 </Tabs>
 

docs/pages/advanced/queries-history.mdx

@@ -0,0 +1,50 @@
+import { Tabs } from 'nextra/components'
+
+# Query History
+
+## 1. Excluding monitoring users by default
+
+![](/excluding-users.png)
+
+You can configured the [`/history-queries`](http://clickhouse-monitoring.vercel.app/0/history-queries) page to filter out the `monitoring` users by default.
+To do this, using these environment variables:
+
+- `CLICKHOUSE_EXCLUDE_USER_DEFAULT`: for example `monitoring,healthcheck`.
+
+Examples
+
+<Tabs items={['Docker', 'Kubernetes']}>
+  <Tabs.Tab>
+    ```bash
+    docker run -it \
+        -e CLICKHOUSE_HOST='http://ch-1:8123' \
+        -e CLICKHOUSE_NAME='ch-1' \
+        -e CLICKHOUSE_USER='monitoring' \
+        -e CLICKHOUSE_PASSWORD='' \
+        -e CLICKHOUSE_EXCLUDE_USER_DEFAULT='monitoring,healthcheck' \
+        --name clickhouse-monitoring \
+    ghcr.io/duyet/clickhouse-monitoring:main
+    ```
+  </Tabs.Tab>
+  <Tabs.Tab>
+    ```bash
+    helm repo add duyet https://duyet.github.io/charts
+    
+    cat <<EOF >> values.yaml
+    env:
+    - name: CLICKHOUSE_HOST
+      value: http://ch-1:8123
+    - name: CLICKHOUSE_NAME
+      value: ch-1
+    - name: CLICKHOUSE_USER
+      value: monitoring
+    - name: CLICKHOUSE_PASSWORD
+      value: ''
+    - name: CLICKHOUSE_EXCLUDE_USER_DEFAULT
+      value: monitoring,healthcheck
+    EOF
+    
+    helm install -f values.yaml clickhouse-monitoring-release duyet/clickhouse-monitoring
+    ```
+  </Tabs.Tab>
+</Tabs>