API doesn't recognise paths with umlauts

[realpixelcode]

On my blog, I have written an article whose German URL contains umlauts: /de/2024/01/zum-wesen-freier-software-wie-dein-handy-zehn-jahre-überdauern-könnte.

In URL encoding, the path looks like this: /de/2024/01/zum-wesen-freier-software-wie-dein-handy-zehn-jahre-%C3%BCberdauern-k%C3%B6nnte.

The API endpoint /api/v0/stats/hits?start=2023-12-31 clearly shows that GoatCounter uses the URL-encoded path internally:

{
      "count": 7,
      "path_id": 25956711,
      "path": "/de/2024/01/zum-wesen-freier-software-wie-dein-handy-zehn-jahre-%C3%BCberdauern-k%C3%B6nnte",
      "event": false,
      "title": "Zum Wesen freier Software – oder: wie dein Handy zehn Jahre überdauern könnte",
      "max": 3,
      "stats": [ … ]
}

However, when trying to limit the results returned by /api/v0/stats/hits to just that path, the response is empty:

pixelcode@europa:~€ curl "https://user:token@forgoodeyesonly-blog.goatcounter.com/api/v0/stats/hits?path_by_name=true&include_paths=/de/2024/01/zum-wesen-freier-software-wie-dein-handy-zehn-jahre-%C3%BCberdauern-k%C3%B6nnte&start=2023-12-31"
{
  "hits": [],
  "total": 0,
  "more": false
}

For the English version of the path, it works fine:

pixelcode@europa:~€ curl "https://user:token@forgoodeyesonly-blog.goatcounter.com/api/v0/stats/hits?path_by_name=true&include_paths=/2024/01/on-the-essence-of-free-software-how-your-smartphone-could-outlast-ten-years&start=2023-12-31"
{
  "hits": [
    {
      "count": 8,
      "path_id": 25827053,
      "path": "/2024/01/on-the-essence-of-free-software-how-your-smartphone-could-outlast-ten-years",
      "event": false,
      "title": "On the essence of free software – or: How your smartphone could outlast ten years",
      "max": 1,
      "stats": [ … ]
}

Also, I learned from Andrea Bortolotti that you can fetch the view count from https://DOMAIN.goatcounter.com/counter/ + page path + .json. (Is this documented anywhere?) For the path /2024/01/on-the-essence-of-free-software-how-your-smartphone-could-outlast-ten-years, this works fine:

pixelcode@europa:~€ curl "https://forgoodeyesonly-blog.goatcounter.com/counter/2024/01/on-the-essence-of-free-software-how-your-smartphone-could-outlast-ten-years.json"
{
  "count": "8",
  "count_unique": "8"
}

However, for the path with the umlauts, it doesn't work:

pixelcode@europa:~€ curl "https://forgoodeyesonly-blog.goatcounter.com/counter/de/2024/01/zum-wesen-freier-software-wie-dein-handy-zehn-jahre-%C3%BCberdauern-k%C3%B6nnte.json"
{
  "count": "0",
  "count_unique": "0"
}

[arp242]

It's stored in the database as URL-encoded:

/de/2024/01/zum-wesen-freier-software-wie-dein-handy-zehn-jahre-%C3%BCberdauern-k%C3%B6nnte

This is true for all paths, not just that one.

Since the select is just a select * where path = ?, you need to "double-escape" the % (as %25), so that it's not interpreted but passed to the query. For example:

https://forgoodeyesonly-blog.goatcounter.com/counter/%2Fde/2024/01/zum-wesen-freier-software-wie-dein-handy-zehn-jahre-%25C3%25BCberdauern-k%25C3%25B6nnte.json

Presumably that also works for the API (didn't test).

It makes sense in a way, but is ... kind of confusing. Passing URLs in URls generally kind of is. I'm not sure if it can be fixed for every single case because some things really do need to be escaped (/, ?, etc.) but we can maybe store it a bit less aggressively URL-encoded.

[realpixelcode]

Thank you, this works fine! Maybe this could be documented somewhere?

[arp242]

Maybe this could be documented somewhere?

Yeah, need to see what I can improve and document the rest.