Add a "filtered" event

[WyohKnott]

Add a filtered event to apply postprocessing (python/metadata) to posts that have been excluded from download by the filtering rules (image and chapter filters).

The goal is to do analysis on filtered content in order to feed a blacklist.

[Twi-Hard]

Does this also work with exec postprocessors? That could be really useful to me.

[WyohKnott]

Does this also work with exec postprocessors? That could be really useful to me.

Welll a postprocesssor can be trigerred by such an event, including exec.

Here is my use case:

        "reddit":
        {
            "client-id": "seeeeecret",
            "user-agent": "Python:gallery-dl:v1.0 (by /u/noyoudont)",
            "comments": 0,
            "morecomments": false,
            "date-format": "%Y%m%d",
            "filename": "{date:%Y%m%d}_{author!g}_{subreddit!g}_{title[:120]!g}_{id}.{extension}",
            "directory": ["reddit", "users", "{author}"],
            "recursion": 1,
            "embeds": true,
            "videos": true,
            "parent-metadata": "_reddit",
            "metadata": true,
            "whitelist": ["redgifs", "soundgasm", "ytdl"],
            "image-filter": [
                "not (re.search(r'(' + '|'.join(line.strip() for line in open('gallery-dl/reddit/users/banned_users.txt', 'r')) + r')', author, re.IGNORECASE))",
                "not (re.search(r'(' + '|'.join(line.strip() for line in open('gallery-dl/reddit/users/banned_groups.txt', 'r')) + r')', subreddit, re.IGNORECASE))",
                "not (re.search(r'(M4F|M4A|M4TF|M4TM|M4NB|F4TF|F4TM|F4NB|TF4M|TF4A|TF4F|TM4M|TM4A|TM4F|NB4M|NB4A|NB4F|Script offer)', title, re.IGNORECASE))"
                ],
            "chapter-filter": [
                "not (re.search(r'(' + '|'.join(line.strip() for line in open('gallery-dl/reddit/users/banned_users.txt', 'r')) + r')', author, re.IGNORECASE))",
                "not (re.search(r'(' + '|'.join(line.strip() for line in open('gallery-dl/reddit/users/banned_groups.txt', 'r')) + r')', subreddit, re.IGNORECASE))",
                "not (re.search(r'(M4F|M4A|M4TF|M4TM|M4NB|F4TF|F4TM|F4NB|TF4M|TF4A|TF4F|TM4M|TM4A|TM4F|NB4M|NB4A|NB4F|Script offer)', title, re.IGNORECASE))"
                ],
            "image-unique": true,
            "postprocessors": [
                {
                    "name": "python",
                    "event": "filtered",
                    "function": "gallery-dl/reddit/users/append_and_sort.py:append_sort_deduplicate"
                }
            ]

I run append_and_sort.py:append_sort_deduplicate for every user who are caught in the filter. I have a list of banned subreddit in banned_groups.txt, every user who posts on theses subreddits get added to banned_users.txt', with the python script:

def append_sort_deduplicate(metadata):
    string_to_append = metadata["author"] if metadata["author"] != "[deleted]" else None

    # Append the new string to the file
    with open("gallery-dl/reddit/users/banned_users.txt", "a") as file:
        file.write(f"{string_to_append}\n")

    # Read all lines, sort them, remove duplicates, and write them back
    with open("/gallery-dl/reddit/users/banned_users.txt", "r+") as file:
        lines = file.readlines()
        # Strip whitespace and newline, sort, and use set for deduplication
        unique_lines = sorted(set(line.strip() for line in lines))
        
        # Clear the file content
        file.seek(0)
        file.truncate()
        
        # Write unique sorted lines back to the file
        file.writelines(f"{line}\n" for line in unique_lines)