Skip to content

Add X/Instagram comments and media ZIP export#74

Open
danilo252020 wants to merge 1 commit intodigitalmethodsinitiative:masterfrom
danilo252020:minha-contribuicao-v1
Open

Add X/Instagram comments and media ZIP export#74
danilo252020 wants to merge 1 commit intodigitalmethodsinitiative:masterfrom
danilo252020:minha-contribuicao-v1

Conversation

@danilo252020
Copy link
Copy Markdown

Summary

This PR adds support for collecting comments from X/Twitter and Instagram, and adds a media ZIP export option for TikTok, Instagram, and X/Twitter.

Changes

  • Add a new X/Twitter comments module for TweetDetail reply/comment responses.
  • Add a new Instagram comments module for comment API responses.
  • Normalize comment records with fields such as comment_id, text, user, post_id, post_url, parent_comment_id, and thread metadata.
  • Add .zip media export action for TikTok, Instagram, and X/Twitter.
  • Include a manifest.json inside media ZIP exports to map each downloaded file back to its source post.
  • Update README support notes and test configuration.

Testing

  • Tested locally with Zeeschuimer exports for X/Twitter, Instagram, and TikTok.
  • Verified media ZIP exports for images/videos across supported platforms.
  • Ran:
npm test
node --check popup\interface.js
node --check modules\twitter-comments.js
node --check modules\instagram-comments.js
git diff --check

@dale-wahl
Copy link
Copy Markdown
Member

Hey @danilo252020, thank you for the PR. We will discuss internally. I definitely see the value in the media download feature (downloading via Zeeschuimer vs requiring 4CAT upload is much more flexible and accessible).

I also see the value in mapping specific data points (and need for the media downloads/metadata to function properly). We have a principle of trying to keep the original item as is and mapping later in 4CAT. This allows timely collection even as platforms change their data structures which we can remap later. But perhaps we could explore a method to adding specific fields to the metadata collected if available but not requiring it (so data is still captured even if data structures are updated causing issues).

Thanks again.

@danilo252020
Copy link
Copy Markdown
Author

Thanks for the thoughtful feedback.

That principle makes sense to me. I’m using Zeeschuimer in a research project where we need to analyze comments from Instagram and X/Twitter, including which profile wrote each comment, the comment text, and which post the comment belongs to. That was the main motivation for proposing the comment modules and the extra mapping fields.

I tried to keep the original platform payload intact and add the normalized fields on top only when they are available, mainly so comments and media files can be mapped back to their source post during local export.

I’m happy to adjust the implementation to better match the project’s architecture. One possible direction would be:

  • keep the original item payload unchanged as much as possible;
  • treat fields such as comment_id, post_id, post_url, user, and media metadata as optional helper metadata;
  • avoid dropping captured records when some of these fields are missing;
  • keep the media ZIP manifest as the main place where local download metadata is normalized.

If helpful, I can revise the PR in that direction, especially making the comment/media mapping more defensive and optional rather than required.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants