Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TwitterScraper does not download all images and videos. #11

Open
elvisyjlin opened this issue May 2, 2019 · 3 comments
Open

TwitterScraper does not download all images and videos. #11

elvisyjlin opened this issue May 2, 2019 · 3 comments
Labels
bug Something isn't working help wanted Extra attention is needed

Comments

@elvisyjlin
Copy link
Owner

My testing account was always realDonaldTrump and mediascraper.twitter works fine on it. However, for other accounts, for instance, sigrid_ig, videos are not captured by mediascraper.twitter. The html structure seems to be different between accounts. It needs handling case by case.

@elvisyjlin elvisyjlin added bug Something isn't working help wanted Extra attention is needed labels May 2, 2019
@archmord
Copy link

archmord commented May 2, 2019

Why dont you first capture all pic.twitter.com links from there download video and images

@elvisyjlin
Copy link
Owner Author

elvisyjlin commented May 3, 2019

Sounds like you have a good idea. I will appreciate your work if you give it a try.

But I want to mention that Twitter video is not saved as a single link. First, you need to retrieve a m3u8 streaming video list. Then combining the videos of .ts together gives you a whole Twitter video.

@archmord
Copy link

You can use https://github.com/twintproject/twint to get all the pic.twitter links
then get https://pbs.twimg.com/media/ from it and save

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

2 participants