Playing: 060: The Undocumented Web - scraping, private APIs, proxies and “alternative solutions”
Potluck - $100/hr × Redux Replacements × Full Stack Designers × JWT × VS Code Tips × More
Potluck - Editor Fonts × Portfolios × Meetup Tips × Switching to Windows × Freelancing Sources
Potluck EP × Remote Work × Headless WordPress × Good Client Questions × Alternate Careers × React API Credentials
Potluck EP × Is Redux Dead × Learning Quickly × Developing Solo × Specialist vs Generalist × Funnest Projects × Wes’ BBQ Course
Potluck EP × Vue.js × Headless WP × Typescript & Flow × Productivity × Server Side Rendering × Yeoman
Wes and Scott's Lives - Breakdancing, BBQ, Wives, Work/Life Balance, Problem Solving, YouTube Subscriptions
Snack Pack — CSS Frameworks, React HOC, Render Props, Coding Designers, Early Career Advice and a sound board!
Hosting & Servers — Heroku, Now, Galaxy, Digital Ocean, Linode, Docker, Netlify and more!
Jul 24th, 2018
The Undocumented Web - scraping, private APIs, proxies and “alternative solutions”👇 Download Show✏️ Edit Show Notes
What is the undocumented web? Scott and Wes dive into it, discussing APIs, faking, scraping, automation, proxies as well as tips and tricks for best practices.
Kyle Prinsloo's Freelancing & Beyond — Sponsor
Kyle Prinsloo teaches you everything you need to know about freelancing, including how to quit your job, earn a side-income and start taking control of your life. Check it out at https://studywebdevelopment.com/freelancing and use the coupon code "syntax" at checkout to get 25% off.
Freshbooks - Sponsor
Get a 30 day free trial of Freshbooks at freshbooks.com/syntax and put SYNTAX in the "How did you hear about us?" section.
3:05 - What Is the undocumented web?
- When traditional methods don’t give you what you want
- Wild West
6:10 - Undocumented APIS
- Instagram's private API
- Use dev tools' XHR Request
- React Dev Tools will also have the data
- Ep 008 - Wes Bos Origin Story
- Realtor.ca - MLS Mapping
- Simon Willison’s Zeit Day Talk about XML files
- Charles Proxy App - also good for debugging requests
17:34 - Faking
- Faking User Agents
- Sending headers from requests
- Free Blackberry wifi on plane
23:52 - Scraping
- What is scraping?
- Cheerio - jQuery like node package
27:30 - Automation
33:30 - Proxies
- Need a fresh IP address
- Need a different country
- Side note: Canadians buying stuff in USA M4S1P1 === 41100
- API throttling
39:00 - Tips and Tricks
- Multiple API keys
- diskDB - Text files as databases
- Search Github for API keys
- Inspect Element and Delete Overlays
- Delete Cookiesokay
44:45 - Stories
- Photo Voting
- MLS Listing
- Top Entrepreneur voting cheating
- Vanity Plate Checker
- My Hockey Team Sucks
- Vehicle auction bidding app