If you are looking for a way to automate your WordPress website by taking the posts or content from other websites, you will need a content scrapping or fetching plugin. These plugins are also known as the RSS Aggregators.
We made a comprehensive list and review of the best free and paid WordPress content scrapping and RSS aggregator plugins.
This article also covers the terms related to the content scrapping so that you get a better picture of how this process works and how to obtain the best possible results. So, let us begin.
Table of content
- What is content scraping and how it works?
- What is RSS?
- RSS Aggregators
- How to fetch another website content
- How to fetch the entire post content from another website
- Best WordPress RSS Aggregators and content scraping plugins
What is content scraping and how it works?
Content scrapping defines the actions needed in order to grab the text and images from another website using programs or scripts. There are a lot of PHP scripts that can provide this functionality. They usually work in the following manner. The script opens the website or page and introduces itself as the regular visitor using a browser, then it finds the content of the page based on REGEX rules or CSS classes and IDs and fetches it. The next step is escaping and sanitizing the fetched data and inserting it into the database.
What is RSS?
RSS stands for Really simple syndication and it’s a standardized format (uses XML for formating) that allows users and applications to pick up the latest news and announcements from the websites. You can read more about the RSS on the official Wikipedia RSS definition page.
All WordPress websites are also generating RSS feeds. The feeds are accessible at www.websitedomain.com/feed/ and they are usually listing the 10 last posts with titles and excerpt texts. This can be useful if you want to track the announcements and the latest news from a website. There are many popular RSS reading programs for computers and mobile devices that allow users to pick various RSS feeds and follow the website’s announcement.
RSS Aggregators
The terms RSS Aggregators are related to the programs or plugins that can fetch the website posts content from the RSS feed. These programs usually working by taking the entered feed addresses and then picking up the changes and accessing the posts based on the feeds URL.
How to fetch another website content?
As we already mentioned, the RSS Aggregators are programs or plugins. If you install the RSS aggregator plugin on your website, you can fill it in with the feed links fro the websites you plan to scrape content from. Then the plugin connects to the feed URL and reads the content from them, check the content and filter it and then insert the feed title and content as the articles or posts.
Please note that scrapping only RSS feed from another website will not grab the entire post content and images, but just the titles and excerpt text.
How to fetch the entire post content from another website?
There are some certain RSS plugins that can scrap the entire posts using the ability to fetch the original post URL from the feed. They will access the post URL and act as the common user. They will identify them as the browser and computer similar to the plain visitor. The plugin will find the part of the post that is related to the content and take the text, images, and videos code from it.
Then the content will get filtered, sanitized, and inserted on your own website by following the rules you set inside the plugin.
Best WordPress RSS Aggregators and content scraping plugins
After you got familiar with how to fetch the posts and how content scraping works, it’s time we present you with the plugins that are capable of doing this.
1. WP Automatic (paid)
We start with the WP Automatic paid WordPress plugin used for content scraping. This is the most complete content scraper plugin available for WordPress.
The plugin is packed with a lot of features, like:
- Fetching post or page content using RSS feeds
- Fetching post or pages from provided website URL
- Amazon and eBay products fetching
- It can fetch posts, videos, and images from Instagram, Facebook, Youtube, and Vimeo, Flick, and iTunes.
- It can be used for fetching Envato marketplace items
- It can set post author and category
- Populates the Featured image, post title, category, and import tags
- Strip the content based on entered CSS Id, CSS class, Regex pattern, and more
- It can truncate titles, search and replace a word in the content
- Link removal, internal or external
- Reading and using data from open graph
- Posting to custom taxonomy and custom post types
- Stripping HTML tags based on your input
- Setting ALT and titles for scraped images
- Scraping products from Amazon and eBay and posting into WooCommerce
- Adding affiliate links to scraped content or products
- Posting data into custom fields
- … and many more…
Fetching via API key
The plugin offers support for fetching content from Envato, Facebook, YouTube, Vimeo, Flickr, and iTunes via API. So, it is all legal and you just need to obtain the API keys from these respected websites.
Another thing that is of most interest to us it the ability to fetch any kind of posts from websites.
Website post scraping
The plugin offers a way of creating campaigns that can run in the desired time intervals. The campaigns can be set to fetch the posts from any kind of website using RSS feed or by entering the website URL. It also offers all needed options to customize the fetched content, like search and replace, setting the post in the desired category, assigning the post author or your choice, downloading the post images to your server, setting up various rules for removing certain part of the fetched content in order to acquire a perfect content copying.
WP Automatic can also fetch multipage posts and it can browse thru the page pagination without any problems.
The plugin also allows the visual selection of the post and page content you plan to scrap.
Automatic content scrapping
You can set the cron job that executes in the desired time frame and the plugin will work automatically. This is useful for setting up an automated process that will post the new articles instead of you. In this way, you can save a lot of time and avoid manual content copying.
You can test-drive WP Automatic plugin by requesting a live demo here.
2. Scrapes (paid)
Scrapes is yet another WordPress content scraping premium plugin. The plugin is much simpler in terms of options and features, but it also works well and scraps the entire post content based on the RSS feed input.
As we already stated, this plugin doesn’t offer as many options as the WP Automatic, but it’s worth checking out. This plugin doesn’t offer visual content selection and it can take the content only from the RSS feeds. It uses the RSS to find the website post URLs and works the rest of the tasks automatically.
The plugin can also fetch the multi-paged posts and it can use the pagination to browser the posts on multiple pages.
Scrapes also support WooCommerce and it can import the products from other e-commerce websites and Amazon or eBay. So, it can be useful for creating an affiliate webshop.
You can try the LIVE DEMO of Scrapes and see if it fits into your needs.
3. WP Scraper (free and paid)
WP Scraper plugin comes with the free and paid plugin version. It’s very simple and easy to use. The plugin offers the visual content selection similar to the one inside the WP Automatic plugin.
This plugin can also support the cron jobs and you can set the plugin to fetch the new content automatically, based on the desired time frame.
WP Scraper plugin is offering great features in both the free and PRO versions. It includes the following options:
- Visual interface for selecting content
- It can use CSS selectors (optional)
- It can download images to the Media library
- Simply to use, just add source page URL
- Automatically populate the featured image, title, tags, and categories.
- Strip unwanted CSS code, iframes, and/or videos from content
- Remove links from the content.
- Post to a selected category.
Overall, the WP Scraper plugin is also a good solution, yet it’s simple to use and working well.
You can also test-drive the WP Scraper by downloading the Free WP Scraper plugin.
4. Crawlomatic (paid)
This is another great premium content scraper that can be used to fetch posts and page, weather forecast, sports results, currency changes, bus and airplane schedules, social profiles, and many more.
The Crawlomatic is another powerful website scraper packed with tons of features. It can also fetch posts and pages, feeds, products, and videos. It supports adding products in the WooCommerce fetched from Amazon or from another webshop. The plugin uses an intelligent approach and it can save cookies and simulate common visitors, avoiding detection.
This plugin also features visual content selection, it can scrape jobs, weather forecast, currencies changes, social profiles, video embed codes, bus and airplane schedules, and more.
5. Scraper by Ink Themes (paid)
The Scraper is a premium content scraping plugin made by Ink Themes. The plugin comes with a simple and easy to use interface, it features all great functionality that this type of plugin should be offering.
Some of the key features are visual content selection, scraping models, search and replace feature, content spinner (like the one inside WP Automatic), automation, can place cookies, create image galleries from fetched images, skip duplicates and more.
The Scaper plugin ais also working with WooCoommerce and can be used to scrap products from Amazon, eBay, and 3rd party online shops. The plugin is also supporting automation and can be executed in desired time intervals via cron jobs.
6. WP RSS Aggregator (free and paid)
WP RSS Aggregator comes in the free and pro version. This is one of the oldest WordPress aggregator plugins that evolved over time. The free plugin version can fetch RSS feeds and scrap the content out of them, but it ads the content via shortcodes and Gutenberg blocks. So, the free version will not populate the posts, categories, features images, and authors for your posts, instead, it will use shortcodes.
It can be used for unlimited feed sources and for adding an unlimited number of scraped posts. The plugin supports fetching of posts and pages, videos, images, weather forecasts, podcasts, currency rates, etc.
You can get the free version of the WP RSS Aggregator from the WordPress plugin repository.
The WP RSS Aggregator PRO version works differently and it will fetch content and populate it into new posts on your website. It will add Featured images, set the categories, author, and will fetch the entire post content, just like other paid plugins.
It also supports the custom field mapping so you can set the plugin to fetch some value from another website and insert it into your custom field.
WP RSS Aggregator Pro also offers the add-ons that can further extend the plugin core features.
Summary on content scraping and RSS Aggregators
All of the above-mentioned plugins are capable of providing you with the tools needed to automate your WordPress website and publish scraped content automatically. Please consider that it might take some time before you set up everything properly.
We advise you to leave the source link back to the original content to avoid any penalties that might be taken against your website by Search engines. If you can, always make sure that you spin the content to gain better results on the search engines in terms of content originality.
If you have any questions, comments, and requests, feel free to write them down in the comments section.
