Web Scraping Leads: A Responsible, GDPR-Aware Guide

You've got a CSV open in one window and your email tool in the other. Eight thousand Instagram profiles, a few hundred addresses in the public email column, and a launch date that isn't moving.
The questions that follow aren't about scraping mechanics. Can you email these people? Should you? Which addresses still work, and does a list this size justify the risk of a complaint, a bounced domain or a regulator's letter?
This guide takes those questions in the order they arrive: what you collected, where it came from, what rules apply, how to clean it, and how to prove the list earned its place. The short version up front: a public email isn't permission, and a bigger list isn't a better one.
Table of Contents
- What Web Scraping Leads Are and Why Teams Use Them
- Where Lead Data Comes From and What Each Source Yields
- Why Public Contact Data Still Needs Rules
- How to Build a Scraped Lead List Step by Step
- What the finished file should look like
- Quality Checks That Keep Wrong Contacts Out
- Measuring Whether Scraped Leads Earn Their Place
- Run a Sample-First Test Before You Scale
What Web Scraping Leads Are and Why Teams Use Them
A small commercial cleaning company has just won a three-building contract with a property management group. The owner wants to contact fifty more property managers in the same city by Friday, but the company has no database subscription and only three names from a trade show. A scraped lead list can help find prospects to assess, though collecting their details is only the first step.
Web scraping leads means automatically collecting publicly visible business information, such as names, websites, public contact fields and audience lists, then organizing it into a file that can be sorted and searched. It works like copying a public directory by hand, entry by entry, except software gathers the information and lets a team filter it by category, location or audience.

Collect, structure, qualify. These are the stages of building a useful lead file. Collection gathers the records. Structuring means giving fields consistent names, recording each row's source and noting when it was collected. Qualification is the judgment step: deciding whether a row represents a business worth contacting or someone whose details happen to be visible.
Automated web crawling began in the early 1990s, with crawlers such as the World Wide Web Wanderer and JumpStation gathering pages and following links. Lead generation applies the same general idea to a narrower audience. Instead of collecting pages across the web, a workflow might focus on the following list of a local property management firm.
Teams turn to scraping when manual research cannot keep pace with a deadline. A structured file makes it easier to sort prospects by location, category or source before an outbound sales representative writes a message. The spreadsheet is a starting map, not a ready-made campaign: public availability does not by itself make contact appropriate or lawful.
That distinction matters because a larger file can create more work without producing a stronger pipeline. Teams need to judge whether records fit their audience, whether the contact details remain usable and whether outreach is permitted. Treat the list as raw material for a measured process, not as permission to market to everyone it contains.
Where Lead Data Comes From and What Each Source Yields
| Source | What You Collect | Typical Use | Yield or Caveat |
|---|---|---|---|
| Following list | Accounts a target brand follows | Finding peer businesses and vendors | Usually a better contact rate than followers |
| Follower list | Accounts that follow a target | Reaching a competitor's audience | Broader and noisier; lower contact rate |
| Hashtag posts | Accounts posting under a topic tag | Finding active niche businesses | Depends on recent posting activity |
| Post commenters and likers | Accounts engaging with one post | Spotting interest in a topic | Shows engagement, not permission to contact |
| Public business directories | Listed names, websites and contact fields | Local service and B2B lists | Listings go stale; verify before use |
| Company websites | Team, contact and press pages | Naming decision-makers | Accuracy varies; personal data rules still apply |
If you take one thing from that table, take this: scraping a following list almost always returns a better email rate than scraping the followers of the same account. The accounts a business chooses to follow tend to be peers, suppliers, competitors and local partners rather than its own customers, so the list skews toward other businesses that publish contact details.
Yield is where expectations usually break. On average, about 10% of scraped users have public emails, and the rate swings heavily by niche. A tight segment of marketing agencies in one metro can produce better usable contacts than a broad hashtag pull five times its size, because the smaller job matches the buyers you sell to.
Two sources are routinely misread. Commenters and likers show that someone engaged with a topic, which is a weak signal of interest and no signal of consent. Hashtag results depend on how recently accounts posted under the tag, so a popular tag can hand you a list of businesses that stopped posting eighteen months ago.
Choose backwards from the buyer. Write down where those buyers already cluster, whether that's a directory, a following list or a trade association page, then sample it before committing to scale. If you're still mapping the territory, this guide to finding the right buyers covers the segment logic that decides which source is worth scraping at all.
Why Public Contact Data Still Needs Rules
Public means permitted is the most expensive assumption in lead generation, and two enforcement cases dismantle it. A documented GDPR scraping case review sets out three numbers worth memorising:
- €220,000, the March 2019 fine against a company that scraped business contact details of more than 6 million Polish citizens from a public commercial register and used them for commercial services.
- €240,000, the December 2024 CNIL fine over scraped LinkedIn contact details belonging to users who had restricted profile visibility to first- and second-degree connections.
- €20 million or 4% of worldwide annual turnover, whichever is higher, as the ceiling on GDPR penalties.
Notice what the Polish register was: public, official, and full of business contacts. The collection wasn't the central failure. The company failed the GDPR's Article 14 transparency duty, which applies when personal data comes from a source other than the individual. Nobody had told the six million people their details were being used.
That reframes the whole exercise. Names, emails and phone numbers can remain regulated personal data even when they sit on a public page, and the obligation follows the people in the dataset, not the country where the collector or the list seller operates.
| Field or Record | Handling | Reason |
|---|---|---|
| Business email on a company profile | Keep, with source and date recorded | Published for professional contact but still needs a documented purpose |
| Personal mobile number | Suppress unless clearly used for business | Personal data that GDPR still covers |
| Bio with personal details | Drop unless the purpose needs it | Data minimisation |
| Contact on a restricted-visibility profile | Exclude | Owner limited who could see it; see the CNIL case |
| Opt-out or deletion request | Remove and add to suppression list | Honors the request and prevents re-import |
| Sensitive or ambiguous profile | Suppress | Risk outweighs the value of the contact |
Data minimisation is the discipline behind most of those rows. Collect the fields the stated purpose needs and nothing else, because every extra column is another piece of personal data you now have to secure, justify and delete on request. If your only use is a one-time introduction email, you don't need a job title history or a personal mobile number.
Before any scrape, confirm five things: separate business or creator contacts from personal accounts; document the source, collection date and intended use for each file; set a retention period and delete data when it expires; keep an auditable suppression list for opt-outs and deletion requests; and define the outreach basis separately for each target market.
Teams selling into several regions find that last point the hardest, because the lawful basis for a cold email in one country isn't automatically the basis for the next. If you operate a SaaS product across borders, this CCPA and GDPR compliance overview is a useful map of where the two regimes diverge on consent, retention and access. Marketing teams can go deeper on the email-specific side in this guide to GDPR rules for email marketing.
How to Build a Scraped Lead List Step by Step
Take the cleaning company again: one city, one buyer type, a Friday deadline, no database subscription. The sequence below is the one I'd hand a new analyst, because it still makes sense when someone from legal asks what happened.

1. Write the target profile as one sentence. "Property managers at commercial real estate firms with at least two buildings in the metro area, in a role that can approve a cleaning contract." Every later filter, source choice and quality flag descends from that sentence, so vagueness here becomes noise later.
2. Pick the source where those buyers cluster. For the firms, local business directories. For the people, the following lists of the larger property management groups, since those accounts follow peers and vendors.
3. Run a test batch of a few hundred profiles first. Field quality and yield become known before you commit to a full run, and you learn whether this niche publishes contact details at all.
4. Run the collection and leave the raw export untouched. The raw file is your record of what was collected and when. Speed isn't the constraint here; a few thousand profiles takes minutes, and our cloud workers run at about two minutes per 1,000 accounts.
5. Structure the fields properly. Username, profile URL, name, bio, category, website, public business email, public phone, source and collection date. A row without a source and a date can't be defended six months later.
6. Drop rows whose contact details aren't clearly meant for professional contact. Suppress ambiguous records instead of keeping them just in case. Ghent University guidance classifies social-media information as "pseudo-public" and holds that GDPR principles apply even when data is visible to the public, as this Instagram scraping compliance guide summarises. Visible isn't the same as reusable.
7. Deduplicate by username first, then by email. One business can appear under three handles and two addresses, and duplicate sends are the fastest way to look like a spam operation.
8. Verify every address and strip anything on the suppression list. The manual and automated email extraction workflow shows where extraction ends and verification should begin. Hard bounces damage the sending domain for every future campaign, not just this one.
9. Score for fit and freshness, then import with the source field intact. The source field is what lets you compare performance later.
If you'd rather watch the pipeline in motion before building it, this walkthrough covers the same stages and the same handoff points.
What the finished file should look like
One row per profile, UTF-8 CSV, with source and collection date visible in every row. Email and phone in separate fields. Column names matching what your CRM expects, so the import doesn't create a new property nobody maintains. No merged cells, no colour coding as the only record of a decision, no tab called "final_final".
Keep the raw export and the working list as two separate files with two separate purposes. The export is evidence of what was collected and when. The working list is the derivative you're willing to defend. Outreach never starts from the raw export, because nothing in it has been qualified yet.
Quality Checks That Keep Wrong Contacts Out
A rep sends a specific email to Sarah Chen, whose profile seems to match a target account. The reply comes from another Sarah Chen. Now the CRM links the company to someone who has never heard of the product. A name can suggest a match, but it cannot confirm one.
A matching display name can produce homonym errors, while pseudonyms and inconsistent usernames make identity harder to establish. Treat identity as a set of clues: profile details should support one another before a contact is marked usable.

| Quality Issue | Warning Sign | Fix |
|---|---|---|
| Homonym or pseudonym match | Common name with no business metadata | Compare bio, category and website; lower confidence |
| Shared email across profiles | Same address on several handles | Flag and check each profile before outreach |
| Wrong entity contacted | Name matches but website differs | Use username and profile URL as the key |
| Stale profile | Inactive account or an old collection date | Refresh and re-verify before sending |
| Bounced or invalid address | Hard bounces in early sends | Verify and remove before the next wave |
Keep the username and profile URL as the primary key, not the display name. Compare the bio, category, website and public contact fields for consistency. If one email appears on multiple profiles, flag it for review rather than assuming the accounts belong to the same person. A generic name or missing business details should lower confidence.
The CRM handoff is another place errors can spread. Deduplicating only by full name or handle can merge different people, or attach a contact to the wrong company. A stable key lets teams add new fields without overwriting a different person's record. The guide to contact data enrichment explains how to match fields while preserving the distinctions between records.
A public email is still only a contact detail, not permission to send. Before a list leaves the team, check that every row has a username, profile URL, source and collection date. Confirm that addresses have been verified, duplicates reviewed, and profile details checked for conflicts. Remove anyone on the suppression list, and use the contact only when the planned outreach is lawful.
Measuring Whether Scraped Leads Earn Their Place
A bigger export isn't a result. B2B contact data decays by roughly 2.1% a month, or about 22.5% a year, and automated extraction accuracy ranges from 85% to 95% depending on site complexity and field type, according to this analysis of verification and data freshness. A list you built in January is measurably weaker by summer, whether or not anyone touched it.
A lead freshness score handles that. Combine the collection date, recent profile activity, email verification status, business category, geography and audience source into one number per row, then set refresh intervals from it. Rows that score low get re-collected or dropped, not mailed.
| Stage | What It Measures | Why It Matters |
|---|---|---|
| Expected yield | Share of profiles with a public email | About 10 percent in a broad job; tighter niches can differ |
| Delivery rate | Share of emails that reach an inbox | Shows whether addresses were verified |
| Reply rate | Share of contacts who respond | Shows whether the message fits the audience |
| Qualified conversations | Replies that match the buyer profile | Separates interest from fit |
| Meetings booked | Conversations that reach a meeting | Shows incremental pipeline |
| Opt-outs | Contacts who ask to be removed | Signals consent risk and feeds the suppression list |
Here's the contrarian part. A 10% email yield from a tighter, better-fitting segment can outperform a much larger but staler list, because broad scraping amplifies low-fit contacts faster than it finds good ones. Volume doesn't create pipeline. Relevance does.
So test by cohort. Split the file by source, run the same message to each group, and compare outcomes per source rather than across the whole export. That's how you prove incremental pipeline instead of a bigger spreadsheet. For the wider picture on how these lists feed a demand engine, see this guide to B2B lead generation.
Run a Sample-First Test Before You Scale
A scraped list is a starting point, not permission to send. Test one defined audience first, check whether the contacts are fresh and relevant, then decide if the results justify a larger job.

Public availability and lawful, useful outreach are different checks. A practitioner framework for compliant Instagram lead generation emphasizes that a public username alone does not authorize enrichment, cold outreach, or indefinite storage. Relevance matters too: the linked framework notes that 73% of B2B buyers avoid sellers whose outreach is irrelevant to their business. A large export can still be a poor investment if its audience does not fit or its contact details have gone stale.
- Confirm the lawful basis, purpose, and retention period before collection. Treat anti-spam obligations and opt-out requests as part of the campaign, not cleanup after it.
- Pull a small cohort from one source and record its yield, delivery rate, and reply rate. This gives you a baseline before you spend more.
- Compare sources side by side and remove cohorts producing stale or low-fit contacts. Keep the audience source attached to your analysis so results remain interpretable.
- Scale only a source that performs, then set a refresh interval. Like produce, contact data loses usefulness with time, so a winning cohort still needs review.
- Ask for a sample file before buying anything larger. Check that its fields, username, name, email, phone, bio, follower count, category, country, and URL, match your CRM needs. Add your own source and collection-date columns.
HarvestMyData extracts publicly available contact information and does not access private accounts or protected content. Users agree to lawful use, anti-spam compliance, and honoring opt-out requests. A sample CSV is available to preview its structure before ordering.
Check compliance, test the cohort, then scale.
HarvestMyData extracts publicly listed emails and phone numbers from Instagram followers, following lists, hashtags, commenters, and likers. Each profile is delivered as a row in UTF-8 CSV or Excel, with username, full name, bio, follower count, category, and website alongside available contact details. Jobs include a live status page and are typically delivered within an hour. Visit HarvestMyData to review extraction rates by audience type and price a job for your segment.
We built HarvestMyData to handle all of this for you.
No proxies, no code, no account needed.