HARVESTMYDATA
by HarvestMyData

Web Scraping Leads: A Responsible, GDPR-Aware Guide

web scraping leadslead generationB2B prospectingGDPR lead datadata scraping
Web Scraping Leads: A Responsible, GDPR-Aware Guide

You've got a CSV open in one window and your email tool in the other. Eight thousand Instagram profiles, a few hundred addresses in the public email column, and a launch date that isn't moving.

The questions that follow aren't about scraping mechanics. Can you email these people? Should you? Which addresses still work, and does a list this size justify the risk of a complaint, a bounced domain or a regulator's letter?

This guide takes those questions in the order they arrive: what you collected, where it came from, what rules apply, how to clean it, and how to prove the list earned its place. The short version up front: a public email isn't permission, and a bigger list isn't a better one.

Table of Contents

- What the finished file should look like

What Web Scraping Leads Are and Why Teams Use Them

A small commercial cleaning company has just won a three-building contract with a property management group. The owner wants to contact fifty more property managers in the same city by Friday, but the company has no database subscription and only three names from a trade show. A scraped lead list can help find prospects to assess, though collecting their details is only the first step.

Web scraping leads means automatically collecting publicly visible business information, such as names, websites, public contact fields and audience lists, then organizing it into a file that can be sorted and searched. It works like copying a public directory by hand, entry by entry, except software gathers the information and lets a team filter it by category, location or audience.

A stressed businesswoman looking at an empty prospect spreadsheet in her office, feeling overwhelmed by sales outreach.

Collect, structure, qualify. These are the stages of building a useful lead file. Collection gathers the records. Structuring means giving fields consistent names, recording each row's source and noting when it was collected. Qualification is the judgment step: deciding whether a row represents a business worth contacting or someone whose details happen to be visible.

Automated web crawling began in the early 1990s, with crawlers such as the World Wide Web Wanderer and JumpStation gathering pages and following links. Lead generation applies the same general idea to a narrower audience. Instead of collecting pages across the web, a workflow might focus on the following list of a local property management firm.

Teams turn to scraping when manual research cannot keep pace with a deadline. A structured file makes it easier to sort prospects by location, category or source before an outbound sales representative writes a message. The spreadsheet is a starting map, not a ready-made campaign: public availability does not by itself make contact appropriate or lawful.

That distinction matters because a larger file can create more work without producing a stronger pipeline. Teams need to judge whether records fit their audience, whether the contact details remain usable and whether outreach is permitted. Treat the list as raw material for a measured process, not as permission to market to everyone it contains.

Where Lead Data Comes From and What Each Source Yields

SourceWhat You CollectTypical UseYield or Caveat
Following listAccounts a target brand followsFinding peer businesses and vendorsUsually a better contact rate than followers
Follower listAccounts that follow a targetReaching a competitor's audienceBroader and noisier; lower contact rate
Hashtag postsAccounts posting under a topic tagFinding active niche businessesDepends on recent posting activity
Post commenters and likersAccounts engaging with one postSpotting interest in a topicShows engagement, not permission to contact
Public business directoriesListed names, websites and contact fieldsLocal service and B2B listsListings go stale; verify before use
Company websitesTeam, contact and press pagesNaming decision-makersAccuracy varies; personal data rules still apply

If you take one thing from that table, take this: scraping a following list almost always returns a better email rate than scraping the followers of the same account. The accounts a business chooses to follow tend to be peers, suppliers, competitors and local partners rather than its own customers, so the list skews toward other businesses that publish contact details.

Yield is where expectations usually break. On average, about 10% of scraped users have public emails, and the rate swings heavily by niche. A tight segment of marketing agencies in one metro can produce better usable contacts than a broad hashtag pull five times its size, because the smaller job matches the buyers you sell to.

Two sources are routinely misread. Commenters and likers show that someone engaged with a topic, which is a weak signal of interest and no signal of consent. Hashtag results depend on how recently accounts posted under the tag, so a popular tag can hand you a list of businesses that stopped posting eighteen months ago.

Choose backwards from the buyer. Write down where those buyers already cluster, whether that's a directory, a following list or a trade association page, then sample it before committing to scale. If you're still mapping the territory, this guide to finding the right buyers covers the segment logic that decides which source is worth scraping at all.

Why Public Contact Data Still Needs Rules

Public means permitted is the most expensive assumption in lead generation, and two enforcement cases dismantle it. A documented GDPR scraping case review sets out three numbers worth memorising:

  • €220,000, the March 2019 fine against a company that scraped business contact details of more than 6 million Polish citizens from a public commercial register and used them for commercial services.
  • €240,000, the December 2024 CNIL fine over scraped LinkedIn contact details belonging to users who had restricted profile visibility to first- and second-degree connections.
  • €20 million or 4% of worldwide annual turnover, whichever is higher, as the ceiling on GDPR penalties.

Notice what the Polish register was: public, official, and full of business contacts. The collection wasn't the central failure. The company failed the GDPR's Article 14 transparency duty, which applies when personal data comes from a source other than the individual. Nobody had told the six million people their details were being used.

That reframes the whole exercise. Names, emails and phone numbers can remain regulated personal data even when they sit on a public page, and the obligation follows the people in the dataset, not the country where the collector or the list seller operates.

Field or RecordHandlingReason
Business email on a company profileKeep, with source and date recordedPublished for professional contact but still needs a documented purpose
Personal mobile numberSuppress unless clearly used for businessPersonal data that GDPR still covers
Bio with personal detailsDrop unless the purpose needs itData minimisation
Contact on a restricted-visibility profileExcludeOwner limited who could see it; see the CNIL case
Opt-out or deletion requestRemove and add to suppression listHonors the request and prevents re-import
Sensitive or ambiguous profileSuppressRisk outweighs the value of the contact

Data minimisation is the discipline behind most of those rows. Collect the fields the stated purpose needs and nothing else, because every extra column is another piece of personal data you now have to secure, justify and delete on request. If your only use is a one-time introduction email, you don't need a job title history or a personal mobile number.

Before any scrape, confirm five things: separate business or creator contacts from personal accounts; document the source, collection date and intended use for each file; set a retention period and delete data when it expires; keep an auditable suppression list for opt-outs and deletion requests; and define the outreach basis separately for each target market.

Teams selling into several regions find that last point the hardest, because the lawful basis for a cold email in one country isn't automatically the basis for the next. If you operate a SaaS product across borders, this CCPA and GDPR compliance overview is a useful map of where the two regimes diverge on consent, retention and access. Marketing teams can go deeper on the email-specific side in this guide to GDPR rules for email marketing.

How to Build a Scraped Lead List Step by Step

Take the cleaning company again: one city, one buyer type, a Friday deadline, no database subscription. The sequence below is the one I'd hand a new analyst, because it still makes sense when someone from legal asks what happened.

A five-step infographic showing how to build a scraped lead list for a commercial cleaning business.

1. Write the target profile as one sentence. "Property managers at commercial real estate firms with at least two buildings in the metro area, in a role that can approve a cleaning contract." Every later filter, source choice and quality flag descends from that sentence, so vagueness here becomes noise later.

2. Pick the source where those buyers cluster. For the firms, local business directories. For the people, the following lists of the larger property management groups, since those accounts follow peers and vendors.

3. Run a test batch of a few hundred profiles first. Field quality and yield become known before you commit to a full run, and you learn whether this niche publishes contact details at all.

4. Run the collection and leave the raw export untouched. The raw file is your record of what was collected and when. Speed isn't the constraint here; a few thousand profiles takes minutes, and our cloud workers run at about two minutes per 1,000 accounts.

5. Structure the fields properly. Username, profile URL, name, bio, category, website, public business email, public phone, source and collection date. A row without a source and a date can't be defended six months later.

6. Drop rows whose contact details aren't clearly meant for professional contact. Suppress ambiguous records instead of keeping them just in case. Ghent University guidance classifies social-media information as "pseudo-public" and holds that GDPR principles apply even when data is visible to the public, as this Instagram scraping compliance guide summarises. Visible isn't the same as reusable.

7. Deduplicate by username first, then by email. One business can appear under three handles and two addresses, and duplicate sends are the fastest way to look like a spam operation.

8. Verify every address and strip anything on the suppression list. The manual and automated email extraction workflow shows where extraction ends and verification should begin. Hard bounces damage the sending domain for every future campaign, not just this one.

9. Score for fit and freshness, then import with the source field intact. The source field is what lets you compare performance later.

If you'd rather watch the pipeline in motion before building it, this walkthrough covers the same stages and the same handoff points.

What the finished file should look like

One row per profile, UTF-8 CSV, with source and collection date visible in every row. Email and phone in separate fields. Column names matching what your CRM expects, so the import doesn't create a new property nobody maintains. No merged cells, no colour coding as the only record of a decision, no tab called "final_final".

Keep the raw export and the working list as two separate files with two separate purposes. The export is evidence of what was collected and when. The working list is the derivative you're willing to defend. Outreach never starts from the raw export, because nothing in it has been qualified yet.

Quality Checks That Keep Wrong Contacts Out

A rep sends a specific email to Sarah Chen, whose profile seems to match a target account. The reply comes from another Sarah Chen. Now the CRM links the company to someone who has never heard of the product. A name can suggest a match, but it cannot confirm one.

A matching display name can produce homonym errors, while pseudonyms and inconsistent usernames make identity harder to establish. Treat identity as a set of clues: profile details should support one another before a contact is marked usable.

An infographic detailing four quality checks to ensure accuracy when building B2B lead generation contact lists.
Quality IssueWarning SignFix
Homonym or pseudonym matchCommon name with no business metadataCompare bio, category and website; lower confidence
Shared email across profilesSame address on several handlesFlag and check each profile before outreach
Wrong entity contactedName matches but website differsUse username and profile URL as the key
Stale profileInactive account or an old collection dateRefresh and re-verify before sending
Bounced or invalid addressHard bounces in early sendsVerify and remove before the next wave

Keep the username and profile URL as the primary key, not the display name. Compare the bio, category, website and public contact fields for consistency. If one email appears on multiple profiles, flag it for review rather than assuming the accounts belong to the same person. A generic name or missing business details should lower confidence.

The CRM handoff is another place errors can spread. Deduplicating only by full name or handle can merge different people, or attach a contact to the wrong company. A stable key lets teams add new fields without overwriting a different person's record. The guide to contact data enrichment explains how to match fields while preserving the distinctions between records.

A public email is still only a contact detail, not permission to send. Before a list leaves the team, check that every row has a username, profile URL, source and collection date. Confirm that addresses have been verified, duplicates reviewed, and profile details checked for conflicts. Remove anyone on the suppression list, and use the contact only when the planned outreach is lawful.

Measuring Whether Scraped Leads Earn Their Place

A bigger export isn't a result. B2B contact data decays by roughly 2.1% a month, or about 22.5% a year, and automated extraction accuracy ranges from 85% to 95% depending on site complexity and field type, according to this analysis of verification and data freshness. A list you built in January is measurably weaker by summer, whether or not anyone touched it.

A lead freshness score handles that. Combine the collection date, recent profile activity, email verification status, business category, geography and audience source into one number per row, then set refresh intervals from it. Rows that score low get re-collected or dropped, not mailed.

StageWhat It MeasuresWhy It Matters
Expected yieldShare of profiles with a public emailAbout 10 percent in a broad job; tighter niches can differ
Delivery rateShare of emails that reach an inboxShows whether addresses were verified
Reply rateShare of contacts who respondShows whether the message fits the audience
Qualified conversationsReplies that match the buyer profileSeparates interest from fit
Meetings bookedConversations that reach a meetingShows incremental pipeline
Opt-outsContacts who ask to be removedSignals consent risk and feeds the suppression list

Here's the contrarian part. A 10% email yield from a tighter, better-fitting segment can outperform a much larger but staler list, because broad scraping amplifies low-fit contacts faster than it finds good ones. Volume doesn't create pipeline. Relevance does.

So test by cohort. Split the file by source, run the same message to each group, and compare outcomes per source rather than across the whole export. That's how you prove incremental pipeline instead of a bigger spreadsheet. For the wider picture on how these lists feed a demand engine, see this guide to B2B lead generation.

Run a Sample-First Test Before You Scale

A scraped list is a starting point, not permission to send. Test one defined audience first, check whether the contacts are fresh and relevant, then decide if the results justify a larger job.

A diagram illustrating the process of filtering scraped lead lists into a compliant, tested email outreach strategy.

Public availability and lawful, useful outreach are different checks. A practitioner framework for compliant Instagram lead generation emphasizes that a public username alone does not authorize enrichment, cold outreach, or indefinite storage. Relevance matters too: the linked framework notes that 73% of B2B buyers avoid sellers whose outreach is irrelevant to their business. A large export can still be a poor investment if its audience does not fit or its contact details have gone stale.

  1. Confirm the lawful basis, purpose, and retention period before collection. Treat anti-spam obligations and opt-out requests as part of the campaign, not cleanup after it.
  2. Pull a small cohort from one source and record its yield, delivery rate, and reply rate. This gives you a baseline before you spend more.
  3. Compare sources side by side and remove cohorts producing stale or low-fit contacts. Keep the audience source attached to your analysis so results remain interpretable.
  4. Scale only a source that performs, then set a refresh interval. Like produce, contact data loses usefulness with time, so a winning cohort still needs review.
  5. Ask for a sample file before buying anything larger. Check that its fields, username, name, email, phone, bio, follower count, category, country, and URL, match your CRM needs. Add your own source and collection-date columns.

HarvestMyData extracts publicly available contact information and does not access private accounts or protected content. Users agree to lawful use, anti-spam compliance, and honoring opt-out requests. A sample CSV is available to preview its structure before ordering.

Check compliance, test the cohort, then scale.

HarvestMyData extracts publicly listed emails and phone numbers from Instagram followers, following lists, hashtags, commenters, and likers. Each profile is delivered as a row in UTF-8 CSV or Excel, with username, full name, bio, follower count, category, and website alongside available contact details. Jobs include a live status page and are typically delivered within an hour. Visit HarvestMyData to review extraction rates by audience type and price a job for your segment.

We built HarvestMyData to handle all of this for you.

No proxies, no code, no account needed.

Service temporarily unavailable — we are not accepting new orders right now. Jobs already in progress are unaffected and will be delivered as normal.