Insurance Claim Scraper: extract, transform & load

Auto Insurance Claim - Custom software development

Weeks to 100% ROI
Insurance provider support growth
Workflow cost reduction


When a person files a vehicle insurance claim, providers turn to a company for scheduling a damage assessment. That company— a K-Optional Software client— specializes in getting every vehicle appraised so that the insurance provider can cut a check for the repair.

Working with insurance providers means adopting long-lived industry conventions; our client receives claims to a dedicated email server in a pseudo-consistent format.

Before working with K-Optional Software, our client had written a script in-house for reading this inbox, attempting to interpret the data, and exporting it into a CSV file. This approach proved brittle: slight variations in claim emails confused the script causing entire CSVs to drop. Worse yet, the evolution of email authentication— actually a welcome improvement for IT security— broke the script entirely.

So K-Optional received an urgent request to quickly stand-up a robust process that reliably processed mangled emails while the client depended on it. This was a textbook “fix the plane while it’s flying” scenario.

The challenge

  • Scraping email content is difficult. For one thing, the underlying SMTP protocol isn’t conducive to fetching data.
  • Plus, unlike, say, websites, email content may consist of varying data formats. Suffice it to say that emails are even less machine readable than HTML web-pages.
  • The predecessor script failed to handle mangled emails, a nontrivial proportion.
  • Finally, this engagement demanded urgency; our client’s pipeline slowed to a trickle without automatic claim-intake.

Our assessment

  1. After investigating the premiere SMTP libraries in a host of programming languages, we concluded that we should find a way to minimize the use of email protocols.
  2. We also determined that a flexible parser would enable our client to support additional providers easily— that huge realization ultimately allowed them to sign a new contract after delivery without any follow-up engagement.
  3. We cataloged this business problem as an Extract, transform, load (ETL) problem and applied industry knowledge in our solution. A 4-step processing pipeline borrowed components from state-of-the-art compiler design, an esoteric field in computer science.


K-Optional delivered a working MVP within a week. While the client employed this release, we solidified our full version 1 build in the span of two more weeks.

The client has noted an immediate elimination in dropped claims, a flexible and intuitive interface, and a boon to their business model.

Merely three weeks after delivery, the client seamlessly supported a new provider with no additional consulting necessary.