ESG reports that compare

Basic was an ESG standardisation pilot at IIT Patna. I joined as a product intern on the sustainability and ML side, over the winter of my first year.

The problem is easy to say. Companies publish ESG data, but each one reports it in its own format, so you can't line two companies up and compare them.

File the percentage too

An illustration of the problem, not figures from a real report. Two lines say the same thing in different units. The third can't be filed until you know what it is a percentage of.

11,000 reports

I started with market research across 11,000 corporate ESG reports.

That many reports is a data problem before it's a reading problem, so the next job was getting them into a shape a machine could work with.

The ingestion pipeline

I built the ingestion pipeline. Any pipeline for reports like these has the same jobs: get the text and tables out of the PDF, find the disclosure you care about, put it in one unit, and file it against a standard field.

Tap a stage.

The unit step looks like the dull one and is where the quiet errors live. A figure in kilotonnes filed as tonnes is off by a thousand, and it still looks like a number.

Two standards, two questions

The prototype maps a company's disclosures to two EU frameworks. CSRD is the reporting directive, and it sets what a company has to disclose about its impact. The ETS is the EU's carbon market, where emissions are measured and covered with allowances.

Tap a box.

CSRD asks what you disclosed. The ETS asks what you emitted and whether it is covered.

A single emissions figure can answer both, but only if it comes with the right scope, unit and year attached. Most of the pipeline's work was keeping those three with the number.

Twelve people, one prototype

I coordinated a 12-person team to a working prototype over that winter.

We split into three groups of four: research on the reports and the rules, data on the pipeline, and product on the mapping and the screens. The prototype let an analyst drop in a report and see each disclosure filed against its CSRD and ETS fields, next to the line in the report it came from.

What I'd do now

This was before I'd worked on LLM evals. Today I'd let a model do the finding step and put my effort into checking it. I'd label a few hundred disclosures by hand, and make the model quote the exact line each number came from. It's the rule I later gave the LLM judges at IBY.

A wrong emissions figure filed under the right field still looks correct, which is why it's the first failure I'd test for.

Disclosures filed correctly, checked by hand

• Coveragereports where the field was found at all
• Unit errorsfigures filed in the wrong unit
• Quote matchnumbers that match the line they came from
• Time per reportfrom the PDF in to the fields out