Data Quality Analyst

(christian dabnor) #1

There’s been some discussion here about bringing someone in to maintain the quality of data we hold. A lot of data maintenance is automated, but we would like to have someone to oversee this. How do other medium sized companies deal with this? Would you bring in a specialist to oversee data quality and any related automation, in partnership with the infrastructure team, or do people see it as a dual role with “day to day” QA?

Thank you in advance for any assistance.

(Tim Martin) #2

Can you expand anymore on what kind of data would be involved and what are typical kinds of problems you currently find with it?

(Robert) #3

This was one of my major roles in a previous job. I worked for one of the sectorial utility regulators in the UK, and we collected a lot of data from external companies. Data quality was a chain, starting with the companies and arrangements for validating and confirming data collection methodologies, then the development, testing and deployment of the data collection tool (this was where I got into software testing), analysis and querying of the data return from the companies, data wrangling with a managed process to edit or amend data, and finally reflecting data changes in updates to the requirements for data collection and the spec for the next year’s data capture system.

This was mainly a manual system, because we were dealing with manual data collection in the real world (and because we are talking about the middle 1990s…). I ended up in this role because my managers felt I had the necessary attention to detail and similar skills that suited the role. It wasn’t deliberately created as a “dual role” with either testing or “day-to-day QA” (such as proof-reading reports, ensuring data quality and maintaining data validation audit trails from raw data to capture systems to databases to data extraction to reports; my strength in particular was reconciling “text to tables” - making certain that publicly-quoted headline numbers were consistent between report texts, press notices, keynote speeches and presentations, and the actual published data tables in the body of reports); but the role evolved into just that many-headed beast.

I wouldn’t say I was a “specialist” in this sort of thing when I started; but fifteen years later I certainly was! (Though that had its own set of pluses and minuses…)

(christian dabnor) #4

We provide software for the auto finance trade. Data we hold includes, but is not restricted to, vehicle data (both new and used vehicles) and usage of the software (things like how many calculations are being made, how many vehicles are reserved, where people join and leave the journey etc). I wouldn’t say we have ‘problems’ per se. but want to improve the way we handle our data.

@robertday Thank you.

(Robert) #5

You’re welcome.

If you’d like any more information about the sort of QA/data management regime I used to work in, let me know. I lived it for the thick end of twenty years, so although I’ve not worked in that role since 2010, it’s still very much alive for me!