How to Compare & Sync JSON Data from Two APIs?

🙋 Questions
,
1

Hi everyone,

I’m working on a data synchronization project and could use some guidance. The goal is to automate the process of ensuring that the data I download from a subscription-based API matches what’s available on another API. Here’s a breakdown of my workflow:

  1. As a customer, I gain access to a subscription and download data in JSON format.
  2. I then must compare this downloaded data with what is available on another API to ensure they match.
  3. I’d like to automate this process, ideally handling differences efficiently.

I’m looking for best practices, tools, or libraries that could help with:

  • Automating the data download and comparison process.
  • Handling discrepancies (e.g., missing/extra records, format inconsistencies).
  • Efficiently managing large JSON files if needed.

Would love to hear from anyone who has tackled something similar! Thanks in advance for your help.

4 Likes
2

Hi,

My understanding of this:

  • there’s two APIs queried
  • each API returns a JSON response;
  • the JSON responses follow same or different schemas?
  • the ids of objects inside one JSON might be the same/different to the corresponding one in the other JSON?
  • the data is dynamic or static-mocked?
  • the scope is to automate one comparison or any comparison of any instance at any time of the download; with a focus on finding if there is a difference

The easiest way I could think of is picking up a library managing APIs in your preferred tool/programming language.
Then compare in memory the JSON objects, using one of the libraries for it.
In python for example, I’d use requests and deepdiff, jsondiff.

3

When you say download, do you need to actually down the json. With the ideas of APIs and JSONs. I’m assuming if they’re restful APIs you could do something like this… This way it’s automated and easy to run

(codes a super SUPER rough idea)

var client = new RestClient();
var request1 = new RestRequest("EndPoint", Method.Post); //or whatever your http method is
request1.AddJsonBody() //this is used to send the data to the api to retrieve the json
var response1 = client.ExecutePostAsync(request);
// do the same thing for request2/response2 targeting the other api
Assert.Equal(response1, response2) //I'm sure there's much better ways of asserting here

EDIT:
by brining in something like Fluent Assertions or Shoudly you can do your assert like this. source

response1.Should().BeEquivalentTo(response2);