I have a vendor that sucks donkey balls. Their systems break often. An endpoint we rely on will start returning [] and take months to fix. They’ll change a data label in their backend and not notice that it flows into all of their filters and stuff.
I have some alerts when my consumers break, but I think I’d like something more direct. What’s the best way to monitor an external API?
I’m imagining some very basic ML that can pop up and tell me that something has changed, like there are more hosts or categories or whatever than usual, that a structure has gone blank or is missing, that some field has gone to 0 or null across the structure. Heck, that a field name has changed.
Is the best way to basically write tests for everything I can think of, and add more as things break, or is there a better tool? I see API monitoring tools but they are for calculating availability for your own APIs, not for enforcing someone else’s!
deleted by creator
Have a regression test runner that runs a suite of tests against the API. Have it run periodically, and be triggered by merge into main/RC branches using webhooks.
There’s a number of commercial offerings from this or you can roll your own without much bother.
Really depends on your infrastructure, but I’d set up some snapshot tests that just make calls to the APIs with known responses, and run that in a cronjob and have it alert you if it fails.
They haven’t so far broken the historical data, so I can’t directly compare a response to a known good, sadly.
So to sum it up:
- API breaks responses only for new data
- API does not provide any metadata, like versioning
- API doesn’t host it’s spec
- the problem isn’t that the API mutates, but that it starts returning garbage for new data, but not historical.
You’re out of luck. You can’t prevent it. You can’t foresee it, unless you know beforehand what you’ll call the API with and you can pre-flight it and detect it earlier.
Luckily the only thing I ever pass it are dates and host IDs. I can check a known input against the known response, but my problem so far hasn’t been them breaking their database but breaking the new data being added or the API itself.
Yeah, it’s a tough one.
Preflight it? If you ask external API every 6 hours about known range of host IDs with a date, then 1h before you need that information call the external API and check if it works or returns garbage? That way you can get some extra time to maybe react earlier to an incident? It honestly depends on the nature of your job and the qualities of your traffic, but generally speaking the problem you have is unfixable and the best you can hope for is early detection (if that matters for you).
If however you’re a pass-through API to the external one, eg. a different service calls your API with a hostID and the hostIDs are not a known finite pool, then you can forget about preflighting.
It is essentially an attitude and value problem. See Torvalds’ email titled “WE DON’T BREAK USER SPACE” and Rich Hickeys talk “Spec-ulation” on YouTube.
Consequently, the fix is to move to another vendor.
I’m just thrown by you saying you have a vendor that sucks donkey balls. If you only have one that sucks donkey balls, that seems unreal to me.
My group supports around 65 applications, and I’d find it a hell of a lot easier to list the vendors that don’t suck donkey balls.
I think there’s one. Maybe.
I highly recommend you change vendors, if possible. Not only will you not have to deal with this kind of issue anymore, you could also find vendors with dedicated support channels, status pages, and documentation detailing breaking changes (which, in an API, should only happen through versioning)
There is no sure-fire technical solution. So you name and shame, far and wide, until it affects their bottom line.
We’re both in a really niche market and the other vendors don’t seem much better!
Does API actually doing anything or it’s pass-through to data source? You could request data feed and build adequate API on your side
That’s a good thought. It’s a connection to one or more databases on their side, but it’s read only. I suppose that’s possible!
That’s if their API implementation itself is just bad and underlying DBs aren’t . If they or someone else with bad practices manages the DB you may be in worse situation than before. In general, to me, shared DB is bad because it is hard to not cut corners in such design and ensure that DB owner does not break contract for all consumers. This is basically why APIs created - to guarantee contracts and encapsulate change. But I digress. My point is that it will be your responsibility to ensure schema changes adopted to expected contract. If data is not normalized/structured, like say, it’s JSON then I would stay away.
I’ve no idea what’s underneath, really. I’ll ask and see what I learn.
Do they use openapi or swagger or something? If so you should be able to do something like use changedetect.io on their swaggerdocs page.
They generate a swaggger file for me on request with a lag time of weeks usually, but for only one of the APIs. The others are documented in emails basically. This is a B2B type of thing, they are not publicly available APIs.
Are any of their apis a GET that returns lists? I create a lot of automated api tests. You might be able to GET a list of users (or whatever) then pick a random 10 user_ids and query another api, say user_addresses and pass in each id one at a time and verify a proper result. You don’t have to verify the data itself, just that the values you care about are not empty and they key exists.
You can dynamically test a lot this way and if a key gets changed from ‘street’ to ‘street_address’ your failing tests should let you know.
Unfortunately on the main API I use of theirs, there’s an endpoint with a list of objects and their IDs, and those IDs are used everywhere else. The rest of the endpoints aren’t connected. I can’t walk e.g. school > students > student > grades or something
I made my career out of automated testing with a focus on apis. I’m not aware of any easy tool to do what you want. The easiest way to quick whip up basic api tests that I’ve found is python/pytest with requests. You can parameterize lots of inputs, run tests in parallel, easily add new endpoints as you go, benchmark the apis for response times, etc. It’ll take a lot of work in the beginning, then save you a lot of work in the end.
Now, AI will be able to make the process go faster. If you give it a sample input and output it can do 95% of a pytest in 10s. But beware that last 5%.
synthetics. the big question is how often to run the checks and how many you will need to make for your use cases.
A couple approaches are setting up a batch process on a frequent interval to call the API and run tests against the responses, another is to have the service consumer publish events to a message bus & monitor the events. It depends on things like do I own both the service and client or just client, can I make changes to the client or just add monitoring externally, and if I can run test requests without creating/updating/destroying data like a read only service, or if I need real requests to observe.
The main one I have issues with is a read only API. I guess I make it harder on myself from this perspective by not maintaining one big client, but lots of separate single-purpose tools.
Yeah then I would setup a call or set of calls on an interval to test the response on, and if a critical test fails send an alert, if there are less critical alerts maybe treat as warnings and send a report periodically. In either case I’d log and archive all of it so if they are bullshitting or violating contact SLAs I’ll have some data to reference.
They do have an API Accuracy SLA but it’s not defined anywhere so we do our best. They’ve only avoided penalties a few months out of the last several years!
Oof that is a rough one. If they are just absorbing the penalties it sounds like the penalties need to be increased to make it more financially necessary to change the incentive to actually do the work, but in the meantime I’d just collect and report on as much data as I could.
You might be losing more money using this one than changing for a more expensive but competent provider.
I have only came across one provider that we couldn’t replace and in that case we got them to export their data directly instead of wasting time using their awful API.
Luckily it’s not up to me, but I agree.
I’ve been complaining about the API for their main custom application, but they also have a ton of data in Salesforce and they screwed up when they set it up, so it’s not multitenanted or anything. I can’t have the API because I would be able to see and modify every customers’ data.
They’re awesome.
