Why we need to publish without access restrictions

Problem

Some publishers publish IATI data on their own website and then put security on the website to try and protect themselves from bad bots (Cloudflare is a popular product that does this). But this often has the side effect of stopping all the users who want to use the data from getting the data.

In this case, it may look like the data is available. It is on the publisher's website and when you open it in a web browser you can see the data. However, the data cannot be used in various tools such as the Dashboard, Validator, Datastore, and more.

Allow-listing unfortunately won’t work

Many of these security products allow publishers to make exceptions for bots that are known and trusted. This is referred to as allow listing (an older term for this is white listing) because by default all bots are blocked but publishers then allow certain known bots access.

The security products will require technical information to allow bot access. This includes information on what the bots declare themselves as (their user agent) and where the bots are located (their IP address).

Often when this problem arises, we are asked to send this information to publishers and they will add exceptions for our bots.

However there are several problems with this approach and ultimately it is not workable.

Firstly, collecting this information and making sure it is kept up to date is not an easy task. It is made more difficult because there are so many products that need access to the data and we need to do this for every single product.

• Unified Platform; including the datastore and validator
• Dashboard
• Publisher Stats
• CDFD
• D-Portal
• Registry
• And more…

Every product may have multiple bits of information and the information may change at any point in the future. It will therefore take a lot of staff time at IATI to collect and maintain the information, a lot of staff time at the publisher to apply it, and a lot of staff time from everyone to debug any issues that may arise.

But most importantly, in an open data ecosystem it is always possible for other people to make their own products to collect data and process it (for example, the Aid Transparency Index from Publish What You Fund). It’s important that these other products can also access publisher’s data. If we take an allow list approach we cannot provide the needed information for other people’s products.

So if people only allowed the products we provided them information for and blocked the rest, we would then have a situation where some products can see the data but other products cannot. This leads to problems for the completeness of IATI data and the analysis that is done using it (for example, if the Aid Transparency Index can not see a publisher's data they will simply not be included in that year's report).

The solution

The only solution that works is that any published files must have security restrictions that block bots removed and be available for all.

In security products like Cloudflare, it is often possible to remove restrictions for one area of a website only - so this does not mean removing these products entirely. Alternatively, files could be moved to a new website on a subdomain with no security issues.

This is a solution that is practical; as it saves staff time for both the publisher and IATI and removes the likelihood of mistakes.

This is a solution that is also fitting for the IATI project; it’s important that publishers make sure the data is available for anyone who wants it without restrictions.

It’s important to note however, that this does not mean that a website must have no security at all. A bot that requests the same file thousands of times per second could cause problems and there is no legitimate use for this; so it’s acceptable to block malicious actors. But websites must not block well-behaved bots by default.