ETags within the REST API end points

Owned by Joël DRIGO
Last updated: Dec 13, 2023
9 min read

2021.2.0 / 11.30

In order to reduce the volume transferred when the REST API is invoked, some end points provide an ETag. Their validity often depending on the information contained in the exposed objects, these are configurable.

Only DAM and RESTful services, with some exceptions, support ETags

Hard coded ETag

Some ETags are handled only in the code. They cannot be configured.

Catalogs

ETags for catalogs and headers services are hard coded.

API Documentation ETags

ETags for API Documentation (Swagger, ReDoc...) are hard coded.

Configurable ETags

Some ETags are configurable with structure tags or in Resource Ent Point JSon configuration

Tree or thesaurus, and others

The types of ETags are possibly different between trees and lists because the volumes are very different.

Structure tag

To configure an ETag on an object, simply add the appropriate tag on the object in the Technical BO, in the Configuration tab of the object's configuration.

Objects without rest_api_etag label do not integrate ETag management

JSon configuration

The ETag can be set to a configuration, by adding the structure described below to the JSon. This definition overrides the definition from the structure tag of the main object served by the resource.

The configuration can be an object for the property ETagDef

"ETagDef": { "tree": type for tree/thesaurus, "list": type for list, "default": type for all }

  • tree: string, type for tree or thesaurus

  • list: string, type for list

  • default: string, type for all not configured specifically for tree or list

Additionnaly, you can set:

  • ETagValue: the constant value for constant type

  • ETagCron: the configuration for periodic type

All these properties are optional: if the value cannot be determined, a default value will be chosen.

Default

To activate the ETag system, simply assign the rest_api_ETAG tag to the object structure configuration.

Without any other information, a default ETag type will be selected (see below).

Do not put several tags that may conflict: one of them will be chosen, without being able to determine in advance which one.

Disabling

To disable ETag use type none: REST_api_ETAG/NONE .

Constant ETag

Keyword: CONST

This type of ETag has a constant value, which does not vary between restarts in particular. It is suitable for objects whose content never changes (fixed list of values, like titles (Mr Mrs, Ms…) for instance).

Example: REST_API_ETAG/CONST
If ever the value should change later, it is possible to indicate a fixed value:

  • in the tag, following the définition. For example rest_api_etag/const/lang.v2 .

  • in the JSon, with the property ETagValue.

Benefits:

For objects that never change, the cache is very efficient since the data is never reloaded if the cache is fresh.
The value of this ETag not changing, it is very fast to calculate and to test.

Disadvantages:

Rare are the objects that never change.

Version ETag

Keyword: VERSION.

The version type takes a new value each time a new version of plugin is installed. It is adapted to objects whose content is enriched during engine upgrade, but will be reset also if plugin is upgrade.

The base is the plugin version, but this value can be overriden by a plugin parameter (it allows to have an ETag crossing the versions).

Example: REST_API_ETAG/VERSION

Setup

The setup of the version is done in the plugin parameter etagVersionValue. By default, if no value is entered (or blank), the version is that of the plugin. Special values can be used:

  • engine: selects the engine version

  • plugin: followed by a plugin name (that exists and is activated), selects the version of this specified plugin

  • value: followed by a value, sets this specified value

Any other value (except blank) sets this value.

Benefits:

The ETag does not change until the version changes. This is particularly suitable for lists of values that will be delivered during a production release.

Disadvantages:

The version must change if the content of the object changes.

Static ETag

Keyword: STATIC.

The static type takes a new value each time the plugin is restarted. It is adapted to objects whose content is enriched during production upgrade (requiring a restart), or a technical restart (including a flush restart).

Example: REST_API_ETAG/STATIC

Benefits:

The ETag does not change until you restart the WXM_RESTAPI plugin. A restart allows to flush the cache.

Disadvantages:

Any restart changes the ETag. In case of a technical restart, the caches will have to refresh, even the answer has not changed.

Periodic ETag

Keyword: CRON.

This type allows to have an ETag that changes regularly, either periodically or at specific times.

It is suitable for objects that change regularly or even often, but it is acceptable to have visibility of these changes with a delay.

The periodicity is indicated after the definition. For example REST_API_ETAG/CRON/@dayly (every day at midnight).

Benefits:

This type was created to compensate for the lack of dynamic ETag in cluster mode.

Disadvantages:

Cached data may not be up to date. Small perverse effect: one service can have a full cache and not the other and therefore one and the other can present inconsistent results. For example, a new keyword is not seen by one service and is seen by the other.

Periodicity syntax

time delay in seconds

An integer defines a time delay of this number in seconds.

time delay (from the plugin startup time)

With the expression @timeout(n,unit), where n is a number (>0) and unit a time unit. The unit can be omitted: the default unit is the minute. The @timeout() value is a shortcut for each minute.

The possible values for unit are:

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

  • WEEKS

  • MONTHS

  • YEARS (the value for n can only be one for this case)

period

The following expressions allow you to change the value at regular times. It is possible to indicate a multiplier in brackets. For example, @min(5) for every five minutes.

  • @yearly
    Every year (January 1st at midnight). Only the value 1 is possible with this.

  • @monthly
    Every month (the first day of the month at midnight).

  • @weekly
    Every week (the first day of the week à midnight).

  • @dayly
    Every day (at midnight)

  • @midnight
    Every day (at midnight)

  • @hourly
    Every hour

  • @min
    Every minute

  • @sec
    Every second

  • @sun
    Every sunday. Only the value 1 is possible with this.

  • @mon
    Every monday. Only the value 1 is possible with this.

  • @tue
    Every tuesday. Only the value 1 is possible with this.

  • @wed
    Every wednesday. Only the value 1 is possible with this.

  • @thu
    Every thursday. Only the value 1 is possible with this.

  • @fri
    Every friday. Only the value 1 is possible with this.

  • @sat
    Every saturday. Only the value 1 is possible with this.

For examples:

  • @hourly(2)

  • @min(15)

  • @hourly()

  • @min

general syntax

A cron expression (type Quartz) (year is not supported)

A cron expressions consists of 6 mandatory fields separated by space.

These are:

Field

 

Allowable values

 

Special Characters

Field

 

Allowable values

 

Special Characters

Seconds (may be omitted)

 

0-59

 

, - * /

Minutes

 

0-59

 

, - * /

Hours

 

0-23

 

, - * /

Day of month

 

1-31

 

, - * ? / L W

Month

 

1-12 or JAN-DEC (note: english abbreviations)

 

, - * /

Day of week

 

1-7 or MON-SUN (note: english abbreviations)

 

, - * ? / L #

'*' Can be used in all fields and means 'for all values'. E.g. "*" in minutes, means 'for all minutes'

'?' Can be used in Day-of-month and Day-of-week fields. Used to signify 'no special value'. It is used when one want to specify something for one of those two fields, but not the other.

'-' Used to specify a time interval. E.g. "10-12" in Hours field means 'for hours 10, 11 and 12'

',' Used to specify multiple values for a field. E.g. "MON,WED,FRI" in Day-of-week field means "for monday, wednesday and friday"

'/' Used to specify increments. E.g. "0/15" in Seconds field means "for seconds 0, 15, 30, ad 45". And "5/15" in seconds field means "for seconds 5, 20, 35, and 50". If '*' s specified before '/' it is the same as saying it starts at 0. For every field there's a list of values that can be turned on or off. For Seconds and Minutes these range from 0-59. For Hours from 0 to 23, For Day-of-month it's 1 to 31, For Months 1 to 12. "/" character helsp turn some of these values back on. Thus "7/6" in Months field specify just Month 7. It doesn't turn on every 6 month following, since cron fields never roll over

'L' Can be used on Day-of-month and Day-of-week fields. It signifies last day of the set of allowed values. In Day-of-month field it's the last day of the month (e.g.. 31 jan, 28 feb (29 in leap years), 31 march, etc.). In Day-of-week field it's Sunday. If there's a prefix, this will be subtracted (5L in Day-of-month means 5 days before last day of Month: 26 jan, 23 feb, etc.)

'W' Can be specified in Day-of-Month field. It specifies closest weekday (monday-friday). Holidays are not accounted for. "15W" in Day-of-Month field means 'closest weekday to 15 i in given month'. If the 15th is a Saturday, it gives Friday. If 15th is a Sunday, the it gives following Monday.

'#' Can be used in Day-of-Week field. For example: "5#3" means 'third friday in month' (day 5 = friday, #3 - the third). If the day does not exist (e.g. "5#5" - 5th friday of month) and there aren't 5 fridays in the month, then it won't match until the next month with 5 fridays.

Data up-to-date aware ETag

Keyword: DYN.

This type allows to have an ETag that changes when the content of the object is modified (during creation, modification and deletion). It is adapted to objects that change very often, and for which we want a real time vision.

This type is not yet available on cluster plateform. Its availability is not determinable yet. see https://crossmedia.atlassian.net/browse/WXM-10298 This feature will be available in release 2024.1

 

Benefits:

Cached data is always up to date.

Disadvantages:

The ETag can change even if the response does not change: the last instance modification or deletion is considered, but the instance concerned may not even be in the response. If the majority of the instances that are loaded are never modified, but others are modified or added or deleted, it is very inefficient.

Disable the management components of this type of ETag on clustered environments. Indeed, the start of the component necessarily passes in error, and this error is raised in the global status of the WXM_RESTAPI plugin, thus in that of the server, which prevents from following the SLA of the platform.

To disable dynamic mode support, set the dynETag parameter to false.

See https://crossmedia.atlassian.net/wiki/spaces/WD/pages/11764307

Other solutions under study (more or less)

Hash based ETag

Comparing response hashes is one of the most effective ways to handle ETag, but it has several problems:
either you have to run the service twice, once to compute the hash, once to serialize the response
or you have to store the answer the first time, which in memory consumes a lot of memory, and on disk consumes disk space and increases the i/o.
Even if the goal is to save time on the transport, the execution time is not necessarily negligible.
The debate on the (expensive) implementation has resulted for the moment in a decision not to implement.

Instance ETag

This type of ETag could be used in the context of instance retrieval by identifier, if the request and response conditions do not involve any dependency on other instances. Its value is based on the modified property of the instance and therefore is supposed to guarantee the most up-to-date ETag with respect to the instance. However, as soon as there are dependencies on other objects (that we want to export data from other objects), the modified property of the instance is no longer valid, and it is difficult to determine the value of the ETag without doing a lot of tests and calculations, and without eliminating many cases. It appears then that there are so few cases where it works, that it is not planned for the moment to implement it.

2021.0.3 For the moment, we have chosen to apply the list strategy for direct access to an instance, which is very inefficient, but still better than doing nothing.

ETag flush

An API is planned that will allow to make an update on ETag on demand, which will have the effect of flushing the cache (until the next restart).

Context override, context customization

It is difficult to establish the necessary granularity on the context, because it depends on the executed code (like security rules). We therefore consider allowing the context to be modified in order to have a more accurate granularity. For example, if the security depends only on a department, or on the role for example.

Cache control

Private and public caches are managed. The max-age is managed according to the rest_api_etag_maxage tag or according to the default system of the ETag type, only in production.

Max-Age control

ETag max-age is handled automatically. You can allow (or disallow) client to set the max-age (independently of etags) with plugin parameter etagMaxAgeControl.

Default ETag

You can set a default ETag type in the plugin parameterdefaultETag. This definition will be used:

  • if no specific definition is set in a tag or a JSon configuration

  • if the specific definition is erroneous

The syntax is the same as for structure tag.

DAM & RESTful end points ETags

The following services support ETag:

  • single object search and paginated list

  • tree/thesaurus

  • catalogs and headers

  • 2021.0.3 access by identifier (non-tree),

The following services don’t supports ETag yet:

  • access by identifier (non-tree), before 2021.0.3

Contextual ETag

Depending on the case, etags can be contextual. This is handled automatically.

Supported contexts:

  • surfer

  • request language (locale), that is surfer by default, or value from request parameter lang if present

Configuration dependencies

The dynamic type (DYN) is aware of the different dependencies to other objects than the main object (dependencies created by child fields, childmulti, etc). But it is not possible to choose a different type for each object: the ETag listens for changes for all objects.

In developpement mode, the ETag are aware of configuration and structures changes, which can make them less efficient than they can be in production.

Monitoring and debugging of ETags

It is possible to see the different ETags, their value, their last modification date, where they have been defined, etc.
Go to the API administration, then to the ETags section.

Starting guide