Activating AI tagging

1 Environment requirements
- 1.1 AWS configuration
  - 1.1.1 Bedrock service access grants
  - 1.1.2 Model access
2 Application parameters
3 Plugin PACKAGED_LLM
4 Configure your prompt
5 Test your prompt
6 Batch Test
7 Activate on the FO application
8 FO application integration
9 Results

Environment requirements

To enable AI tagging, additional configuration is required

AWS configuration
Application parameters

AWS configuration

Bedrock service access grants

The AWS account used by the application must have the following permission allowed: AmazonBedrockFullAccess

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "BedrockAll",
            "Effect": "Allow",
            "Action": [
                "bedrock:*"
            ],
            "Resource": "*"
        },
        {
            "Sid": "DescribeKey",
            "Effect": "Allow",
            "Action": [
                "kms:DescribeKey"
            ],
            "Resource": "arn:*:kms:*:::*"
        },
        {
            "Sid": "APIsWithAllResourceAccess",
            "Effect": "Allow",
            "Action": [
                "iam:ListRoles",
                "ec2:DescribeVpcs",
                "ec2:DescribeSubnets",
                "ec2:DescribeSecurityGroups"
            ],
            "Resource": "*"
        },
        {
            "Sid": "PassRoleToBedrock",
            "Effect": "Allow",
            "Action": [
                "iam:PassRole"
            ],
            "Resource": "arn:aws:iam::*:role/*AmazonBedrock*",
            "Condition": {
                "StringEquals": {
                    "iam:PassedToService": [
                        "bedrock.amazonaws.com"
                    ]
                }
            }
        }
    ]
}

Model access

Request access to Anthropic models.

Models might not be available in your zone. eu-west-3 is the preferred zone for requesting access.

Application parameters

As of 2024.3.0, the application configuration admin_config_family_mediacloud section must be filled (use dummy data)

Plugin PACKAGED_LLM

Using a Developper connected user, browse to plugin PACKAGED_LLM

Activate plugin, then from URLs tab, click on url.index (/_plugins/PACKAGED_LLM/page/config.jspz)

This will open a small UI allowing you to setup the plugin integration.

Configure your prompt

You need to configure the prompt that will be applied to describe images. The default value is “Describe this image”

Within the left menu, select Configure

You should adapt the prompt for instance:

Generate a concise caption including information such as the main subject, action or state, and the background or context. Limit to 2 sentences.
Do not introduce your text with "This image depicts" or any other introduction, go straight to the point.
Specify the position of characters or objects or their orientation relative to the image. Indicate the location, season, or time of day if known.
Only include objective information. Use the provided examples for inspiration:
Examples :
Silhouettes of workers in backlight on a construction site with a crane in the background
Close-up of a lab technician's hand wearing gloves and dropping a purple liquid into a vial with a pipette
Offshore wind farm of Rampion, United Kingdom
Stack of three macarons, one pink, one green, and one yellow, isolated on a gray background
Top view of a pile of watermelons with one cut watermelon on top
Tamnavulin whisky bottle seen from a 3/4 rear angle, isolated on a brown background
Close-up of a cheeseburger isolated on a white background
Airbus A310 in flight of Air Transat seen from a low angle
Overall view of the platform and railway tracks of the Bank metro station, London, United Kingdom
Partial view of the Roselend dam in its environment, France
High voltage pylons in a field at dusk

Most of claude3 options should be configurable from the application.

The [Image] box with dotted border represents the image that will be analysed.

If you bookmark a configuration, it will be available in the History, from which you will be able to load from.

The prompt is stored in appconfigs/poc_multimodal folder.

Test your prompt

You can use the Prompt entry from the menu to make some tests with a specific image, before configuring it.

Batch Test

You can also use Test entry to upload a set of images from the test screen to check your prompt among multiple images

Activate on the FO application

If you are managing the FO application configuration (json) on the server, you can define this configuration from the same UI

From the Portal integration menu entry,

Within Enable AI caption generation section, make sure Claude3 is selected.

You may adapt the target field on which you want to provide the tagging feature.

Click save.

This will adapt the FO configuration to enable LLM trigger.

The configuration is stored on the customer layer of the configuration:

FO application integration

If your client FO application configuration is not resolved by the server, you can activate the feature by adding to $.features

{
  "pocAiCaption": true,
  "pocAiCaption": "<FIELD_NAME>"
}