Note: This package is only relevant if you need to create new AI providers or extend existing functionality. For simple integration of AI features, use the @imgly/plugin-ai-apps-web package instead.

This package provides the foundation for creating AI generation plugins for CreativeEditor SDK. It offers a standardized interface for implementing AI generation providers that can create images, videos, audio, or text assets. The package includes utilities for handling:

Provider registration and initialization
User interface generation
Global action registry for quick actions and plugin actions
Type-safe quick action definitions
Cross-plugin action support

Getting Started

Installation

npm install @imgly/plugin-ai-generation-web

Creating a Custom Provider

The core of this package is the Provider interface which defines the contract for AI generation providers. Here's how to implement a basic provider:

import {
    Provider,
    ImageOutput,
    initializeProvider,
    loggingMiddleware,
    CommonProviderConfiguration
} from '@imgly/plugin-ai-generation-web';

// Define your provider configuration interface
interface MyProviderConfiguration 
    extends CommonProviderConfiguration<MyInputType, ImageOutput> {
    // Add any provider-specific configuration here
    baseURL?: string;
}

// Create a provider factory function
function createMyImageProvider(config: MyProviderConfiguration): Provider<'image', MyInputType, ImageOutput> {
  return {
    // Unique identifier for this provider
    id: 'my-image-provider',

    // Define output asset type, other options are 'video', 'audio', 'text'
    kind: 'image',

    // Initialize the provider
    initialize: async ({ engine, cesdk }) => {
        // Setup APIs, register further components, etc.
        myAIApi.configure({ 
            apiKey: 'YOUR_API_KEY',
            headers: config.headers  // Use custom headers if provided
        });
    },

    // Define input panel and UI components
    input: {
        // Define how the input panel is rendered
        panel: {
            // Option 1: Schema-based UI (using OpenAPI)
            type: 'schema',
            document: myApiSchema,
            inputReference: '#/components/schemas/GenerationInput',
            getBlockInput: async (input) => ({
                image: { width: 1024, height: 1024 }
            })
        },

        // Quick actions supported by this provider
        quickActions: {
            supported: {
                'ly.img.editImage': {
                    mapInput: (input) => ({
                        prompt: input.prompt,
                        image_url: input.uri
                    })
                },
                'ly.img.styleTransfer': {
                    mapInput: (input) => ({
                        prompt: input.style,
                        image_url: input.uri
                    })
                }
            }
        }
    },

    // Define output generation behavior
    output: {
        // Allow cancellation
        abortable: true,

        // Store generated assets, options are:
        // - false: No history
        // - '@imgly/local': In-memory storage (lost on refresh)
        // - '@imgly/indexedDB': Browser IndexedDB storage
        // - any other string: Handled as a custom asset source ID
        history: '@imgly/indexedDB',

        // Add middleware for pre/post-processing of the generation
        middleware: [loggingMiddleware()],

        // Configure success/error notifications
        notification: {
            success: {
                show: true,
                message: 'Generation successful!'
            }
        },

        // Core generation function
        generate: async (input, { abortSignal, engine }) => {
            // Call your AI API and return result
            const response = await myAIApi.generateImage(input, {
                headers: config.headers  // Pass custom headers to API
            });

            return {
                kind: 'image',
                url: response.imageUrl
            };
        }
    }
  };
}

// Usage example
const myImageProvider = createMyImageProvider({
    proxyUrl: 'http://your-proxy-server.com/api/proxy',
    headers: {
        'x-client-version': '1.0.0',
        'x-request-source': 'cesdk-plugin'
    },
    debug: false,
    middleware: [loggingMiddleware()],
    baseURL: 'https://assets.example.com'
});

Action Registry

The package includes a global ActionRegistry for managing quick actions and plugin actions. To register a new action:

import { ActionRegistry } from '@imgly/plugin-ai-generation-web';

// Get the global registry instance
const registry = ActionRegistry.get();

// Register a quick action
const unregister = registry.register({
    id: 'my-quick-action',
    type: 'quick',
    kind: 'image',
    label: 'My Quick Action',
    enable: true,
    render: (context) => {
        // Render the quick action UI
        context.builder.Button('my-button', {
            label: 'Generate',
            onClick: async () => {
                await context.generate({ prompt: 'Hello world' });
            }
        });
    }
});

Provider Interface

The Provider interface is generic and type-safe, supporting four output kinds:

// K: Output kind ('image', 'video', 'audio', 'text')
// I: Input type specific to your provider, i.e. what does the generate function need
// O: Output type (ImageOutput, VideoOutput, AudioOutput, TextOutput)
// C: Chunk type for streaming (optional, defaults to O)
interface Provider<K extends OutputKind, I, O extends Output, C = O> { ... }

Common Provider Configuration

All providers should extend the CommonProviderConfiguration interface, which provides standardized configuration options:

interface CommonProviderConfiguration<I, O extends Output> {
    // The proxy URL to use for the provider
    proxyUrl: string;
    
    // Enable debug mode for additional logging
    debug?: boolean;
    
    // Middleware for request/response processing
    middleware?: Middleware<I, O>[];
    
    // Custom headers to include in all API requests
    headers?: Record<string, string>;
    
    // Override provider's default history asset source
    history?: false | '@imgly/local' | '@imgly/indexedDB' | (string & {});
    
    // Configure supported quick actions
    supportedQuickActions?: {
        [quickActionId: string]: Partial<QuickActionSupport<I>> | false | null;
    };
}

Extended Configuration Options

History Configuration

The history field allows you to override the provider's default history storage behavior:

const provider = createMyImageProvider({
    proxyUrl: 'https://proxy.example.com',
    
    // Disable history storage entirely
    history: false,
    
    // Or use temporary local storage (not persistent)
    // history: '@imgly/local',
    
    // Or use persistent browser storage (default for most providers)
    // history: '@imgly/indexedDB',
    
    // Or use a custom asset source ID
    // history: 'my-custom-asset-source'
});

Available Options:

false: Disable history storage entirely
'@imgly/local': Use temporary local storage (not persistent across sessions)
'@imgly/indexedDB': Use browser IndexedDB storage (persistent across sessions)
string: Use your own custom asset source ID

Quick Actions Configuration

The supportedQuickActions field allows you to customize which quick actions are supported and how they behave:

const provider = createMyImageProvider({
    proxyUrl: 'https://proxy.example.com',
    
    // Configure quick actions
    supportedQuickActions: {
        // Remove an unwanted quick action
        'ly.img.editImage': false,
        
        // Override with custom mapping
        'ly.img.swapBackground': {
            mapInput: (input) => ({
                prompt: input.prompt,
                image_url: input.uri,
                strength: 0.9, // Custom strength
                style: 'REALISTIC' // Force realistic style
            })
        },
        
        // Add support for new quick action
        'ly.img.customAction': {
            mapInput: (input) => ({
                prompt: `Custom prefix: ${input.text}`,
                image_url: input.imageUrl
            })
        }
    }
});

Configuration Values:

false or null: Remove the quick action entirely
true: Keep the provider's default implementation
Object with mapInput: Override the quick action with custom input mapping
Object with other properties: Override with custom configuration

Headers Configuration

The headers property allows you to include custom HTTP headers in all API requests made by your provider. This is useful for:

Adding custom client identification headers
Including version information
Passing through metadata required by your API
Adding correlation IDs for request tracing

Implementation Note: When implementing your provider's generate function, ensure you merge the custom headers with any required headers for your API:

// In your generate function
const response = await fetch(apiUrl, {
    method: 'POST',
    headers: {
        'Content-Type': 'application/json',
        'Authorization': `Bearer ${apiKey}`,
        ...config.headers  // Spread custom headers
    },
    body: JSON.stringify(requestData)
});

Key Provider Options

id: Unique identifier for your provider
kind: Type of asset generated ('image', 'video', 'audio', 'text')
name: Optional human-readable name
initialize: Setup function called when the provider is loaded
input: Configuration for input UI and parameters
output: Configuration for generation and result handling

Provider Output Options

The output property has several important options:

generate: Main function that performs the actual generation
history: Asset storage strategy ('false', '@imgly/local', '@imgly/indexedDB', or custom ID)
abortable: Whether generation can be cancelled by the user
middleware: Array of middleware functions for pre/post-processing
notification: Success and error notification configuration
generationHintText: Text to display below the generation button

Notification Configuration

The notification system allows fine-grained control over success and error messages:

notification: {
  success: {
    // Control whether to show notifications (can be dynamic)
    show: true, // or (context) => shouldShow(context)

    // Message text or i18n key (can be dynamic)
    message: 'Generation successful!', // or (context) => getMessage(context)

    // Optional action button
    action: {
      label: 'View', // or (context) => getLabel(context)
      onClick: (context) => { /* handle click */ }
    },

    // How long to show the notification
    duration: 'short' // or 'medium', 'long', 'infinite'
  },

  error: {
    // Similar options for error notifications
    show: true,
    message: 'Generation failed', // or (context) => getErrorMessage(context)
    // ...
  }
}

Streaming Generation

The generate function can return a simple output object or an AsyncGenerator for streaming results:

// Simple response
generate: async (input, options) => {
  const result = await api.generateImage(input);
  return { kind: 'image', url: result.url };
}

// Streaming response (currently only supported for text)
generate: async function* (input, options) {
  const stream = api.streamGenerationResult(input);

  let inferredText: string = '';
  // Yield interim results
  for await (const chunk of stream) {
    inferredText += chunk;
    yield { kind: 'text', text: inferredText };
  }

  // Return final result
  return { kind: 'text', text: inferredText };
}

Generation Hint Text

The generationHintText property allows providers to display helpful information below the generation button:

generationHintText: "Generation may take up to a minute. You can close this panel and will be notified when ready."

Input Panel Types

The package supports two approaches for creating input panels:

1. Schema-based Input Panels

The schema type uses OpenAPI specification to declaratively define your input form.

input: {
  panel: {
    type: 'schema',
    // Complete OpenAPI v3 document describing your inputs
    document: myOpenAPISchema,
    // JSON pointer to your input schema within the document
    inputReference: '#/components/schemas/GenerationInput',
    // Optional property to control display order
    orderExtensionKeyword: 'x-order-properties',
    // Function that converts input to block parameters
    getBlockInput: async (input) => ({
      image: { width: 1024, height: 1024 }
    }),
    // Optional custom renderers for specific properties found in the schema
    renderCustomProperty: {
      // This is a custom renderer for a fictional `imageUrl` property
      imageUrl: (context, property) => {
        const valueState = context.state('imageUrl', '');
        context.builder.TextInput('imageUrl', {
          inputLabel: 'Image URL',
          ...valueState
        });

        // Return a function that returns the value for this property
        return () => { id: property.id, type: 'string', value: valueState.value };
      }
    }
  }
}

OpenAPI Schema Example

{
    "openapi": "3.0.0",
    "components": {
        "schemas": {
            "GenerationInput": {
                "type": "object",
                "required": ["prompt"],
                "properties": {
                    "prompt": {
                        "type": "string",
                        "title": "Prompt",
                        "description": "Describe what you want to generate",
                        "x-imgly-builder": {
                            "component": "TextArea"
                        }
                    },
                    "width": {
                        "type": "integer",
                        "title": "Width",
                        "default": 1024,
                        "enum": [512, 1024, 2048],
                        "x-imgly-builder": {
                            "component": "Select"
                        }
                    }
                },
                "x-order-properties": ["prompt", "width"]
            }
        }
    }
}

Benefits of Schema-based Input

Built-in validation based on schema constraints
AI providers like fal.ai provide schemas for their models
Automatic UI component generation based on property types
Extensions like x-imgly-builder to specify component types
Property ordering via orderExtensionKeyword
Customizable property rendering with renderCustomProperty

2. Custom Input Panels

The custom type gives you complete control over UI components. For more details on how to build custom panels and see all available builder components, refer to the Create a Custom Panel guide.

input: {
  panel: {
    type: 'custom',
    render: (context, options) => {
      // Use the builder pattern to create UI components
      const promptState = context.state('prompt', '');
      context.builder.TextArea('prompt', {
        inputLabel: 'Prompt',
        ...promptState
      });

      // Set up width selection
      const widthState = context.state('width', 1024);
      context.builder.Select('width', {
        inputLabel: 'Width',
        options: [
          { value: 512, label: '512px' },
          { value: 1024, label: '1024px' },
          { value: 2048, label: '2048px' }
        ],
        ...widthState
      });

      // Return functions to get input values and block parameters
      return {
        // The input for the generate function
        getInput: () => ({
          prompt: promptState.value,
          width: widthState.value
        }),
        // The input for the block creation
        getBlockInput: () => ({
          image: {
            width: widthState.value,
            height: widthState.value,
            label: `AI Image: ${promptState.value.substring(0, 20)}...`
          }
        })
      };
    }
  }
}

Benefits of Custom Input Panels

Complete control over UI components and layout
Complex logic between fields (dependencies, conditionals)
Dynamic UI that changes based on user interactions

Panel User Flow Options

Both panel types accept additional configuration:

panel: {
  type: 'schema', // or 'custom'
  // ...panel type specific options

  // Control the generation flow
  userFlow: 'placeholder', // or 'generation-only' (default)

  // Include/exclude history library from panel
  includeHistoryLibrary: true // (default)
}

userFlow:
- placeholder: Creates a block as a placeholder with loading state when generation starts
- generation-only: Only triggers generation without creating a placeholder
includeHistoryLibrary: Controls whether the history library is shown in the panel

The `getBlockInput` Function

The getBlockInput function is crucial for both panel types. It converts your input into the parameters needed to create a block in CreativeEditor SDK.

What It Does

Defines dimensions, duration, and appearance of asset blocks
Creates placeholders before generation completes
Maps your AI provider's inputs to standardized block parameters

Required Return Values by Output Kind

Each output kind requires specific parameters:

For Images

getBlockInput: async (input) => ({
    image: {
        width: 1024, // Required - Width in pixels
        height: 1024, // Required - Height in pixels
        label: 'My Image' // Optional - Display name
    }
});

For Videos

getBlockInput: async (input) => ({
    video: {
        width: 1280, // Required - Width in pixels
        height: 720, // Required - Height in pixels
        duration: 10, // Required - Duration in seconds
        label: 'My Video' // Optional - Display name
    }
});

For Audio

getBlockInput: async (input) => ({
    audio: {
        duration: 30, // Optional - Duration in seconds
        thumbnailUrl: 'path/to/img.jpg', // Optional - URL for thumbnail
        label: 'My Audio' // Optional - Display name
    }
});

For Text

getBlockInput: async (input) => ({
    text: {
        length: 250, // Required - Approximate character length
        label: 'My Text' // Optional - Display name
    }
});

Quick Actions

Quick Actions provide context-aware AI generation capabilities directly in CreativeEditor SDK's canvas menu. Unlike panels (which appear in the side panel), quick actions appear when users select elements on the canvas.

Available Quick Action IDs

Here are all the quick action IDs that can be used in the supported field of your provider configuration:

Image Quick Actions

ly.img.artistTransfer: Transform image in the style of famous artists
- Input: { artist: string, uri: string }
ly.img.combineImages: Combine multiple images with instructions
- Input: { prompt: string, uris: string[], exportFromBlockIds: number[] }
ly.img.createVariant: Create a variation of the image
- Input: { prompt: string, uri: string }
ly.img.editImage: Change image based on description
- Input: { prompt: string, uri: string }
ly.img.remixPage: Convert the page into a single image
- Input: { prompt: string, uri: string }
ly.img.remixPageWithPrompt: Remix the page with custom instructions
- Input: { prompt: string, uri: string }
ly.img.styleTransfer: Transform image into different art styles
- Input: { style: string, uri: string }
ly.img.swapBackground: Change the background of the image
- Input: { prompt: string, uri: string }
ly.img.gpt-image-1.changeStyleLibrary: Apply different art styles (GPT-specific)
- Input: { prompt: string, uri: string }

Text Quick Actions

ly.img.changeTextTo: Change text to a different format or style
- Input: { prompt: string, customPrompt: string }
ly.img.changeTone: Change the tone of the text
- Input: { prompt: string, type: string }
ly.img.fix: Fix spelling and grammar
- Input: { prompt: string }
ly.img.improve: Improve writing quality
- Input: { prompt: string }
ly.img.longer: Make text longer
- Input: { prompt: string }
ly.img.shorter: Make text shorter
- Input: { prompt: string }
ly.img.translate: Translate text to different languages
- Input: { prompt: string, language: string }

Video Quick Actions

ly.img.createVideo: Opens the image2video generation panel with the current image
- Input: { uri: string }

Provider Quick Action Support

Providers declare which quick actions they support and how to map quick action inputs to provider inputs:

const myProvider = {
    // ... other provider config
    input: {
        // ... panel config
        quickActions: {
            supported: {
                'ly.img.editImage': {
                    mapInput: (quickActionInput) => ({
                        prompt: quickActionInput.prompt,
                        image_url: quickActionInput.uri
                    })
                },
                'ly.img.styleTransfer': {
                    mapInput: (quickActionInput) => ({
                        style: quickActionInput.style,
                        image_url: quickActionInput.uri
                    })
                }
            }
        }
    }
};

Quick Action Expanded View

Quick actions can have two rendering modes:

Collapsed View: Shows as a simple button in the quick action menu alongside other actions
Expanded View: Takes over the entire menu space, hiding other actions while the user interacts with this specific action

The expanded view is useful for quick actions that need user input (like text prompts). When a quick action is expanded, the complete menu is replaced with the expanded interface, and other menu items are not shown until the user either completes the action or cancels back to the collapsed view.

render: ({ builder, isExpanded, toggleExpand }) => {
    if (isExpanded) {
        // Expanded view - takes over the entire menu
        builder.TextArea('prompt', { /* input fields */ });
        builder.ButtonRow('actions', { /* confirm/cancel buttons */ });
    } else {
        // Collapsed view - simple button alongside other actions
        builder.Button('expand', { 
            label: 'Edit Image...',
            onClick: toggleExpand 
        });
    }
}

Customizing Labels and Text

You can customize all labels and text in the AI generation interface using the translation system. This allows you to provide better labels for your users in any language.

Translation Priority

The system checks for translations in this order (highest to lowest priority):

Provider & Kind-specific: ly.img.plugin-ai-${kind}-generation-web.${provider}.property.${field} - Override labels for a specific AI provider and generation type
Generic: ly.img.plugin-ai-generation-web.property.${field} - Override labels for all AI plugins

Where ${kind} can be:

image for image generation plugins
video for video generation plugins
audio for audio generation plugins
text for text generation plugins

Basic Example

// Customize labels for your AI generation interface
cesdk.i18n.setTranslations({
  en: {
    // Generic labels (applies to ALL AI plugins)
    'ly.img.plugin-ai-generation-web.property.prompt': 'Describe what you want to create',
    'ly.img.plugin-ai-generation-web.property.image_size': 'Image Dimensions',
    'ly.img.plugin-ai-generation-web.property.duration': 'Video Length',

    // Provider-specific for images (highest priority)
    'ly.img.plugin-ai-image-generation-web.fal-ai/recraft-v3.property.prompt': 'Describe your Recraft image',
    'ly.img.plugin-ai-image-generation-web.fal-ai/recraft-v3.property.image_size': 'Canvas Size',

    // Provider-specific for videos (highest priority)
    'ly.img.plugin-ai-video-generation-web.fal-ai/veo3.property.prompt': 'Describe your video scene',
    'ly.img.plugin-ai-video-generation-web.fal-ai/veo3.property.duration': 'Video Duration'
  }
});

For dropdown menus, add the option value to the translation key:

cesdk.i18n.setTranslations({
  en: {
    // Image generation dropdown options
    'ly.img.plugin-ai-image-generation-web.fal-ai/recraft-v3.property.image_size.square_hd': 'Square HD (1024×1024)',
    'ly.img.plugin-ai-image-generation-web.fal-ai/recraft-v3.property.image_size.portrait_4_3': 'Portrait 4:3 (768×1024)',

    // Video generation dropdown options
    'ly.img.plugin-ai-video-generation-web.fal-ai/veo3.property.duration.5': '5 seconds',
    'ly.img.plugin-ai-video-generation-web.fal-ai/veo3.property.duration.10': '10 seconds'
  }
});

QuickAction Translations

QuickActions use their own translation keys with provider-specific overrides:

cesdk.i18n.setTranslations({
  en: {
    // Provider-specific translations (highest priority)
    'ly.img.plugin-ai-image-generation-web.fal-ai/gemini-flash-edit.quickAction.editImage': 'Edit with Gemini',
    'ly.img.plugin-ai-text-generation-web.anthropic.quickAction.improve': 'Improve with Claude',

    // Generic plugin translations
    'ly.img.plugin-ai-image-generation-web.quickAction.editImage': 'Edit Image...',
    'ly.img.plugin-ai-image-generation-web.quickAction.editImage.prompt': 'Edit Image...',
    'ly.img.plugin-ai-image-generation-web.quickAction.editImage.apply': 'Change',
    'ly.img.plugin-ai-text-generation-web.quickAction.improve': 'Improve Text',
    'ly.img.plugin-ai-text-generation-web.quickAction.translate': 'Translate Text',
    'ly.img.plugin-ai-video-generation-web.quickAction.createVideo': 'Create Video'
  }
});

QuickAction Translation Priority:

Provider-specific: ly.img.plugin-ai-${kind}-generation-web.${provider}.quickAction.${action}.${field}
Generic plugin: ly.img.plugin-ai-${kind}-generation-web.quickAction.${action}.${field}

Translation Structure:

Base key (e.g., .quickAction.editImage): Button text when QuickAction is collapsed
.prompt: Label for input field when expanded
.prompt.placeholder: Placeholder text for input field
.apply: Text for action/submit button

Using Your Provider

Once you've created your provider, you need to initialize it with CreativeEditor SDK and integrate it into the UI.

Initializing Your Provider

Use the initializeProvider function to register your provider:

import { initializeProvider } from '@imgly/plugin-ai-generation-web';

// Create your provider
const myProvider = createMyProvider({
    proxyUrl: 'http://your-proxy-server.com/api/proxy',
    headers: {
        'x-custom-header': 'value',
        'x-client-version': '1.0.0'
    }
});

// Initialize the provider
function setupMyProvider(cesdk) {
    const result = initializeProvider(
        myProvider,
        {
            engine: cesdk.engine,
            cesdk
        },
        {
            debug: false,
            dryRun: false
        }
    );

    return result;
}

Panel IDs and Registration

When a provider is initialized, it automatically registers panels with specific IDs:

ly.img.plugin-ai-{kind}-generation-web.{provider-id}

For example:

A provider with ID my-image-provider for images registers a panel with ID ly.img.plugin-ai-image-generation-web.my-image-provider
A provider with ID fal-ai/recraft-v3 for images registers a panel with ID ly.img.plugin-ai-image-generation-web.fal-ai/recraft-v3

You can programmatically get a panel ID using the getPanelId function:

import { getPanelId } from '@imgly/plugin-ai-generation-web';

// Get panel ID for a provider
const panelId = getPanelId('my-image-provider');

// Open the panel
cesdk.ui.openPanel(panelId);

Quick actions are automatically registered in canvas menus with these IDs:

ly.img.plugin-ai-{kind}-generation-web.canvasMenu

For example:

Image quick actions: ly.img.plugin-ai-image-generation-web.canvasMenu
Video quick actions: ly.img.plugin-ai-video-generation-web.canvasMenu
Audio quick actions: ly.img.plugin-ai-audio-generation-web.canvasMenu
Text quick actions: ly.img.plugin-ai-text-generation-web.canvasMenu

Customizing Quick Action Availability

You can control which quick actions appear in your application using the Feature API. This is useful when you want to:

Show only specific AI capabilities to certain user groups
Simplify the UI by hiding advanced features
Create different feature sets for different subscription tiers
Disable actions that aren't relevant to your use case

Disabling Specific Quick Actions

All quick actions are enabled by default. To hide specific quick actions from the UI:

// Hide the "Edit Image" quick action from users
cesdk.feature.enable(
  'ly.img.plugin-ai-image-generation-web.quickAction.editImage',
  false
);

// Hide multiple text quick actions for a simpler experience
cesdk.feature.enable('ly.img.plugin-ai-text-generation-web.quickAction.changeTone', false);
cesdk.feature.enable('ly.img.plugin-ai-text-generation-web.quickAction.translate', false);

Dynamic Feature Control

You can also pass a function to dynamically control feature availability based on context. This is powerful for implementing complex business logic, time-based features, or context-sensitive UI. See the CE.SDK Feature API documentation for more details.

// Show advanced AI features only during business hours
cesdk.feature.enable(
  'ly.img.plugin-ai-image-generation-web.quickAction.artistTransfer',
  ({ isPreviousEnable }) => {
    const hour = new Date().getHours();
    const isBusinessHours = hour >= 9 && hour < 18;
    return isBusinessHours && isPreviousEnable();
  }
);

// Disable certain quick actions based on selected content
cesdk.feature.enable(
  'ly.img.plugin-ai-text-generation-web.quickAction.translate',
  ({ engine, isPreviousEnable }) => {
    const selectedBlocks = engine.block.findAllSelected();
    if (selectedBlocks.length === 0) return false;
    
    const blockId = selectedBlocks[0];
    const textContent = engine.block.getString(blockId, 'text/text');
    
    // Only show translate if text is long enough
    const hasEnoughText = textContent && textContent.length > 20;
    return hasEnoughText && isPreviousEnable();
  }
);

Creating Feature Sets for Different User Tiers

// Configure features based on user subscription
function configureAIFeatures(cesdk, userTier) {
  if (userTier === 'basic') {
    // Basic users only get simple text improvements
    cesdk.feature.enable('ly.img.plugin-ai-text-generation-web.quickAction.improve', true);
    cesdk.feature.enable('ly.img.plugin-ai-text-generation-web.quickAction.fix', true);
    cesdk.feature.enable('ly.img.plugin-ai-text-generation-web.quickAction.translate', false);
    cesdk.feature.enable('ly.img.plugin-ai-text-generation-web.quickAction.changeTone', false);
    
    // Disable advanced image features
    cesdk.feature.enable('ly.img.plugin-ai-image-generation-web.quickAction.artistTransfer', false);
    cesdk.feature.enable('ly.img.plugin-ai-image-generation-web.quickAction.styleTransfer', false);
  } else if (userTier === 'premium') {
    // Premium users get all features (default behavior)
    // All quick actions are enabled by default
  }
}

Available Feature Flags

Core Plugin Features

These feature flags control the main functionality of each AI plugin:

General Features:

ly.img.plugin-ai-{kind}-generation-web.providerSelect - Enable/disable provider selection dropdown in panels
ly.img.plugin-ai-{kind}-generation-web.quickAction - Enable/disable all quick actions for a plugin
ly.img.plugin-ai-{kind}-generation-web.quickAction.providerSelect - Enable/disable provider selection dropdown in quick actions

Input Type Features (Image & Video only):

ly.img.plugin-ai-image-generation-web.fromText - Enable/disable text-to-image generation
ly.img.plugin-ai-image-generation-web.fromImage - Enable/disable image-to-image generation
ly.img.plugin-ai-video-generation-web.fromText - Enable/disable text-to-video generation
ly.img.plugin-ai-video-generation-web.fromImage - Enable/disable image-to-video generation

Usage Examples:

// Hide provider selection dropdown in video generation panel
cesdk.feature.enable('ly.img.plugin-ai-video-generation-web.providerSelect', false);

// Only allow text-to-image, disable image-to-image editing
cesdk.feature.enable('ly.img.plugin-ai-image-generation-web.fromImage', false);
cesdk.feature.enable('ly.img.plugin-ai-image-generation-web.fromText', true);

// Hide provider selection dropdown in quick actions (use default provider only)
cesdk.feature.enable('ly.img.plugin-ai-image-generation-web.quickAction.providerSelect', false);

// Disable all quick actions but keep panel generation available
cesdk.feature.enable('ly.img.plugin-ai-image-generation-web.quickAction', false);

Quick Action Features

The quick action feature flags follow this pattern: ly.img.plugin-ai-{kind}-generation-web.quickAction.{actionName}

Image Quick Actions:

ly.img.plugin-ai-image-generation-web.quickAction.editImage
ly.img.plugin-ai-image-generation-web.quickAction.swapBackground
ly.img.plugin-ai-image-generation-web.quickAction.styleTransfer
ly.img.plugin-ai-image-generation-web.quickAction.artistTransfer
ly.img.plugin-ai-image-generation-web.quickAction.createVariant
ly.img.plugin-ai-image-generation-web.quickAction.combineImages
ly.img.plugin-ai-image-generation-web.quickAction.remixPage
ly.img.plugin-ai-image-generation-web.quickAction.remixPageWithPrompt

Text Quick Actions:

ly.img.plugin-ai-text-generation-web.quickAction.improve
ly.img.plugin-ai-text-generation-web.quickAction.fix
ly.img.plugin-ai-text-generation-web.quickAction.shorter
ly.img.plugin-ai-text-generation-web.quickAction.longer
ly.img.plugin-ai-text-generation-web.quickAction.changeTone
ly.img.plugin-ai-text-generation-web.quickAction.translate
ly.img.plugin-ai-text-generation-web.quickAction.changeTextTo

Video Quick Actions:

ly.img.plugin-ai-video-generation-web.quickAction.createVideo

Note: Quick actions are automatically enabled when their plugin is loaded. Each quick action manages its own feature flag internally, ensuring proper initialization and registration.

Provider-Specific Style Features

Some providers (like RecraftV3 and Recraft20b) support style groups that can be controlled independently:

RecraftV3 Style Groups:

ly.img.plugin-ai-image-generation-web.fal-ai/recraft-v3.style.image - Enable/disable image styles (realistic, digital illustration)
ly.img.plugin-ai-image-generation-web.fal-ai/recraft-v3.style.vector - Enable/disable vector styles (vector illustration and variants)

Recraft20b Style Groups:

ly.img.plugin-ai-image-generation-web.fal-ai/recraft/v2/text-to-image.style.image - Enable/disable image styles
ly.img.plugin-ai-image-generation-web.fal-ai/recraft/v2/text-to-image.style.vector - Enable/disable vector styles
ly.img.plugin-ai-image-generation-web.fal-ai/recraft/v2/text-to-image.style.icon - Enable/disable icon styles

When all style groups are disabled for a provider, it automatically falls back to the 'any' style. For more details on style control, see the @imgly/plugin-ai-image-generation-web documentation.

Using with Existing AI Generation Plugins

IMG.LY offers several pre-built AI generation packages that work with this base plugin:

import CreativeEditorSDK from '@cesdk/cesdk-js';

// Import plugin packages
import ImageGeneration from '@imgly/plugin-ai-image-generation-web';
import FalAiImage from '@imgly/plugin-ai-image-generation-web/fal-ai';
import VideoGeneration from '@imgly/plugin-ai-video-generation-web';
import FalAiVideo from '@imgly/plugin-ai-video-generation-web/fal-ai';

// Initialize CreativeEditor SDK
CreativeEditorSDK.create(domElement, {
    license: 'your-license-key'
}).then(async (cesdk) => {
    // Add default asset sources
    await cesdk.addDefaultAssetSources();

    // Image generation with Fal.ai models
    cesdk.addPlugin(
        ImageGeneration({
            text2image: FalAiImage.RecraftV3({
                proxyUrl: 'http://your-proxy-server.com/api/proxy'
            }),
            // Alternative: FalAiImage.Recraft20b({ proxyUrl: 'http://your-proxy-server.com/api/proxy' }),
            image2image: FalAiImage.GeminiFlashEdit({
                proxyUrl: 'http://your-proxy-server.com/api/proxy'
            })
        })
    );

    // Video generation
    cesdk.addPlugin(
        VideoGeneration({
            text2video: FalAiVideo.MinimaxVideo01Live({
                proxyUrl: 'http://your-proxy-server.com/api/proxy'
            })
        })
    );

    // Add quick action menus to canvas
    cesdk.ui.setCanvasMenuOrder([
        'ly.img.plugin-ai-image-generation-web.canvasMenu',
        'ly.img.plugin-ai-video-generation-web.canvasMenu',
        ...cesdk.ui.getCanvasMenuOrder()
    ]);
});

Advanced Features

Middleware

The package includes a middleware system to augment the generation flow:

Rate Limiting Middleware

import { rateLimitMiddleware } from '@imgly/plugin-ai-generation-web';

// Create a rate limiting middleware
const rateLimit = rateLimitMiddleware({
    maxRequests: 10,
    timeWindowMs: 60000, // 1 minute
    onRateLimitExceeded: (input, options, info) => {
        console.log(
            `Rate limit exceeded: ${info.currentCount}/${info.maxRequests}`
        );
        return false; // Reject request
    }
});

// Apply middleware to your provider
const provider = {
    // ...provider config
    output: {
        middleware: [rateLimit]
        // ...other output config
    }
};

Note: This middleware provides client-side rate limiting for UI purposes only. Always implement proper server-side rate limiting and authentication for production APIs.

Upload Middleware

The uploadMiddleware allows you to upload generated content to your own servers:

import { uploadMiddleware } from '@imgly/plugin-ai-generation-web';

// Create an upload middleware
const upload = uploadMiddleware(async (output) => {
  // Upload the output to your server/storage
  const response = await fetch('https://your-api.example.com/upload', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify(output)
  });
  
  const result = await response.json();
  
  // Return the output with the updated URL
  return {
    ...output,
    url: result.url
  };
});

// Apply middleware to your provider
const provider = {
  // ...provider config
  output: {
    middleware: [upload]
    // ...other output config
  }
};

Provider Registry

The ProviderRegistry is a global singleton that manages all registered providers:

import { ProviderRegistry } from '@imgly/plugin-ai-generation-web';

// Get the global registry
const registry = ProviderRegistry.get();

// Get all registered providers
const allProviders = registry.getAll();

// Get providers by kind
const imageProviders = registry.getByKind('image');

// Find a specific provider
const myProvider = registry.getById('my-provider-id');

TypeScript Support

This package is fully typed with TypeScript, providing excellent IntelliSense support during development:

Generic Provider Types: Strongly typed providers with input/output validation
Quick Action Types: Type-safe quick action definitions with proper input mapping
Registry Types: Fully typed action and provider registries
Middleware Types: Typed middleware functions for better composition

API Reference

Core Exports

// Provider types and interfaces
export { Provider, ImageOutput, VideoOutput, AudioOutput, TextOutput } from './core/provider';

// Action registry
export { ActionRegistry, QuickActionDefinition, PluginActionDefinition } from './core/ActionRegistry';

// Provider registry
export { ProviderRegistry } from './core/ProviderRegistry';

// Initialization functions
export { initializeProvider, initializeProviders } from './providers/';

// Middleware
export { loggingMiddleware, rateLimitMiddleware, uploadMiddleware } from './middleware/';

// Utilities
export { getPanelId, enableQuickActionForImageFill, mergeQuickActionsConfig } from './utils/';

Initialization Functions

initializeProviders

The initializeProviders function is used to initialize multiple providers at once. It creates a composite history asset source and library entry for all providers of the same kind.

const result = await initializeProviders(
    providers,
    { engine, cesdk },
    config
);

// Return value structure:
{
    panel: {
        builderRenderFunction: Function // UI builder function for provider selection
    },
    history: {
        assetSourceId: string,         // ID of the composite history asset source
        assetLibraryEntryId: string    // ID of the automatically created asset library entry
    },
    providerInitializationResults: Array<{
        provider: Provider,
        result: ProviderInitializationResult
    }>
}

Key Points:

Creates a composite history asset source with ID format: ly.img.plugin-ai-{kind}-generation-web.history
Automatically creates an asset library entry with the same ID as the asset source
The library entry is configured with appropriate settings (sortBy: insertedAt descending, canRemove: true, etc.)
Returns both the asset source ID and library entry ID for reference

Common Types

// Provider configuration
interface CommonProviderConfiguration<I, O extends Output> {
    proxyUrl: string;
    debug?: boolean;
    middleware?: Middleware<I, O>[];
    headers?: Record<string, string>;
    history?: false | '@imgly/local' | '@imgly/indexedDB' | (string & {});
    supportedQuickActions?: {
        [quickActionId: string]: Partial<QuickActionSupport<I>> | false | null;
    };
}

// Quick action definition
interface QuickActionDefinition<Q extends Record<string, any>> {
    id: string;
    type: 'quick';
    kind: OutputKind;
    label?: string;
    enable: boolean | ((context: { engine: CreativeEngine }) => boolean);
    render: (context: QuickActionRenderContext<Q>) => void;
}

Translations

For customization and localization, see the translations.json file which contains base translation keys that can be overridden for all AI plugins.

Keywords

CE.SDK

plugin

ai generation

FAQs

What is @imgly/plugin-ai-generation-web?

Is @imgly/plugin-ai-generation-web popular?

Is @imgly/plugin-ai-generation-web well maintained?

Package last updated on 09 Sep 2025

Did you know?

Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.

Install

@imgly/plugin-ai-generation-web

IMG.LY AI Generation Utilities

Overview