@imgly/plugin-ai-apps-web

IMG.LY AI Apps for Web

A plugin for orchestrating all AI generation capabilities in CreativeEditor SDK.

Overview

The @imgly/plugin-ai-apps-web package provides a unified interface for accessing and organizing all AI generation features within the CreativeEditor SDK. It combines image, video, audio, and text generation capabilities into a single cohesive user experience.

Key Features

• Unified AI Experience: Centralized access to all AI generation capabilities
• AI Dock Component: Single entry point for all AI features with loading indicators
• Smart Provider Management: Automatically organizes providers based on editor mode
• Integrated History: Seamless integration with asset libraries for generated content
• Mode-Aware Interface: Different UI layouts for Design and Video modes
• Cross-Plugin Support: Works with all IMG.LY AI generation plugins

Installation

npm install @imgly/plugin-ai-apps-web

Basic Usage

To use the plugin, import it and configure it with your preferred providers:

import CreativeEditorSDK from '@cesdk/cesdk-js';
import AiApps from '@imgly/plugin-ai-apps-web';

// Import providers from individual AI generation packages
import { AnthropicProvider } from '@imgly/plugin-ai-text-generation-web/anthropic';
import FalAiImage from '@imgly/plugin-ai-image-generation-web/fal-ai';
import FalAiVideo from '@imgly/plugin-ai-video-generation-web/fal-ai';
import ElevenLabs from '@imgly/plugin-ai-audio-generation-web/elevenlabs';
import FalAiSticker from '@imgly/plugin-ai-sticker-generation-web/fal-ai';

// Initialize CreativeEditor SDK
CreativeEditorSDK.create(domElement, {
  license: 'your-license-key'
  // Other configuration options...
}).then(async (cesdk) => {
  // Add the AI Apps plugin
  cesdk.addPlugin(
    AiApps({
      providers: {
        // Text generation
        text2text: AnthropicProvider({
          proxyUrl: 'http://your-proxy-server.com/api/proxy'
        }),

        // Image generation
        text2image: FalAiImage.RecraftV3({
          proxyUrl: 'http://your-proxy-server.com/api/proxy'
        }),
        image2image: FalAiImage.GeminiFlashEdit({
          proxyUrl: 'http://your-proxy-server.com/api/proxy'
        }),

        // Video generation (used in video mode)
        text2video: FalAiVideo.MinimaxVideo01Live({
          proxyUrl: 'http://your-proxy-server.com/api/proxy'
        }),
        image2video: FalAiVideo.MinimaxVideo01LiveImageToVideo({
          proxyUrl: 'http://your-proxy-server.com/api/proxy'
        }),

        // Audio generation (used in video mode)
        text2speech: ElevenLabs.MonolingualV1({
          proxyUrl: 'http://your-proxy-server.com/api/proxy'
        }),
        text2sound: ElevenLabs.SoundEffects({
          proxyUrl: 'http://your-proxy-server.com/api/proxy'
        }),

        // Sticker generation
        text2sticker: FalAiSticker.Recraft20b({
          proxyUrl: 'http://your-proxy-server.com/api/proxy'
        })
      }
    })
  );

  // Position the AI dock button in the dock order
  cesdk.ui.setDockOrder(['ly.img.ai.apps.dock', ...cesdk.ui.getDockOrder()]);
});

Configuration Options

The plugin accepts the following configuration options:

Option	Type	Description
providers	Providers	Object containing all AI providers to be used
debug	boolean	Print debug messages

Providers Configuration

The providers object can include the following provider functions:

Provider	Type	Description
text2text	Provider<'text'>	Provider for text generation and transformation
text2image	Provider<'image'>	Provider for text-to-image generation
image2image	Provider<'image'>	Provider for image-to-image transformation
text2video	Provider<'video'>	Provider for text-to-video generation (video mode only)
image2video	Provider<'video'>	Provider for image-to-video generation (video mode only)
text2speech	Provider<'audio'>	Provider for text-to-speech generation (video mode only)
text2sound	Provider<'audio'>	Provider for sound effects generation (video mode only)
text2sticker	Provider<'sticker'>	Provider for sticker generation

Provider Selection Strategy

The plugin intelligently selects which providers to use based on the current editor mode:

Design Mode

• Uses: text2text, text2image, image2image, text2sticker
• Focus: Image, text, and sticker generation for design workflows
• UI: Shows AI apps cards for different generation types

Video Mode

• Uses: All providers including text2video, image2video, text2speech, text2sound, text2sticker
• Focus: Comprehensive media generation for video production
• UI: Shows AI apps cards for different generation types

Advanced Configuration

Multiple Providers per Type

You can configure multiple providers for the same generation type by passing an array. When multiple providers are configured, a selection box will be rendered in the AI app interface allowing users to choose between different providers:

import FalAiImage from '@imgly/plugin-ai-image-generation-web/fal-ai';
import OpenAiImage from '@imgly/plugin-ai-image-generation-web/open-ai';

cesdk.addPlugin(
  AiApps({
    providers: {
      // Multiple image providers - selection box will be shown
      text2image: [
        FalAiImage.RecraftV3({
          proxyUrl: 'http://your-proxy-server.com/api/proxy'
        }),
        FalAiImage.Recraft20b({
          proxyUrl: 'http://your-proxy-server.com/api/proxy'
        }),
        OpenAiImage.GptImage1.Text2Image({
          proxyUrl: 'http://your-proxy-server.com/api/proxy'
        })
      ],
      
      // Other providers...
    }
  })
);

Custom Headers and Configuration

Pass custom headers and configuration to providers:

cesdk.addPlugin(
  AiApps({
    providers: {
      text2image: FalAiImage.RecraftV3({
        proxyUrl: 'http://your-proxy-server.com/api/proxy',
        headers: {
          'x-client-version': '1.0.0',
          'x-request-source': 'cesdk-plugin',
          'x-user-id': 'user-123'
        },
        debug: true
      })
    },
    debug: true
  })
);

Property Configuration

Providers support configuring default values for their properties. These defaults can be static or dynamic based on context:

cesdk.addPlugin(
  AiApps({
    providers: {
      text2image: FalAiImage.RecraftV3({
        proxyUrl: 'http://your-proxy-server.com/api/proxy',
        properties: {
          // Static default
          image_size: { default: 'square_hd' },

          // Dynamic default based on locale
          style: {
            default: (context) => {
              return context.locale === 'ja' ? 'anime' : 'realistic';
            }
          }
        }
      })
    }
  })
);

UI Integration

The plugin adds the following UI components to CreativeEditor SDK:

AI Dock Button

The main entry point for all AI features, accessible from the dock:

• ID: ly.img.ai.apps.dock
• Functionality: Opens AI generation interface
• Loading States: Shows progress indicators during generation
• Mode Awareness: Adapts interface based on current editor mode

AI Apps Menu

The plugin shows a card-based interface for different AI capabilities:

• Generate Image: Access to image generation providers
• Generate Video: Access to video generation providers
• Generate Audio: Access to audio generation providers
• Edit Text: Access to text generation providers

Dock Integration

To position the AI dock button in your editor's dock, use the setDockOrder method:

// Add the AI dock component to the beginning of the dock
cesdk.ui.setDockOrder(['ly.img.ai.apps.dock', ...cesdk.ui.getDockOrder()]);

// Or add it at a specific position
const currentOrder = cesdk.ui.getDockOrder();
currentOrder.splice(2, 0, 'ly.img.ai.apps.dock');
cesdk.ui.setDockOrder(currentOrder);

Asset History Integration

The plugin automatically integrates generated assets into the appropriate asset libraries. For each generation type, both a history asset source and a corresponding asset library entry are automatically created with the same ID:

• Image Generation History: ly.img.ai.image-generation.history
• Video Generation History: ly.img.ai.video-generation.history
• Audio Generation History: ly.img.ai.audio-generation.history
• Sticker Generation History: ly.img.ai.sticker-generation.history

These asset library entries are automatically configured with:

• Sorted by insertion date (newest first)
• Square grid item height
• Cover background type
• Removable items

Integrating with Default Asset Libraries

To add these history sources to CE.SDK's default asset library entries, use the following approach:

// Add AI image history to the default image asset library
const imageEntry = cesdk.ui.getAssetLibraryEntry('ly.img.image');
if (imageEntry != null) {
  cesdk.ui.updateAssetLibraryEntry('ly.img.image', {
    sourceIds: [...imageEntry.sourceIds, 'ly.img.ai.image-generation.history']
  });
}

// Add AI video history to the default video asset library
const videoEntry = cesdk.ui.getAssetLibraryEntry('ly.img.video');
if (videoEntry != null) {
  cesdk.ui.updateAssetLibraryEntry('ly.img.video', {
    sourceIds: [...videoEntry.sourceIds, 'ly.img.ai.video-generation.history']
  });
}

// Add AI audio history to the default audio asset library
const audioEntry = cesdk.ui.getAssetLibraryEntry('ly.img.audio');
if (audioEntry != null) {
  cesdk.ui.updateAssetLibraryEntry('ly.img.audio', {
    sourceIds: [...audioEntry.sourceIds, 'ly.img.ai.audio-generation.history']
  });
}

// Add AI sticker history to the default sticker asset library
const stickerEntry = instance.ui.getAssetLibraryEntry('ly.img.sticker');
if (stickerEntry != null) {
  instance.ui.updateAssetLibraryEntry('ly.img.sticker', {
    sourceIds: [...stickerEntry.sourceIds, 'ly.img.ai.sticker-generation.history']
  });
}

This integration creates a seamless experience where users can easily find and reuse their AI-generated assets alongside other content.

Quick Actions Integration

The plugin automatically integrates with the quick actions system, providing context-sensitive AI operations directly in the canvas menu. You need to specify the children with the quick action order:

// Quick actions are automatically registered and will appear in canvas menus
cesdk.ui.setCanvasMenuOrder([
  {
    id: 'ly.img.ai.text.canvasMenu',
    children: [
      'ly.img.improve',
      'ly.img.fix',
      'ly.img.shorter',
      'ly.img.longer',
      'ly.img.separator',
      'ly.img.changeTone',
      'ly.img.translate',
      'ly.img.separator',
      'ly.img.changeTextTo'
    ]
  },
  {
    id: 'ly.img.ai.image.canvasMenu',
    children: [
      'ly.img.styleTransfer',
      'ly.img.artistTransfer',
      'ly.img.separator',
      'ly.img.editImage',
      'ly.img.swapBackground',
      'ly.img.createVariant',
      'ly.img.combineImages',
      'ly.img.separator',
      'ly.img.remixPage',
      'ly.img.separator',
      'ly.img.createVideo'
    ]
  },
  ...cesdk.ui.getCanvasMenuOrder()
]);

Quick actions provide:

• Context-aware operations: Work with currently selected blocks
• One-click transformations: Apply AI operations without opening panels
• Cross-plugin functionality: Actions from one plugin can work with providers from another

Error Handling and Debugging

Debug Mode

Enable debug mode to get detailed logging information:

cesdk.addPlugin(
  AiApps({
    providers: {
      text2image: FalAiImage.RecraftV3({
        proxyUrl: 'http://your-proxy-server.com/api/proxy',
        debug: true // Provider-level debugging
      })
    },
    debug: true // Plugin-level debugging
  })
);

Common Issues and Solutions

Provider Not Loading

• Check proxy URLs: Ensure all proxy URLs are correctly configured and accessible
• Verify licenses: Make sure your CreativeEditor SDK license includes AI features
• Check browser console: Look for network errors or API issues

Quick Actions Not Appearing

• Verify canvas menu order: Ensure quick action menus are added to canvas menu order with proper children configuration
• Check provider support: Verify that providers declare support for the quick actions
• Scope permissions: Ensure blocks have the required scopes for quick actions

Generation Failures

• API connectivity: Check that your proxy endpoints are working
• Rate limiting: Verify you're not exceeding API rate limits
• Input validation: Ensure inputs meet provider requirements

Plugin Architecture

How It Works

The AI Apps plugin acts as an orchestrator that:

• Initializes Providers: Sets up all configured AI providers
• Manages UI: Creates appropriate interface based on editor mode
• Coordinates Actions: Integrates quick actions across different plugins
• Handles Assets: Manages generated content and asset library integration

Provider Lifecycle

• Registration: Providers are registered with the global ProviderRegistry
• Initialization: Provider initialize methods are called
• UI Setup: Panels and quick actions are registered
• Event Handling: The plugin coordinates between providers and UI

TypeScript Support

The plugin is fully typed with TypeScript, providing excellent development experience:

import AiApps, { Providers } from '@imgly/plugin-ai-apps-web';

// Strongly typed provider configuration
const providers: Providers = {
  text2image: FalAiImage.RecraftV3({
    proxyUrl: 'http://your-proxy-server.com/api/proxy'
  }),
  // TypeScript will enforce correct provider types
};

cesdk.addPlugin(AiApps({ providers }));

Related Packages

This plugin works with the following IMG.LY AI generation packages:

• @imgly/plugin-ai-generation-web - Core utilities for AI generation
• @imgly/plugin-ai-image-generation-web - AI image generation
• @imgly/plugin-ai-video-generation-web - AI video generation
• @imgly/plugin-ai-audio-generation-web - AI audio generation
• @imgly/plugin-ai-text-generation-web - AI text generation

Examples

Simple Setup for Design Mode

import CreativeEditorSDK from '@cesdk/cesdk-js';
import AiApps from '@imgly/plugin-ai-apps-web';
import FalAiImage from '@imgly/plugin-ai-image-generation-web/fal-ai';

CreativeEditorSDK.create(domElement, {
  license: 'your-license-key'
}).then(async (cesdk) => {
  cesdk.addPlugin(
    AiApps({
      providers: {
        text2image: FalAiImage.RecraftV3({
          proxyUrl: 'http://your-proxy-server.com/api/proxy'
        })
      }
    })
  );

  cesdk.ui.setDockOrder(['ly.img.ai.apps.dock', ...cesdk.ui.getDockOrder()]);
});

Complete Setup for Video Mode

import CreativeEditorSDK from '@cesdk/cesdk-js';
import AiApps from '@imgly/plugin-ai-apps-web';
import FalAiImage from '@imgly/plugin-ai-image-generation-web/fal-ai';
import FalAiVideo from '@imgly/plugin-ai-video-generation-web/fal-ai';
import ElevenLabs from '@imgly/plugin-ai-audio-generation-web/elevenlabs';
import { AnthropicProvider } from '@imgly/plugin-ai-text-generation-web/anthropic';

CreativeEditorSDK.create(domElement, {
  license: 'your-license-key',
  ui: {
    elements: {
      panels: {
        settings: true
      }
    }
  }
}).then(async (cesdk) => {
  cesdk.addPlugin(
    AiApps({
      providers: {
        text2text: AnthropicProvider({
          proxyUrl: 'http://your-proxy-server.com/api/proxy'
        }),
        text2image: FalAiImage.RecraftV3({
          proxyUrl: 'http://your-proxy-server.com/api/proxy'
        }),
        image2image: FalAiImage.GeminiFlashEdit({
          proxyUrl: 'http://your-proxy-server.com/api/proxy'
        }),
        text2video: FalAiVideo.MinimaxVideo01Live({
          proxyUrl: 'http://your-proxy-server.com/api/proxy'
        }),
        image2video: FalAiVideo.MinimaxVideo01LiveImageToVideo({
          proxyUrl: 'http://your-proxy-server.com/api/proxy'
        }),
        text2speech: ElevenLabs.MonolingualV1({
          proxyUrl: 'http://your-proxy-server.com/api/proxy'
        }),
        text2sound: ElevenLabs.SoundEffects({
          proxyUrl: 'http://your-proxy-server.com/api/proxy'
        })
      }
    })
  );

  // Setup dock and canvas menus
  cesdk.ui.setDockOrder(['ly.img.ai.apps.dock', ...cesdk.ui.getDockOrder()]);
  cesdk.ui.setCanvasMenuOrder([
    {
      id: 'ly.img.ai.text.canvasMenu',
      children: [
        'ly.img.improve',
        'ly.img.fix',
        'ly.img.shorter',
        'ly.img.longer',
        'ly.img.separator',
        'ly.img.changeTone',
        'ly.img.translate',
        'ly.img.separator',
        'ly.img.changeTextTo'
      ]
    },
    {
      id: 'ly.img.ai.image.canvasMenu',
      children: [
        'ly.img.styleTransfer',
        'ly.img.artistTransfer',
        'ly.img.separator',
        'ly.img.editImage',
        'ly.img.swapBackground',
        'ly.img.createVariant',
        'ly.img.combineImages',
        'ly.img.separator',
        'ly.img.remixPage',
        'ly.img.separator',
        'ly.img.createVideo'
      ]
    },
    ...cesdk.ui.getCanvasMenuOrder()
  ]);
});

Translations

The AI Apps plugin uses translations from individual AI generation plugins. For customization, refer to the respective translation files:

• Base translations - Core translation keys
• Image generation translations - Image generation interfaces
• Video generation translations - Video generation interfaces
• Text generation translations - Text generation interfaces
• Audio generation translations - Audio generation interfaces

License

This plugin is part of the IMG.LY plugin ecosystem for CreativeEditor SDK. Please refer to the license terms in the package.