ai-multimodal

The AI Multimodal Processing skill provides a comprehensive interface for interacting with the Google Gemini API (2.0/2.5 series). It is designed for software agents and engineers requiring advanced media analysis, document extraction, and generation capabilities. By leveraging Gemini's massive context window (up to 2M tokens), this skill enables end-to-end processing of long-form audio, hours of video content, and multi-page documents, making it an essential tool for data-heavy workflows and automated content production.

• Advanced Audio Processing: Generate accurate transcriptions with precise timestamps, summarize multi-hour recordings, perform speaker identification, and analyze environmental sounds.

• Computer Vision & Image Understanding: Execute object detection, pixel-level segmentation, visual Q&A, and high-volume image comparison. Includes OCR for extracting text from complex layouts.

• Video Intelligence: Analyze video content via file upload or YouTube URL. Capabilities include scene detection, temporal Q&A, and frame-level analysis for large datasets up to 6 hours.

• Document Extraction: Native vision-based parsing for PDFs (up to 1,000 pages). Extract structured data from tables, forms, charts, and diagrams into clean JSON or Markdown formats.

• Generative Capabilities: Generate high-quality images from text prompts, support for iterative refinement, image editing, and multi-image composition with multiple aspect ratios.

• Supports both Google AI Studio and Vertex AI platforms for maximum deployment flexibility.

• Requires API configuration via environment variables (GEMINI_API_KEY) with tiered priority loading for secure and local development.

• Integrates with standard media formats including MP3, WAV, MP4, PDF, and various image types (JPEG, PNG, WEBP).

• Performance is optimized via automated media compression and batch processing scripts to handle large inputs within token limits.

• Designed for technical environments using Python, providing clean wrappers for the google-genai SDK to ensure repeatable, production-ready AI pipelines.