Using AI to Generate Text, Images, and Videos in a Single Workflow.
FROM Module 6: Prompt Engineering: Techniques and Approaches
Introduction
AI is evolving beyond just text-based interactions. Multimodal AI allows users to generate text, images, audio, and videos within a single workflow. This lesson will cover:
✅ What multimodal AI is
✅ Techniques for combining different media types
✅ Real-world applications
✅ Hands-on exercises
What is Multimodal AI?
Definition: Multimodal AI can process and generate content in multiple formats (text, images, video, speech, etc.).
Example:
- You provide a text prompt, and AI generates an image.
- AI then uses the image to generate a descriptive caption or video.
🤖 Popular Multimodal AI Models:
Several advanced AI models can process and generate multiple formats (text, images, video, and speech). Here are some of the top multimodal AI models:
- GPT-4V (Vision) – OpenAI’s multimodal version of GPT-4 that understands images and text together.
- DALL·E 3 – Generates high-quality AI images from text prompts and can now refine images using natural language.
- Gemini 1.5 (Google DeepMind) – Can process text, images, audio, and code in a single model.
- Grok-1.5V (xAI by Elon Musk) – A multimodal version of Grok that can interpret images and text-based inputs.
- Claude 3 (Anthropic) – Capable of handling text and some multimodal tasks (but not as visual-focused as GPT-4V or Gemini).
- Runway Gen-2 – A powerful AI video generator that transforms text prompts into short video clips.
- Pika Labs – Another AI tool for generating animated videos from text descriptions.
- Whisper (OpenAI) – An AI speech-to-text model that accurately transcribes and translates audio.
These models enable seamless multimodal workflows, making it possible to generate, edit, and enhance content across text, images, and video.
Multimodal Prompting Techniques
1. Text-to-Image Generation (Prompting for Images)
AI converts a detailed text prompt into an image.
Example Prompt:
“A futuristic city skyline at sunset, with flying cars and neon holograms reflecting off the glass buildings, in cyberpunk style.”
Best Practices:
- Be descriptive (e.g., “A cozy library with warm lighting and wooden bookshelves.”)
- Specify styles (e.g., “A Van Gogh-style painting of a sunflower field.”)
- Define composition (e.g., “A close-up portrait of a smiling astronaut on Mars.”)
2. Text-to-Video Generation (Prompting for Videos)
AI creates short videos from text descriptions or enhances images into animations.
- Example Prompt for Video AI (Runway ML):
“A golden retriever running on a beach at sunrise, slow motion, cinematic lighting.”
Best Practices:
- Use clear scene descriptions (e.g., “A waterfall in a dense jungle, viewed from a drone.”)
- Define camera movements (e.g., “A slow zoom into a spaceship cockpit.”)
- Add mood settings (e.g., “Dramatic lighting, 4K quality, cinematic tone.”)
Image-to-Text (Descriptive AI Captions & Summaries)
AI analyzes an image and generates text descriptions.
Example Use Case:
- Input: Upload a photo of the Eiffel Tower.
- AI Output: “A stunning view of the Eiffel Tower at night, illuminated against a deep blue sky.”
Best Practices:
- Request detailed descriptions (e.g., “Describe this image in 50 words.”)
- Use contextual instructions (e.g., “Generate a social media caption for this image.”)
4. Text-to-Speech (AI Voice Generation)
AI converts text into realistic voice narration.
Example Prompt for AI Voice:
“Read this article in a warm, friendly voice with natural pauses.”
Best Practices:
- Choose a tone (e.g., “Excited, formal, or calm.”)
- Set a pacing style (e.g., “Slow narration for storytelling.”)
- Specify emotion (e.g., “Sound enthusiastic while describing the product.”)
5. Combining Modalities in a Single Workflow
🔹 Example: AI-Powered Marketing Workflow
1. Generate a product description (Text)
- “A sleek, lightweight smartwatch with 7-day battery life and AI fitness tracking.”
2. Convert it into an ad image (Text-to-Image) - AI generates a high-quality product image.
3. Create a short promo video (Image-to-Video) - AI animates the product with smooth transitions.
4. Add AI voice narration (Text-to-Speech) - A professional AI voice reads the product features.
Best Practices:
- Define the end goal before prompting.
- Use consistent prompts across all media types.
- Fine-tune details to make outputs more realistic.
Real-World Applications of Multimodal AI
1. Content Creation & Marketing
- AI writes blog posts, generates matching images, and creates promotional videos.
- Example: An AI-generated travel blog that includes AI-created images and narrated videos.
2. Virtual Assistants & AI Chatbots
- AI chatbots can answer questions with text and images.
- Example: A virtual home designer suggests furniture and generates room mockups.
3. Art & Design
- AI helps concept artists generate quick sketches before turning them into 3D models.
- Example: Game designers use AI-generated landscapes for virtual worlds.
4. AI-Powered Video Editing
- AI can animate still images into short films.
- Example: Runway AI helps filmmakers create visual effects without green screens.
5. Journalism & Fact-Checking
- AI generates news summaries, verifies images, and detects deepfakes.
- Example: AI scans images to confirm their authenticity in breaking news.
Hands-On Exercise: Create a Multimodal AI Workflow
🔹 Goal: Use different AI tools to generate text, images, and video from a single concept.
Step 1: Generate a Concept
Pick a theme for your multimodal AI project.
- Example: “A futuristic eco-friendly city with AI-powered transportation.”
Step 2: Generate Text Content
🔹 Prompt:
“Write a 100-word description of a futuristic green city powered by AI and renewable energy.”
Step 3: Generate an Image Based on the Text
🔹 Prompt for an AI Image Generator:
“Create a detailed digital artwork of a futuristic eco-city with solar panels, flying cars, and green skyscrapers.”
Step 4: Generate a Short Video from the Image
🔹 Prompt for a Video Generator:
“Animate this futuristic city scene with moving traffic, flying drones, and changing weather effects.”
Step 5: Add AI Voice Narration
🔹 Prompt for AI Voice Generator:
“Narrate this description in an inspiring documentary-style voice.”
✅ End Result: A cohesive AI-generated project combining text, images, video, and speech!
Reflection Questions
- What was the most challenging part of using multimodal AI?
- How did changing the prompts affect AI’s output?
- How could you use multimodal AI in your field (marketing, education, design, etc.)?
1. The most challenging part of multimodal AI is getting the right an all-in-one model to do both text, image, audio, and video. I finally had to generate the prompt with deepseek and use DALL.E AND RUNWAY to generate the images. I tried to generate music with runway and realized it only generate images and videos though you have to pay to access the video generation feature.
2. One of the images generated by Gemini looked like she was on bended knees, I had to adjust the prompt to yield a better output.
3. I can use Multimodal AI to generate a workflow making it easier for me to achieve my goal in one work space without going back and forth.
The most challenging part is finding the right prompt and also getting the right tool to use
Changing the prompt in different tools help me see how output deferred at each stage
I can use multimodal AI to generate business pitchs and also generate contents for my business promotions
1. The prompt, giving the model little to insufficient details on the prompt gives me unsatisfied answers which makes me angry.
2. Changing of the prompt i.e giving more detailed prompt changes the entire answer or result.
3. In my field as a Microbiologist, I would use the Multimodal AI to research more into the world of microbes related to the aspect I’m diving into.
1. For me, the hardest part was figuring out how to communicate exactly what I wanted. Sometimes I assumed the AI would understand my intention from the image alone, but I realized I still needed to guide it with clear instructions. Getting the result I wanted often required several attempts and adjustments.
2. I noticed that even small changes in the wording of my prompts could lead to very different responses. When I provided more details and explained my expectations better, the output became more useful and closer to what I had in mind. It showed me that the quality of the result depends a lot on how the prompt is written.
3. In an administrative role, I think Multimodal AI could help with handling documents, converting information from images into editable text, preparing meeting summaries, and organizing files. It could also assist in drafting correspondence and managing routine office tasks more efficiently, allowing administrators to focus on more important responsibilities.
1)The most challenging part was creating clear and specific prompts that helped the AI accurately understand and interpret both text and visual inputs. Small mistakes in instructions could lead to inaccurate or incomplete outputs.
2) Changing the prompt significantly affected the quality and relevance of the output. More detailed and precise prompts produced more accurate, useful, and context-specific responses, while vague prompts resulted in generic answers.
3) In my health field, multimodal AI can be used to analyze medical images, create health education materials from text and visuals, support disease awareness campaigns, simplify complex health information for patients, and enhance health communication through engaging content such as infographics, videos, and presentations.
This was an exciting experience. but i could not get a good output because i was working with free vision of the apps.
Thank you.
Ginikanwa Njoku
What was the most challenging part of using multimodal AI?
The most challenging part was making sure the text, image, video, and voice outputs stayed consistent with the same idea. Each tool interprets prompts differently, so adjusting them for similar results required extra effort.
How did changing the prompts affect AI’s output?
Changing the prompts made the outputs more detailed and accurate. More specific instructions improved the quality, style, and relevance of the generated content, while vague prompts produced more general results.
How could you use multimodal AI in your field (marketing, education, design, etc.)?
Multimodal AI can be used to create engaging content quickly. In education, it can generate learning materials and visual explanations. In marketing, it can create promotional content. In design, it can assist with concept development and visual presentations.
The most challenging part of using multimodal A.I was getting the accurate output at once
Changing the prompts changed the desired output
I can use multimodal a.i to generate blogs and content creation