TwelveLabs Pegasus 1.5 视频语言模型上线 Amazon Bedrock

On Oct 9, 2026 we announced the availability of TwelveLabs’ Pegasus 1.5 on Amazon Bedrock. Pegasus 1.5 is a video language model that turns long-form video into structured, timestamped metadata. You describe the moments that matter to your workflow, and the model returns every matching segment as JSON.

Video represents a significant and complex category of unstructured data, yet most of it remains unsearchable and underutilized. Extracting structured, actionable insights from that content has historically required labor-intensive manual review or fragmented toolchains.

With earlier versions of Pegasus, you can point the model at a clip, ask a question, and get an answer. Now, Pegasus 1.5 analyzes the entire video, finds every segment that matches user-defined criteria, and returns structured, timestamped metadata. It also introduces video segmentation, multimodal prompting with reference images, and on-the-fly processing.

With Pegasus 1.5 on Amazon Bedrock, you can analyze videos directly from an Amazon Simple Storage Service (Amazon S3) URI or a base64 string with no pre-indexing required, and your data remains within the AWS data boundary.

In this post, we cover what’s new in Pegasus 1.5, how the model is packaged on Amazon Bedrock, and how to send your first request. We then walk through video segmentation, structured JSON output, and multimodal prompting.

What’s new in Pegasus 1.5

Pegasus 1.5 has the following changes compared to previous versions:

• Time-based metadata generation. You define the segments, events, or domain-specific cues that matter to your workflow, such as editorial segments, sports plays, speaker changes, or brand appearances. Pegasus 1.5 returns every matching segment as JSON, with start and end times and the requested fields populated.
• Multimodal prompting. A segment definition can combine a text description with a visual reference. This makes it possible to locate a specific product, logo, or person, along with what they are doing on screen.
• On-the-fly processing. No ingestion pipeline or pre-indexing step is required. Videos are sent directly from Amazon S3 and results are returned in the response.
• Extended processing capacity. Pegasus 1.5 analyzes videos up to two hours long in a single request, up from one hour in Pegasus 1.2.

Pegasus 1.5 retains and enhances the core capabilities that made Pegasus 1.2 a powerful tool for video understanding, including:

• Video-to-text generation: Creates detailed textual descriptions based on video content.
• Granular visual comprehension: Analyzes objects, on-screen text, and numerical content within video frames.
• Temporal grounding: Accurately identifies timestamps of specific events within videos.
• Multimodal understanding: Combines visual, audio, and textual information for comprehensive analysis.
• Multilingual support: Full support for English, with partial support for Arabic, Chinese, French, German, Italian, Japanese, Korean, Portuguese, Russian, Spanish, Thai, and Vietnamese.

How Pegasus 1.5 works on Amazon Bedrock

Pegasus 1.5 takes a video and either a text prompt or a set of segment definitions, and returns text or JSON. You call it on the bedrock-runtime endpoint with the InvokeModel API, or with InvokeModelWithResponseStream when you want the response streamed as it’s generated.

Each request runs in one of two analysis modes, chosen with the analysisMode field:

You can point the model at video in two ways. An S3 location (s3Location) takes the object’s uri and the bucketOwner account ID, and suits long-form video. A base64 string (base64String) embeds the video in the request body, up to 25 MB, and suits short clips.

To track cost by application, create an application inference profile from the Pegasus 1.5 model, tag it, and call it by its Amazon Resource Name (ARN) in place of the model ID.

Getting started with Pegasus 1.5

The following section outlines how to get started with Pegasus 1.5.

Prerequisites

Before you begin, make sure you have:

1. An active AWS account with Amazon Bedrock access.
2. Python 3.10 or later, with the AWS SDK for Python (Boto3) installed.
3. AWS Identity and Access Management (IAM) permissions for bedrock:InvokeModel and bedrock:InvokeModelWithResponseStream.
4. Model access enabled for TwelveLabs Pegasus 1.5 in the Amazon Bedrock console.
5. Videos stored in an Amazon S3 bucket, or short clips you can send inline.

The following policy grants both invoke actions on Pegasus 1.5.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "bedrock:InvokeModel",
                "bedrock:InvokeModelWithResponseStream"
            ],
            "Resource": [
                "arn:aws:bedrock:us-west-2::foundation-model/twelvelabs.pegasus-1-5-v1:0"
            ]
        }
    ]
}

Boto3 signs requests with your usual AWS credentials. You can also authenticate with an Amazon Bedrock API key by setting it in the AWS_BEARER_TOKEN_BEDROCK environment variable. For production, prefer a short-term key generated from your IAM credentials with the aws-bedrock-token-generator package, because it expires automatically. Bearer-token authentication also requires the bedrock:CallWithBearerToken permission.

Send your first request

Install Boto3:

pip install boto3

The following example analyzes a video stored in Amazon S3 in general mode. This example uses the global Cross Region Inference (CRIS) endpoint. To target the US geographic endpoint instead, replace “global.” with “us.” in the endpoint URL.

import boto3
import json
from botocore.config import Config

AWS_REGION = "us-west-2"
MODEL_ID = "global.twelvelabs.pegasus-1-5-v1:0"

bedrock_runtime = boto3.client(
    service_name="bedrock-runtime",
    region_name=AWS_REGION,
    config=Config(
        # keep a longer read_timeout for lengthier invocations such as video segmentation
        read_timeout=900,
        retries={"max_attempts": 0},
        connect_timeout=10,
    ),
)

# Analyze a video stored in Amazon S3
payload = {
    "inputPrompt": "Summarize this video and identify the key topics discussed.",
    "mediaSource": {
        "s3Location": {
            "uri": "s3://amzn-s3-demo-pegasus-15/your-video.mp4",
            "bucketOwner": "your-account-id"
        }
    },
    "temperature": 0.2,
    "maxOutputTokens": 4096,
    "includeMetering": True
}

response = bedrock_runtime.invoke_model(
    modelId=MODEL_ID,
    body=json.dumps(payload)
)
result = json.loads(response["body"].read())
print(result["message"])

The response carries the answer in message, the token counts in input_tokens and output_tokens, and the reason generation ended in stopReason. A stopReason of length means the answer reached maxOutputTokens before it finished, so check it before you rely on the output:

if result["stopReason"] == "length":
    print("The answer was cut off. Raise maxOutputTokens and try again.")
print("Tokens in:", result["input_tokens"], "out:", result["output_tokens"])

Request fields for general mode:

For a short clip, you can send the video inline instead of from S3:

import base64

with open("your-video.mp4", "rb") as f:
    video_b64 = base64.b64encode(f.read()).decode("utf-8")

payload["mediaSource"] = {"base64String": video_b64}
response = bedrock_runtime.invoke_model(modelId=MODEL_ID, body=json.dumps(payload))
print(json.loads(response["body"].read())["message"])

Note that the preceding example uses the optional includeMetering attribute. includeMetering: True makes Pegasus 1.5 add two billing-usage fields to the non-streaming response: outputTokenCount and inputVideoSecond (the analyzed video’s duration in seconds).

Stream the response

For interactive applications, InvokeModelWithResponseStream returns the answer in chunks as it’s generated. Each chunk carries the next piece of text in message:

response = bedrock_runtime.invoke_model_with_response_stream(
    modelId=MODEL_ID,
    body=json.dumps(payload)
)
for event in response["body"]:
    chunk = json.loads(event["chunk"]["bytes"])
    print(chunk.get("message", ""), end="", flush=True)
print()

Return structured JSON

In general mode, you can constrain the answer to a JSON Schema so downstream code can parse it directly. Set responseFormat.type to json_schema and put the schema under jsonSchema. The answer arrives in message as a JSON string:

payload = {
    "inputPrompt": "Analyze this video and return a structured summary.",
    "mediaSource": {
        "s3Location": {
            "uri": "s3://amzn-s3-demo-pegasus-15/your-video.mp4",
            "bucketOwner": "your-account-id"
        }
    },
    "temperature": 0,
    "maxOutputTokens": 2048,
    "includeMetering": True,
    "responseFormat": {
        "type": "json_schema",
        "jsonSchema": {
            "type": "object",
            "properties": {
                "summary": {"type": "string"},
                "key_scenes": {"type": "array", "items": {"type": "string"}},
                "setting": {"type": "string"}
            },
            "required": ["summary", "key_scenes", "setting"]
        }
    }
}
response = bedrock_runtime.invoke_model(modelId=MODEL_ID, body=json.dumps(payload))
summary = json.loads(json.loads(response["body"].read())["message"])
print(summary["summary"])
for scene in summary["key_scenes"]:
    print(" -", scene)

Video segmentation: Turning video into structured data

Time-based metadata generation is where Pegasus 1.5 differs most from earlier versions. Instead of a prompt, you send one or more segment definitions, and the model returns every segment of the video that matches each one.

Set analysisMode to time_based_metadata and responseFormat.type to segment_definitions. The definitions take the place of inputPrompt, so leave that field out. Each definition has:

• id: a name for the definition, returned on every segment it matches.
• description: what a matching segment looks like.
• fields: the values to extract for each segment, each with a name, a type (string, boolean, number, integer, array, or timestamp), and a description.

The following example identifies each editorial section in a news broadcast and returns a title, summary, and key topics for each one:

payload = {
    "analysisMode": "time_based_metadata",
    "mediaSource": {
        "s3Location": {
            "uri": "s3://amzn-s3-demo-pegasus-15/news-broadcast.mp4",
            "bucketOwner": "your-account-id"
        }
    },
    "responseFormat": {
        "type": "segment_definitions",
        "segmentTimeFormat": "hh:mm:ss",
        "segmentDefinitions": [
            {
                "id": "editorial_section",
                "description": "A distinct editorial section of the broadcast, "
                "such as a new story, topic, or segment of the program.",
                "fields": [
                    {"name": "segment_title", "type": "string",
                     "description": "A descriptive title"},
                    {"name": "summary", "type": "string",
                     "description": "A brief summary of the segment content"},
                    {"name": "key_topics", "type": "array", "items": {"type": "string"},
                     "description": "Main topics discussed"}
                ]
            }
        ]
    },
    "temperature": 0,
    "maxOutputTokens": 32768,
    "includeMetering": True
}
response = bedrock_runtime.invoke_model(
    modelId=MODEL_ID,
    contentType="application/json",
    accept="application/json",
    body=json.dumps(payload)
)
result = json.loads(response["body"].read())
for segment in result["segments"]:
    print(segment["start_sec"], "-", segment["end_sec"], segment["segment_title"])
    print(" ", segment["summary"])
    print(" topics:", ", ".join(segment["key_topics"]))

The previous sample code when tested against a sports clip may produce an output like the following sample.

{
    "segments": [
        {
            "start_sec": "00:04:24",
            "end_sec": "00:05:22",
            "segment_title": "Substitutions and Added Time",
            "summary": "The referee announces three minutes of added time, and both teams make substitutions in preparation for a potential penalty shootout.",
            "key_topics": ["Substitutions", "Added time", "Penalty shootout preparation", "Tactical changes"]
        },
        {
            "start_sec": "00:05:22",
            "end_sec": "00:08:57",
            "segment_title": "Last Gasp Attempts and Final Whistle",
            "summary": "The final minutes of the match see intense action from both sides, with several close calls before the referee blows the final whistle, ending the game in a 3-3 draw.",
            "key_topics": ["Final whistle", "Draw"]
        },
        {
            "start_sec": "00:08:57",
            "end_sec": "00:09:29",
            "segment_title": "Match Conclusion and Penalty Shootout Setup",
            "summary": "The match concludes with a 3-3 draw, leading to a penalty shootout. Players from both teams shake hands and prepare for the decisive moments.",
            "key_topics": ["Match conclusion", "Penalty shootout", "Player handshakes", "Final"]
        }
    ],
    "definition_errors": [],
    "input_tokens": 249408,
    "output_tokens": 802,
    "stopReason": "stop",
    "finish_reason": "stop"
}

Each segment carries start_sec, end_sec, and the fields you defined. segmentTimeFormat controls how the times are written: seconds returns numbers you can do arithmetic on, while hh:mm:ss and hh:mm:ss.fff return readable timestamps as strings. The response also includes a definition_errors list, which is empty when every definition was processed.

A single request can include multiple segment definitions, for example one for editorial sections and another for on-screen brand appearances. Pegasus 1.5 returns the matches for all of them in one segments list, and tags each segment with _definition_id so you can tell them apart. The following example adds a second definition to the previous request and groups the results:

from collections import defaultdict

payload["responseFormat"]["segmentDefinitions"].append({
    "id": "brand_appearance",
    "description": "A shot where a company logo or branded product "
    "is clearly visible on screen.",
    "fields": [
        {"name": "brand", "type": "string", "description": "The brand shown"},
        {"name": "is_prominent", "type": "boolean",
         "description": "Whether the brand is the main focus of the shot"}
    ]
})
response = bedrock_runtime.invoke_model(modelId=MODEL_ID, body=json.dumps(payload))
result = json.loads(response["body"].read())
by_definition = defaultdict(list)
for segment in result["segments"]:
    by_definition[segment["_definition_id"]].append(segment)
for definition_id, segments in by_definition.items():
    print(definition_id, len(segments), "segments")

Because the schema is defined once, the same definitions can be applied across an entire video library, producing consistent metadata for every asset.

Multimodal prompting: Adding visual context to your queries

Pegasus 1.5 introduces multimodal prompting, so you can include reference images alongside your text prompts. This is useful when you want to ask the model about specific objects, people, or scenes that are easier to show than describe.

For example, you can provide a product image and ask the model to identify every moment where that product appears in a video. Or you can provide a reference image of a specific logo and ask the model to detect brand appearances and their timestamps. This capability enhances the precision of video analysis, and opens up use cases such as visual brand monitoring, product placement analysis, and image-guided content search.

Reference images attach differently depending on the mode. In general mode, put them in promptV2.mediaSources, alongside the question in promptV2.inputText. In segmentation mode (analysisMode: time_based_metadata), attach them to the mediaSources of each segment definition inside responseFormat.segmentDefinitions. A definition can then match a visual reference as well as a text description. Each image needs a name, a media_type of image, and exactly one source: either an inline base64String or an s3Location with uri and bucketOwner. A single request can carry up to 30 reference images.

A general-mode request asks the model about an object shown in a reference image. The following sample payload and response provide an example in general mode.

payload = {
    "analysisMode": "general",
    "mediaSource": {"s3Location": {"uri": "s3://...", "bucketOwner": "..."}},
    "promptV2": {
        "inputText": "Does the object in the reference image appear in this video?",
        "mediaSources": [
            {"name": "ref1", "media_type": "image", "base64String": "..."}
        ]
    }
    …
}

{
    "input_tokens": 144,
    "output_tokens": 11,
    "message": "Yes, the green canoe appears in the video.",
    "stopReason": "stop"
}

In segmentation mode, the reference image lives inside the definition, so Pegasus returns only the segments where that product appears. The following sample payload and response provide an example in segmentation mode.

payload = {
    "analysisMode": "time_based_metadata",
    "mediaSource": {"s3Location": {"uri": "s3://...", "bucketOwner": "..."}},
    "responseFormat": {
        "type": "segment_definitions",
        "segmentTimeFormat": "hh:mm:ss",
        "segmentDefinitions": [
            {
                "id": "product_appearance",
                "description": "A shot showing the product in the reference image",
                "fields": [
                    {"name": "what", "type": "string", "description": "what is on screen"}
                ],
                "mediaSources": [
                    {"name": "ref1", "media_type": "image",
                     "s3Location": {"uri": "s3://...", "bucketOwner": "..."}}
                ]
            }
        ]
    }
}

{
    "segments": [
        {"start_sec": "00:00:00", "end_sec": "00:00:02",
         "what": "A family of four in a green canoe, paddling on a calm river surrounded by towering red rock formations."},
        {"start_sec": "00:00:09", "end_sec": "00:00:10",
         "what": "The family back in the canoe, paddling away from the shore, with the children smiling and enjoying the serene surroundings."}
    ],
    "definition_errors": [],
    "input_tokens": 4498,
    "output_tokens": 292,
    "stopReason": "stop",
    "finish_reason": "stop"
}

Monitoring usage

Pegasus 1.5 publishes Amazon CloudWatch metrics in the AWS/Bedrock namespace under its model ID, including Invocations, OutputTokenCount, InvocationLatency, and InvocationClientErrors. Use them to track volume and latency. Pair them with application inference profiles when you need usage broken down by application.

Use cases

Pegasus 1.5 is designed for organizations that manage large volumes of long-form video:

• News and broadcast archives: Teams can define editorial segments, topics, and speaker changes once, then catalog years of footage that was previously searchable only by transcript keywords.
• Sports broadcasters and leagues: Detect plays, scoring events, and other key moments across full-length game footage, speeding up highlight production.
• Streaming applications: Scene-level metadata supports moment-based recommendations, contextual ad placement, and segment-level brand safety review.

In each case, people remain part of the workflow. Their work shifts from finding and tagging every moment by hand to reviewing and refining the metadata generated by the model.

Input requirements

Pegasus 1.5 supports the following video specifications:

• Duration: Up to 2 hours.
• File size: Up to 2 GB from Amazon S3, or up to 25 MB as a base64 string.
• Resolution: 360×360 to 5184×2160.
• Aspect ratio: Between 1:1 and 2.4:1, or their inverses.
• Formats: FFmpeg-supported formats.

Now available

TwelveLabs Pegasus 1.5 is available today on Amazon Bedrock in the US West (Oregon) AWS Region. Check the full Region list for future updates, and visit the Amazon Bedrock pricing page for pricing information. Pegasus 1.5 is billed through AWS Marketplace, so its charges appear on your AWS bill and in AWS Cost Explorer under TwelveLabs rather than Amazon Bedrock.

To get started:

• Visit the TwelveLabs models page on Amazon Bedrock to learn more about the model family.
• Review the Pegasus 1.5 model card for feature support, Region availability, and sample code.
• Send us feedback at AWS re:Post for Amazon Bedrock or through your usual AWS Support contacts.

We’re excited to see what you build with TwelveLabs Pegasus 1.5 in Amazon Bedrock.

About the authors

Ron Tal

Ron is a Machine Learning Engineer at TwelveLabs, where he leads inference deployment of TwelveLabs models through third-party platforms, including Amazon Bedrock. He has spent more than a decade building machine learning platforms and production inference infrastructure, from founding ML platform teams to serving foundation models at scale.

Shannon Hong

Shannon is a Product Manager at TwelveLabs, where she leads the development of Pegasus, a video language model. She is Research Affiliate of the Oxford Martin School AI Governance Initiative and has conducted research on democratic governance of AI with Open Future Foundation and Creative Commons. She started her career as a Product Manager at Scale AI, where she led technical teams to build data labeling products for autonomous vehicles. Shannon holds an M.B.A from Oxford University, and a B.A. in Development Studies and Data Science at the University of California, Berkeley.

Nate Stecz

Nate is a Senior Partner Development Manager at AWS, where he helps AI ISVs build and execute go-to-market strategies. With 10 years at AWS, Nate focuses on helping AI model providers build, market, and sell their solutions – spanning Amazon Bedrock integrations, AWS Marketplace, and field enablement. In his spare time, Nate enjoys being outdoors. You can connect with or follow Nate on LinkedIn.

Narcisse Zekpa

Narcisse is a Principal Solutions Architect based in Boston. He helps customers in the Northeast U.S. accelerate their business transformation through innovative, and scalable solutions, on the AWS Cloud. He is passionate about enabling organizations to transform their business, using advanced analytics and AI. When Narcisse is not building, he enjoys spending time with his family, traveling, and road racing.

Suheel Farooq

Suheel is a Principal Solutions Architect at AWS, specializing in artificial intelligence, machine learning, and generative AI. He helps Foundation Model Provider customers design, build, modernize, and scale their AI/ML and generative AI workloads on AWS — from bringing foundation models to Amazon Bedrock to optimizing training and inference and building Agentic AI solutions. His experience spans the AWS AI/ML and generative AI portfolio, particularly Amazon Bedrock, Amazon Bedrock AgentCore, and Amazon SageMaker AI. In his free time, Suheel enjoys working out and hiking.

Badri Raghunathan Srikumar

Badri is a Software Development Engineer at Amazon SageMaker AI. He helps bring foundation models from leading model providers to customers by building scalable, secure and observable hosting infrastructure behind them on Amazon SageMaker. Beyond AWS, he is deeply passionate about robotics, especially autonomous vehicles. When not working, Badri likes to go on hikes, swimming and cafe-hopping.