跳到正文
AWS Machine Learning Blog· Shubham Gupta·· 6 小时前AI 评分38

ArisGlobal 如何在 Amazon Bedrock 上扩展生成式 AI 推理

How ArisGlobal scaled generative AI inference on Amazon Bedrock

AI 导读

ArisGlobal 通过调整 Amazon Bedrock 提示词缓存顺序,将共享提示词前缀而非源文档放入缓存,从而降低有效输入成本,支撑更多租户而不带来成本等比增长。

正文 · 原文

ArisGlobal, a provider of life sciences technology spanning pharmacovigilance, regulatory, medical affairs, clinical, and quality functions, developed NavaX, an AI engine purpose-built for life sciences on Amazon Bedrock, to automate adverse-event case processing. ArisGlobal chose Amazon Bedrock for managed access to foundation models, EU geographic cross-Region inference, and managed prompt caching. As enterprise customers added distinct compliance requirements, the shared prompt prefix grew by about 150% over 6 months, while per-case inference costs increased by more than 200%.

ArisGlobal first cached each source document because it was the largest part of a request. That left the growing shared prompt prefix outside the cache. By reversing the order and caching the shared prefix instead, ArisGlobal reduced effective input cost and supported additional tenants without proportional cost growth.

Who is ArisGlobal?

ArisGlobal is the creator of LifeSphere, a unified SaaS platform for pharmacovigilance, regulatory affairs, quality, and medical information. LifeSphere serves pharmaceutical companies globally, including multiple Top 20 pharma organizations by revenue (source: ArisGlobal press release). The entire system runs on AWS.

LifeSphere NavaX is ArisGlobal’s AI engine that transforms drug safety from a system of record into a system of strategic advantage. ArisGlobal productionized their first generative AI solution on Amazon Bedrock with Anthropic’s Claude in March 2025. According to ArisGlobal’s internal production telemetry, they have processed over 1 million real transactions in regulated pharmaceutical environments, handling billions of tokens across multiple enterprise tenants. Model availability varies by Region. See Model support by AWS Region.

How NavaX processes an adverse event case

When a pharmaceutical company receives an adverse-event report, typically a multipage PDF describing a patient’s reaction to a medication, NavaX’s Dynamic Data Extraction (DDE) agent extracts around 130 fields. These fields include patient demographics, suspect drugs, reporter information, event timelines, seriousness criteria, and regulatory classifications. Because this content includes protected health information, ArisGlobal applies responsible-AI controls as a standard part of its production deployment. It draws on capabilities such as Amazon Bedrock Guardrails to filter content and detect or mask sensitive information. Extracted fields are also validated against the schema and grounded in the source document.

This extraction happens across multiple sequential requests to the model. Each request focuses on a different section of the case form, and each carries:

  • System instructions — model behavior rules, output format specifications, and regulatory compliance constraints.
  • Customer-specific rules — per-tenant extraction logic, field mappings, and business rules that differ by pharmaceutical company.
  • Extraction schema — around 130 target fields with types, validation rules, and few-shot examples.
  • The source document — the adverse event PDF (often 10–200 pages).

Two inputs repeat at different scopes. The shared prompt prefix includes the system rules, extraction schema, and few-shot examples. It repeats across requests, cases, and tenants using the same agent. The source document repeats only for the requests associated with one case.

The prompt bloat problem

When NavaX launched with its first enterprise tenant, the system prompt was lean. As ArisGlobal onboarded additional pharmaceutical companies, each brought specific requirements:

  • Custom field mappings (Tenant A maps reaction onset differently from Tenant B).
  • Regulatory jurisdiction rules (EU ICSR compared with FDA MedWatch and PMDA formatting).
  • Company-specific business logic for seriousness assessment and causality.
  • Per-tenant few-shot examples calibrated to each company’s historical cases.

Over 6 months, the shared prompt prefix grew from approximately 72,000 to 180,000 tokens, a 150% increase. With multiple sequential requests per case, per-case inference costs increased by more than 200%.

The prefix grew with every tenant and was reprocessed on every request, so prompt growth compounded with transaction volume.

Table 1 compares the original document-first request order with the production prompt-first design.

Aspect First approach Production design
Request order Per-case document, cache checkpoint, shared prompt prefix Shared prompt prefix, cache checkpoint, per-case document
Cached prefix The document, which changes for every case The shared prompt prefix, which repeats across requests
Reuse scope Only the remaining requests for one case Requests across cases and tenants using the same agent
Cost effect The growing prompt is repeatedly processed at the full input rate Fleet-wide prompt reuse supports sublinear cost growth
EU routing EU geographic cross-Region inference remains unchanged EU geographic cross-Region inference remains unchanged

Amazon Bedrock prompt caching strategies

Amazon Bedrock prompt caching reduces latency and input token costs by caching a contiguous prompt prefix from the start of a request through a cache checkpoint. A later request with the same prefix can read cached tokens at a reduced rate instead of processing them again. Cache entries have a time-to-live (TTL) that resets after a successful cache hit.

Caching is prefix-based rather than block-based. Request sections are evaluated in sequence—tools, system, then messages—so changing an earlier section invalidates matching content that follows it.

Cache checkpoints (explicit control)

With explicit cache checkpoints, you can choose the exact prefix boundary. Supported Claude models allow up to four checkpoints per request.

  • Place each checkpoint after a stable, reusable prefix.
  • Meet the model-specific minimum prefix size. For Claude Sonnet 4.5 the minimum is 1,024 tokens (see supported models and Regions).
  • Checkpoints are evaluated in this order: tools, system, and messages.
  • Changing an earlier section invalidates cache matches for later sections.
  • The default TTL is 5 minutes. Select models support a 1-hour TTL by setting "ttl": "1h". Longer-TTL entries must precede shorter-TTL entries.

Simplified cache management (automated)

With simplified cache management, you place one checkpoint after static content. Amazon Bedrock searches backward from that point for the longest matching prefix, up to about 20 content blocks.

  • One checkpoint can locate the longest matching prefix automatically.
  • Use explicit checkpoints when stable content extends beyond the lookback range.

What ArisGlobal cached, and what they deliberately did not

ArisGlobal uses one explicit checkpoint immediately after the shared prompt prefix, which is sent first. The per-case document follows the checkpoint and is processed on each request.

This reverses the initial request order. Because only a prefix can be cached, ordering determines which content can be reused.

The launch design: Prioritizing document reuse

The first design put the 10–200 page source document at the start of each request and placed the checkpoint after it. The shared prompt prefix and section-specific question followed the checkpoint.

{
  "modelId": "anthropic.claude-sonnet-4-5-20250929-v1:0",
  "messages": [
    {
      "role": "user",
      "content": [
        {
          "text": "[Full adverse event PDF content]..." // <-- Document sent FIRST
        },
        {
          "cachePoint": {"type": "default"} // <-- Checkpoint right after the document
        },
        {
          "text": "[System instructions + compliance rules + schema +
          few-shot examples + customer-specific prompt]..."
          // <-- After the checkpoint: never cached
        },
        {
          "text": "Extract fields for Section 3: Suspect Drug Information"
        }
      ]
    }
  ]
}

The design reused the document within one case: the first request wrote it to cache, and later requests for that case read it at the reduced rate.

The next case used a different document and required a new cache write, so reuse never extended across cases or tenants.

More importantly, the shared prompt prefix remained after the checkpoint and was processed at the full input rate on every request. The content driving the cost curve was excluded from the cache.

The evolved production design: Prioritizing shared-prefix reuse

The production design changed the order rather than adding another checkpoint: the shared prompt prefix comes first, the checkpoint follows it, and the document comes last.

The same checkpoint now creates a reusable fleet-wide cache entry:

  • Before — the cached document prefix was unique to one case and the shared prompt was never cached.
  • After — the cached shared prompt prefix repeats across requests, cases, and tenants using the same agent.

ArisGlobal gave up limited document reuse within one case and gained prompt reuse across the fleet. The shared prefix is also the part that grows as customers are added.

Updated implementation

{
  "modelId": "anthropic.claude-sonnet-4-5-20250929-v1:0",
  "system": [
    {
      "text": "[Shared system instructions + compliance rules +
      base extraction schema + few-shot examples]..."
    },
    {
      "cachePoint": {"type": "default"} // <-- Checkpoint after the stable prompt
    }
  ],
  "messages": [
    {
      "role": "user",
      "content": [
        {
          "text": "[Full adverse event PDF content — sits after the
          checkpoint, not cached]..."
        },
        {
          "text": "Extract fields for Section 5: Event Assessment"
          // <-- Only this changes per request
        }
      ]
    }
  ]
}

For most active accounts, cases arrive about every 5 minutes per tenant and multiple tenants run concurrently. This traffic keeps the shared prefix warm during business hours because each cache hit resets the TTL.

Caching both the prompt and the document

A second checkpoint after the document would cache a combined prefix consisting of the shared prompt and that case’s document. The entry would still be unique to one case.

ArisGlobal therefore prioritized the prefix whose reads amortize across the fleet. The shared prompt prefix scales across cases and tenants. The per-case document does not.

In production, 65% of input tokens on a typical request are served from the shared prompt cache. Figure 1 shows cache reads consistently exceeding cache writes.

Amazon CloudWatch chart with cache reads about 10 times cache writes during production traffic

Figure 1: Amazon CloudWatch prompt-cache usage. The blue cache-read series rises to roughly 10 times the cache-write series during periods of production traffic

Results

Caching the shared prompt prefix produced the following results while the EU geographic cross-Region inference profile remained in place:

  • 65% of input tokens served from cache at the reduced read rate.
  • Up to 25% lower end-to-end latency on cached production requests.
  • Approximately 50% lower effective input cost.
  • Approximately 40% lower overall monthly inference spend at more than 19 billion tokens per month.
  • Around 15% more enterprise tenants onboarded without proportional cost increases.
  • EU geographic routing unchanged. Prompt caching operates within the existing inference configuration.

Line chart showing lower, more stable end-to-end invocation latency after prompt-cache adoption

Figure 2: End-to-end invocation latency. The chart shows a lower and more stable latency range after prompt-cache adoption

Chart showing sustained invocation count while inference spend grew less than proportionally

Figure 3: Invocation count. The chart shows sustained production request volume while inference spend grew less than proportionally

Based on ArisGlobal’s production billing data, serving 65% of input tokens from cache reduced effective input cost by approximately 50%. At more than 19 billion tokens per month, this produced approximately 40% overall monthly savings in inference spend.

Note: All performance and cost figures in this section reflect ArisGlobal’s production metrics as reported to AWS. Results will vary based on prompt structure, traffic patterns, and cache hit rates.

Production learnings

ArisGlobal’s experience offers guidance for many workloads with large, repeated prompt prefixes:

Cache what repeats, not what is largest. A large per-case document offers limited reuse, while a byte-identical shared prompt prefix can serve requests across the fleet.

Put stable content first. Order tool definitions, system instructions, schemas, and examples before the checkpoint. Place tenant identifiers, case data, and the per-request question afterward.

Use fleet traffic to amortize cache writes. Requests from tenants using the same agent can read the same prefix, and cache hits reset the TTL. For sporadic traffic, evaluate the 1-hour TTL on supported models.

Reduce avoidable calls. Each call still processes the uncached document, so bundle sequential questions when output limits and schema requirements permit.

Keep the prefix byte-identical. Timestamps, request IDs, tenant names, and reordered tool definitions can cause misses. Move every dynamic element after the checkpoint.

Monitor the production baseline. Track cache reads, cache writes, and total input tokens in CloudWatch. Alert on a sustained drop below the normal 65% cache-hit baseline rather than using an incompatible 80% threshold.

Check model-specific limits. Prefix minimums and TTL options vary by model. Claude Sonnet 4.5 requires at least 1,024 tokens for a cache checkpoint.

Conclusion

ArisGlobal combined EU geographic cross-Region inference with prompt caching aimed at the right content. Caching the stable shared prefix instead of the per-case document reduced effective input cost by approximately 50%, contributed to approximately 40% overall monthly inference savings, and supported around 15% more enterprise tenants without proportional cost increases.

The key lesson is that cache value depends on reuse, not size. For multi-tenant agentic workloads, a repeated shared prompt prefix can be more valuable than a larger document that changes for every case.

To learn more, see the Amazon Bedrock prompt caching documentation, the cross-Region inference guide, and the AWS blog post Effectively use prompt caching on Amazon Bedrock.


About the authors

Hitesh Sharma

Hitesh Sharma

Hitesh is a technology leader with 25 years of experience in software engineering and product innovation, including 20 years in the Life Sciences industry. He specializes in applying Generative AI, AI/ML, and cloud-native technologies to solve complex challenges across Pharmacovigilance, Drug Safety, regulatory compliance, and patient safety. Hitesh has led the development of scalable AI platforms that deliver business value. He is passionate about GenAI, innovation, and building high-performing teams to advance Life Sciences technology.

Rakesh Kumar Sharma

Rakesh Kumar Sharma

Rakesh Kumar Sharma is an AI/ML leader and researcher with deep expertise in Generative AI, Agentic AI, and enterprise AI architecture. He specializes in designing and deploying scalable AI solutions using LLMs, AI agents, RAG, and modern ML frameworks, enabling intelligent automation, knowledge-driven decision-making, and production-ready AI platforms. His work focuses on architecting agentic systems that combine reasoning, retrieval, tools, and autonomous workflows to solve complex enterprise challenges.

Shubham Gupta

Shubham Gupta

Shubham is a Solutions Architect at AWS based in Bengaluru, India, supporting independent software vendors (ISVs). He works with engineering and leadership teams to design, build, and run their products on AWS, from first architecture to production, with a deep focus on generative AI and resilience at scale. Outside of work, Shubham is an avid hiker who brings the same preparation and persistence from the trail to building systems that last.

Vibhu Pareek

Vibhu is a Sr. Solutions Architect at AWS. Since 2016, he has guided customers in cloud adoption using well-architected, repeatable patterns. With his specialization in databases, data analytics, and AI, he thrives on transforming complex challenges into innovative solutions. Outside work, he enjoys short treks and sports like badminton, football, and swimming.

来源:AWS Machine Learning Blog · aws.amazon.com