Skip to content

Commit

Permalink
Anthropic: add support through AWS/Bedrock
Browse files Browse the repository at this point in the history
  • Loading branch information
@enricoros
enricoros committed Oct 24, 2023
1 parent fcf5316 commit b526998
Show file tree
Hide file tree
Showing 5 changed files with 37 additions and 18 deletions.
20 changes: 11 additions & 9 deletions docs/environment-variables.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ OPENAI_API_ORG_ID=
AZURE_OPENAI_API_ENDPOINT=
AZURE_OPENAI_API_KEY=
ANTHROPIC_API_KEY=
ANTHROPIC_API_HOST=
OPENROUTER_API_KEY=

# Model Observability: Helicone
Expand Down Expand Up @@ -63,15 +64,16 @@ first time (or update it on a later stage).
The following variables when set will enable the corresponding LLMs on the server-side, without
requiring the user to enter an API key

| Variable | Description | Required |
|-----------------------------|----------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------|
| `OPENAI_API_KEY` | API key for OpenAI | Recommended |
| `OPENAI_API_HOST` | Changes the backend host for the OpenAI vendor, to enable platforms such as Helicone and CloudFlare AI Gateway | Optional |
| `OPENAI_API_ORG_ID` | Sets the "OpenAI-Organization" header field to support organization users | Optional |
| `AZURE_OPENAI_API_ENDPOINT` | Azure OpenAI endpoint - host only, without the path | Optional, but if set `AZURE_OPENAI_API_KEY` must also be set |
| `AZURE_OPENAI_API_KEY` | Azure OpenAI API key, see [config-azure-openai.md](config-azure-openai.md) | Optional, but if set `AZURE_OPENAI_API_ENDPOINT` must also be set |
| `ANTHROPIC_API_KEY` | The API key for Anthropic | Optional |
| `OPENROUTER_API_KEY` | The API key for OpenRouter | Optional |
| Variable | Description | Required |
|-----------------------------|-------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------|
| `OPENAI_API_KEY` | API key for OpenAI | Recommended |
| `OPENAI_API_HOST` | Changes the backend host for the OpenAI vendor, to enable platforms such as Helicone and CloudFlare AI Gateway | Optional |
| `OPENAI_API_ORG_ID` | Sets the "OpenAI-Organization" header field to support organization users | Optional |
| `AZURE_OPENAI_API_ENDPOINT` | Azure OpenAI endpoint - host only, without the path | Optional, but if set `AZURE_OPENAI_API_KEY` must also be set |
| `AZURE_OPENAI_API_KEY` | Azure OpenAI API key, see [config-azure-openai.md](config-azure-openai.md) | Optional, but if set `AZURE_OPENAI_API_ENDPOINT` must also be set |
| `ANTHROPIC_API_KEY` | The API key for Anthropic | Optional |
| `ANTHROPIC_API_HOST` | Changes the backend host for the Anthropic vendor, to enable platforms such as [config-aws-bedrock.md](config-aws-bedrock.md) | Optional |
| `OPENROUTER_API_KEY` | The API key for OpenRouter | Optional |

### Model Observability: Helicone

Expand Down
1 change: 1 addition & 0 deletions src/common/types/env.d.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ declare namespace NodeJS {

// LLM: Anthropic
ANTHROPIC_API_KEY?: string;
ANTHROPIC_API_HOST?: string;

// LLM: OpenRouter
OPENROUTER_API_KEY: string;
Expand Down
15 changes: 10 additions & 5 deletions src/modules/llms/transports/server/anthropic.router.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ import { fetchJsonOrTRPCError } from '~/server/api/trpc.serverutils';

import { LLM_IF_OAI_Chat } from '../../store-llms';

import { openAIChatGenerateOutputSchema, openAIHistorySchema, openAIModelSchema } from './openai.router';
import { fixupHost, openAIChatGenerateOutputSchema, openAIHistorySchema, openAIModelSchema } from './openai.router';
import { listModelsOutputSchema, ModelDescriptionSchema } from './server.common';

import { AnthropicWire } from './anthropic.wiretypes';
Expand All @@ -17,6 +17,7 @@ import { AnthropicWire } from './anthropic.wiretypes';
export const anthropicAccessSchema = z.object({
dialect: z.literal('anthropic'),
anthropicKey: z.string().trim(),
anthropicHost: z.string().trim().nullable(),
heliconeKey: z.string().trim().nullable(),
});
export type AnthropicAccessSchema = z.infer<typeof anthropicAccessSchema>;
Expand Down Expand Up @@ -135,6 +136,7 @@ async function anthropicPOST<TOut, TPostBody>(access: AnthropicAccessSchema, bod
}

const DEFAULT_ANTHROPIC_HOST = 'api.anthropic.com';
const DEFAULT_HELICONE_ANTHROPIC_HOST = 'anthropic.hconeai.com';

export function anthropicAccess(access: AnthropicAccessSchema, apiPath: string): { headers: HeadersInit, url: string } {
// API version
Expand All @@ -146,13 +148,16 @@ export function anthropicAccess(access: AnthropicAccessSchema, apiPath: string):
throw new Error('Missing Anthropic API Key. Add it on the UI (Models Setup) or server side (your deployment).');

// API host
let anthropicHost = `https://${DEFAULT_ANTHROPIC_HOST}`;
let anthropicHost = fixupHost(access.anthropicHost || process.env.ANTHROPIC_API_HOST || DEFAULT_ANTHROPIC_HOST, apiPath);

// https://docs.helicone.ai/getting-started/integration-method/anthropic
// Helicone for Anthropic
// https://docs.helicone.ai/getting-started/integration-method/anthropic
const heliKey = access.heliconeKey || process.env.HELICONE_API_KEY || false;
if (heliKey)
anthropicHost = 'https://anthropic.hconeai.com';
if (heliKey) {
if (!anthropicHost.includes(DEFAULT_ANTHROPIC_HOST) && !anthropicHost.includes(DEFAULT_HELICONE_ANTHROPIC_HOST))
throw new Error(`The Helicone Anthropic Key has been provided, but the host is set to custom. Please fix it in the Models Setup page.`);
anthropicHost = `https://${DEFAULT_HELICONE_ANTHROPIC_HOST}`;
}

return {
headers: {
Expand Down
15 changes: 12 additions & 3 deletions src/modules/llms/vendors/anthropic/AnthropicSourceSetup.tsx
View file
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ export function AnthropicSourceSetup(props: { sourceId: DModelSourceId }) {
useSourceSetup(props.sourceId, ModelVendorAnthropic.getAccess);

// derived state
const { anthropicKey, heliconeKey } = access;
const { anthropicKey, anthropicHost, heliconeKey } = access;

const needsUserKey = !ModelVendorAnthropic.hasServerKey;
const keyValid = isValidAnthropicApiKey(anthropicKey);
Expand All @@ -46,7 +46,7 @@ export function AnthropicSourceSetup(props: { sourceId: DModelSourceId }) {
return <Box sx={{ display: 'flex', flexDirection: 'column', gap: settingsGap }}>

<FormInputKey
id='anthropic-key' label='Anthropic API Key'
id='anthropic-key' label={!!anthropicHost ? 'API Key' : 'Anthropic API Key'}
rightLabel={<>{needsUserKey
? !anthropicKey && <Link level='body-sm' href='https://www.anthropic.com/earlyaccess' target='_blank'>request Key</Link>
: '✔️ already set in server'
Expand All @@ -58,7 +58,16 @@ export function AnthropicSourceSetup(props: { sourceId: DModelSourceId }) {
/>

{advanced.on && <FormTextField
title='Helicone Key'
title='API Host'
description={<>e.g., <Link level='body-sm' href='https://github.com/enricoros/big-agi/blob/main/docs/config-aws-bedrock.md' target='_blank'>bedrock-claude</Link></>}
placeholder='deployment.service.region.amazonaws.com'
isError={false}
value={anthropicHost || ''}
onChange={text => updateSetup({ anthropicHost: text })}
/>}

{advanced.on && <FormTextField
title='Helicone Key' disabled={!!anthropicHost}
description={<>Generate <Link level='body-sm' href='https://www.helicone.ai/keys' target='_blank'>here</Link></>}
placeholder='sk-...'
value={heliconeKey || ''}
Expand Down
4 changes: 3 additions & 1 deletion src/modules/llms/vendors/anthropic/anthropic.vendor.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -13,10 +13,11 @@ import { AnthropicSourceSetup } from './AnthropicSourceSetup';


// special symbols
export const isValidAnthropicApiKey = (apiKey?: string) => !!apiKey && apiKey.startsWith('sk-') && apiKey.length > 40;
export const isValidAnthropicApiKey = (apiKey?: string) => !!apiKey && (apiKey.startsWith('sk-') ? apiKey.length > 40 : apiKey.length >= 40);

export interface SourceSetupAnthropic {
anthropicKey: string;
anthropicHost: string;
heliconeKey: string;
}

Expand All @@ -37,6 +38,7 @@ export const ModelVendorAnthropic: IModelVendor<SourceSetupAnthropic, LLMOptions
getAccess: (partialSetup): AnthropicAccessSchema => ({
dialect: 'anthropic',
anthropicKey: partialSetup?.anthropicKey || '',
anthropicHost: partialSetup?.anthropicHost || null,
heliconeKey: partialSetup?.heliconeKey || null,
}),
callChatGenerate(llm, messages: VChatMessageIn[], maxTokens?: number): Promise<VChatMessageOut> {
Expand Down

1 comment on commit b526998

@vercel
Copy link

@vercel vercel bot commented on b526998 Oct 24, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Successfully deployed to the following URLs:

big-agi – ./

get.big-agi.com
big-agi-enricoros.vercel.app
big-agi-git-main-enricoros.vercel.app

Please sign in to comment.