my-project-public

repository

loading code, commits, and activity

repositories

loading repo index

#1	---
#2	title: ElevenLabs
#3	---
#4
#5	Create voice-based conversational AI agents with memory capabilities by integrating ElevenLabs and Mem0. This integration enables persistent, context-aware voice interactions that remember past conversations.
#6
#7	## Overview
#8
#9	In this guide, we'll build a voice agent that:
#10	1. Uses ElevenLabs Conversational AI for voice interaction
#11	2. Leverages Mem0 to store and retrieve memories from past conversations
#12	3. Provides personalized responses based on user history
#13
#14	## Setup and Configuration
#15
#16	Install necessary libraries:
#17
#18	```bash
#19	pip install elevenlabs mem0ai python-dotenv
#20	```
#21
#22	Configure your environment variables:
#23
#24	<Note>You'll need both an ElevenLabs API key and a Mem0 API key to use this integration.</Note>
#25
#26	```bash
#27	# Create a .env file with these variables
#28	AGENT_ID=your-agent-id
#29	USER_ID=unique-user-identifier
#30	ELEVENLABS_API_KEY=your-elevenlabs-api-key
#31	MEM0_API_KEY=your-mem0-api-key
#32	```
#33
#34	## Integration Code Breakdown
#35
#36	Let's break down the implementation into manageable parts:
#37
#38	### 1. Imports and Environment Setup
#39
#40	First, we import required libraries and set up the environment:
#41
#42	```python
#43	import os
#44	import signal
#45	import sys
#46	from mem0 import AsyncMemoryClient
#47
#48	from elevenlabs.client import ElevenLabs
#49	from elevenlabs.conversational_ai.conversation import Conversation
#50	from elevenlabs.conversational_ai.default_audio_interface import DefaultAudioInterface
#51	from elevenlabs.conversational_ai.conversation import ClientTools
#52	```
#53
#54	These imports provide:
#55	- Standard Python libraries for system operations and signal handling
#56	- `AsyncMemoryClient` from Mem0 for memory operations
#57	- ElevenLabs components for voice interaction
#58
#59	### 2. Environment Variables and Validation
#60
#61	Next, we validate the required environment variables:
#62
#63	```python
#64	def main():
#65	# Required environment variables
#66	AGENT_ID = os.environ.get('AGENT_ID')
#67	USER_ID = os.environ.get('USER_ID')
#68	API_KEY = os.environ.get('ELEVENLABS_API_KEY')
#69	MEM0_API_KEY = os.environ.get('MEM0_API_KEY')
#70
#71	# Validate required environment variables
#72	if not AGENT_ID:
#73	sys.stderr.write("AGENT_ID environment variable must be set\n")
#74	sys.exit(1)
#75
#76	if not USER_ID:
#77	sys.stderr.write("USER_ID environment variable must be set\n")
#78	sys.exit(1)
#79
#80	if not API_KEY:
#81	sys.stderr.write("ELEVENLABS_API_KEY not set, assuming the agent is public\n")
#82
#83	if not MEM0_API_KEY:
#84	sys.stderr.write("MEM0_API_KEY environment variable must be set\n")
#85	sys.exit(1)
#86
#87	# Set up Mem0 API key in the environment
#88	os.environ['MEM0_API_KEY'] = MEM0_API_KEY
#89	```
#90
#91	This section:
#92	- Retrieves required environment variables
#93	- Performs validation to ensure required variables are present
#94	- Exits the application with an error message if required variables are missing
#95	- Sets the Mem0 API key in the environment for the Mem0 client to use
#96
#97	### 3. Client Initialization
#98
#99	Initialize both the ElevenLabs and Mem0 clients:
#100
#101	```python
#102	# Initialize ElevenLabs client
#103	client = ElevenLabs(api_key=API_KEY)
#104
#105	# Initialize memory client and tools
#106	client_tools = ClientTools()
#107	mem0_client = AsyncMemoryClient()
#108	```
#109
#110	Here we:
#111	- Create an ElevenLabs client with the API key
#112	- Initialize a ClientTools object for registering function tools
#113	- Create an AsyncMemoryClient instance for Mem0 interactions
#114
#115	### 4. Memory Function Definitions
#116
#117	Define the two key memory functions that will be registered as tools:
#118
#119	```python
#120	# Define memory-related functions for the agent
#121	async def add_memories(parameters):
#122	"""Add a message to the memory store"""
#123	message = parameters.get("message")
#124	await mem0_client.add(
#125	messages=message,
#126	user_id=USER_ID
#127	)
#128	return "Memory added successfully"
#129
#130	async def retrieve_memories(parameters):
#131	"""Retrieve relevant memories based on the input message"""
#132	message = parameters.get("message")
#133
#134	# For Platform API, user_id goes in filters
#135	filters = {"user_id": USER_ID}
#136
#137	# Search for relevant memories using the message as a query
#138	results = await mem0_client.search(
#139	query=message,
#140	filters=filters
#141	)
#142
#143	# Extract and join the memory texts
#144	memories = ' '.join([result["memory"] for result in results.get('results', [])])
#145	print("[ Memories ]", memories)
#146
#147	if memories:
#148	return memories
#149	return "No memories found"
#150	```
#151
#152	These functions:
#153
#154	#### `add_memories`:
#155	- Takes a message parameter containing information to remember
#156	- Stores the message in Mem0 using the `add` method
#157	- Associates the memory with the specific USER_ID
#158	- Returns a success message to the agent
#159
#160	#### `retrieve_memories`:
#161	- Takes a message parameter as the search query
#162	- Sets up filters to only retrieve memories for the current user
#163	- Uses semantic search to find relevant memories
#164	- Joins all retrieved memories into a single text
#165	- Prints retrieved memories to the console for debugging
#166	- Returns the memories or a "No memories found" message if none are found
#167
#168	### 5. Registering Memory Functions as Tools
#169
#170	Register the memory functions with the ElevenLabs ClientTools system:
#171
#172	```python
#173	# Register the memory functions as tools for the agent
#174	client_tools.register("addMemories", add_memories, is_async=True)
#175	client_tools.register("retrieveMemories", retrieve_memories, is_async=True)
#176	```
#177
#178	This allows the ElevenLabs agent to:
#179	- Access these functions through function calling
#180	- Wait for asynchronous results (is_async=True)
#181	- Call these functions by name ("addMemories" and "retrieveMemories")
#182
#183	### 6. Conversation Setup
#184
#185	Configure the conversation with ElevenLabs:
#186
#187	```python
#188	# Initialize the conversation
#189	conversation = Conversation(
#190	client,
#191	AGENT_ID,
#192	# Assume auth is required when API_KEY is set
#193	requires_auth=bool(API_KEY),
#194	audio_interface=DefaultAudioInterface(),
#195	client_tools=client_tools,
#196	callback_agent_response=lambda response: print(f"Agent: {response}"),
#197	callback_agent_response_correction=lambda original, corrected: print(f"Agent: {original} -> {corrected}"),
#198	callback_user_transcript=lambda transcript: print(f"User: {transcript}"),
#199	# callback_latency_measurement=lambda latency: print(f"Latency: {latency}ms"),
#200	)
#201	```
#202
#203	This sets up the conversation with:
#204	- The ElevenLabs client and Agent ID
#205	- Authentication requirements based on API key presence
#206	- DefaultAudioInterface for handling audio I/O
#207	- The client_tools with our memory functions
#208	- Callback functions for:
#209	- Displaying agent responses
#210	- Showing corrected responses (when the agent self-corrects)
#211	- Displaying user transcripts for debugging
#212	- (Commented out) Latency measurements
#213
#214	### 7. Conversation Management
#215
#216	Start and manage the conversation:
#217
#218	```python
#219	# Start the conversation
#220	print(f"Starting conversation with user_id: {USER_ID}")
#221	conversation.start_session()
#222
#223	# Handle Ctrl+C to gracefully end the session
#224	signal.signal(signal.SIGINT, lambda sig, frame: conversation.end_session())
#225
#226	# Wait for the conversation to end and get the conversation ID
#227	conversation_id = conversation.wait_for_session_end()
#228	print(f"Conversation ID: {conversation_id}")
#229
#230
#231	if __name__ == '__main__':
#232	main()
#233	```
#234
#235	This final section:
#236	- Prints a message indicating the conversation has started
#237	- Starts the conversation session
#238	- Sets up a signal handler to gracefully end the session on Ctrl+C
#239	- Waits for the session to end and gets the conversation ID
#240	- Prints the conversation ID for reference
#241
#242	## Memory Tools Overview
#243
#244	This integration provides two key memory functions to your conversational AI agent:
#245
#246	### 1. Adding Memories (`addMemories`)
#247
#248	The `addMemories` tool allows your agent to store important information during a conversation, including:
#249	- User preferences
#250	- Important facts shared by the user
#251	- Decisions or commitments made during the conversation
#252	- Action items to follow up on
#253
#254	When the agent identifies information worth remembering, it calls this function to store it in the Mem0 database with the appropriate user ID.
#255
#256	#### How it works:
#257	1. The agent identifies information that should be remembered
#258	2. It formats the information as a message string
#259	3. It calls the `addMemories` function with this message
#260	4. The function stores the memory in Mem0 linked to the user's ID
#261	5. Later conversations can retrieve this memory
#262
#263	#### Example usage in agent prompt:
#264	```
#265	When the user shares important information like preferences or personal details,
#266	use the addMemories function to store this information for future reference.
#267	```
#268
#269	### 2. Retrieving Memories (`retrieveMemories`)
#270
#271	The `retrieveMemories` tool allows your agent to search for and retrieve relevant memories from previous conversations. The agent can:
#272	- Search for context related to the current topic
#273	- Recall user preferences
#274	- Remember previous interactions on similar topics
#275	- Create continuity across multiple sessions
#276
#277	#### How it works:
#278	1. The agent needs context for the current conversation
#279	2. It calls `retrieveMemories` with the current conversation topic or question
#280	3. The function performs a semantic search in Mem0
#281	4. Relevant memories are returned to the agent
#282	5. The agent incorporates these memories into its response
#283
#284	#### Example usage in agent prompt:
#285	```
#286	At the beginning of each conversation turn, use retrieveMemories to check if we've
#287	discussed this topic before or if the user has shared relevant preferences.
#288	```
#289
#290	## Configuring Your ElevenLabs Agent
#291
#292	To enable your agent to effectively use memory:
#293
#294	1. Add function calling capabilities to your agent in the ElevenLabs platform:
#295	- Go to your agent settings in the ElevenLabs platform
#296	- Navigate to the "Tools" section
#297	- Enable function calling for your agent
#298	- Add the memory tools as described below
#299
#300	2. Add the `addMemories` and `retrieveMemories` tools to your agent with these specifications:
#301
#302	For `addMemories`:
#303	```json
#304	{
#305	"name": "addMemories",
#306	"description": "Stores important information from the conversation to remember for future interactions",
#307	"parameters": {
#308	"type": "object",
#309	"properties": {
#310	"message": {
#311	"type": "string",
#312	"description": "The important information to remember"
#313	}
#314	},
#315	"required": ["message"]
#316	}
#317	}
#318	```
#319
#320	For `retrieveMemories`:
#321	```json
#322	{
#323	"name": "retrieveMemories",
#324	"description": "Retrieves relevant information from past conversations",
#325	"parameters": {
#326	"type": "object",
#327	"properties": {
#328	"message": {
#329	"type": "string",
#330	"description": "The query to search for in past memories"
#331	}
#332	},
#333	"required": ["message"]
#334	}
#335	}
#336	```
#337
#338	3. Update your agent's prompt to instruct it to use these memory functions. For example:
#339
#340	```
#341	You are a helpful voice assistant that remembers past conversations with the user.
#342
#343	You have access to memory tools that allow you to remember important information:
#344	- Use retrieveMemories at the beginning of the conversation to recall relevant context from prior conversations
#345	- Use addMemories to store new important information such as:
#346	* User preferences
#347	* Personal details the user shares
#348	* Important decisions made
#349	* Tasks or follow-ups promised to the user
#350
#351	Before responding to complex questions, always check for relevant memories first.
#352	When the user shares important information, make sure to store it for future reference.
#353	```
#354
#355	## Example Conversation Flow
#356
#357	Here's how a typical conversation with memory might flow:
#358
#359	1. User speaks: "Hi, do you remember my favorite color?"
#360
#361	2. Agent retrieves memories:
#362	```python
#363	# Agent calls retrieve_memories
#364	memories = retrieve_memories({"message": "user's favorite color"})
#365	# If found: "The user's favorite color is blue"
#366	```
#367
#368	3. Agent processes with context:
#369	- If memories found: Prepares a personalized response
#370	- If no memories: Prepares to ask and store the information
#371
#372	4. Agent responds:
#373	- With memory: "Yes, your favorite color is blue!"
#374	- Without memory: "I don't think you've told me your favorite color before. What is it?"
#375
#376	5. User responds: "It's actually green."
#377
#378	6. Agent stores new information:
#379	```python
#380	# Agent calls add_memories
#381	add_memories({"message": "The user's favorite color is green"})
#382	```
#383
#384	7. Agent confirms: "Thanks, I'll remember that your favorite color is green."
#385
#386	## Example Use Cases
#387
#388	- Personal Assistant - Remember user preferences, past requests, and important dates
#389	```
#390	User: "What restaurants did I say I liked last time?"
#391	Agent: retrieves memories "You mentioned enjoying Bella Italia and The Golden Dragon."
#392	```
#393
#394	- Customer Support - Recall previous issues a customer has had
#395	```
#396	User: "I'm having that same problem again!"
#397	Agent: retrieves memories "Is this related to the login issue you reported last week?"
#398	```
#399
#400	- Educational AI - Track student progress and tailor teaching accordingly
#401	```
#402	User: "Let's continue our math lesson."
#403	Agent: retrieves memories "Last time we were working on quadratic equations. Would you like to continue with that?"
#404	```
#405
#406	- Healthcare Assistant - Remember symptoms, medications, and health concerns
#407	```
#408	User: "Have I told you about my allergy medication?"
#409	Agent: retrieves memories "Yes, you mentioned you're taking Claritin for your pollen allergies."
#410	```
#411
#412	## Troubleshooting
#413
#414	- Missing API Keys:
#415	- Error: "API_KEY environment variable must be set"
#416	- Solution: Ensure all environment variables are set correctly in your .env file or system environment
#417
#418	- Connection Issues:
#419	- Error: "Failed to connect to API"
#420	- Solution: Check your network connection and API key permissions. Verify the API keys are valid and have the necessary permissions.
#421
#422	- Empty Memory Results:
#423	- Symptom: Agent always responds with "No memories found"
#424	- Solution: This is normal for new users. The memory database builds up over time as conversations occur. It's also possible your query isn't semantically similar to stored memories - try different phrasing.
#425
#426	- Agent Not Using Memories:
#427	- Symptom: The agent retrieves memories but doesn't incorporate them in responses
#428	- Solution: Update the agent's prompt to explicitly instruct it to use the retrieved memories in its responses
#429
#430	## Conclusion
#431
#432	By integrating ElevenLabs Conversational AI with Mem0, you can create voice agents that maintain context across conversations and provide personalized responses based on user history. This powerful combination enables:
#433
#434	- More natural, context-aware conversations
#435	- Personalized user experiences that improve over time
#436	- Reduced need for users to repeat information
#437	- Long-term relationship building between users and AI agents
#438
#439	<CardGroup cols={2}>
#440	<Card title="LiveKit Integration" icon="video" href="/integrations/livekit">
#441	Build real-time voice and video agents
#442	</Card>
#443	<Card title="Pipecat Integration" icon="waveform" href="/integrations/pipecat">
#444	Create voice-first AI applications
#445	</Card>
#446	</CardGroup>
#447
#448

z6Mkq5mY3JWtxoxUobWcfNHm7AkRubgSWEZTkBVqZXJviFZ5/my-project-public