llama-2-lmi/llama-2-70b-chat/2-inference-llama-2-70b-chat-lmi-response-streaming.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "7b95befd-d31a-444d-9ce4-249fa222282c",
   "metadata": {},
   "source": [
    "# SageMaker Realtime Inference LLama 2 70B Chat LMI Model using response streaming\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "360f7c94-3bab-4b45-95cf-8c6604443948",
   "metadata": {},
   "source": [
    "## Set up development environment\n",
    "Upgrade `pip` and install the latest version of `sagemaker` and `boto3` packages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "725d4373-0362-47e8-ad52-d1a973cba496",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!pip install -Uq pip\n",
    "!pip install -Uq sagemaker boto3"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f813b50c-265e-4183-972d-30cf638b2bb0",
   "metadata": {},
   "source": [
    "Restore the `endpoint_name` from the deployment notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cdba3b70-84f0-412a-a341-4fcf4c86ce35",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%store -r \\\n",
    "endpoint_name \\\n",
    "bucket \\\n",
    "s3_prefix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "93b83714-6720-4ab9-87ad-e074583fa0ad",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "endpoint_name"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fc4caf08-a625-4061-b0b9-e4ccbf9b86a2",
   "metadata": {},
   "source": [
    "## Setup Code for Realtime Inference for response streaming\n",
    "\n",
    "Amazon SageMaker's new `InvokeEndpointWithResponseStream` API allows developers to stream responses back from SageMaker models, which can help to improve customer satisfaction by reducing the perceived latency. This is especially important for applications built with generative AI models, where immediate processing is more important than waiting for the entire response.\n",
    "\n",
    "The `LineIterator` helper class will parse the response stream received from the inference via `invoke_endpoint_with_response_stream` API made to Llama 2 Chat model running on LMI container.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2b7bd61a-ede5-493e-84f1-5b20dd5d0b72",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import sagemaker\n",
    "import json\n",
    "import boto3\n",
    "\n",
    "sagemaker_runtime = boto3.client('sagemaker-runtime')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "808f51e0-bedf-43c5-b7d3-0208459478b9",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def get_realtime_response_stream(sagemaker_runtime, endpoint_name, payload):\n",
    "    response_stream = sagemaker_runtime.invoke_endpoint_with_response_stream(\n",
    "        EndpointName=endpoint_name,\n",
    "        Body=json.dumps(payload), \n",
    "        ContentType=\"application/json\",\n",
    "        CustomAttributes='accept_eula=true'\n",
    "    )\n",
    "    return response_stream"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "45275274-427d-4755-967e-1fc1e5b5f59c",
   "metadata": {},
   "source": [
    "The following `print_response_stream` helper function will parse the response stream received from the inference request made via `InvokeEndpointWithResponseStream` API."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b8a37038-0101-4f33-b927-34fdece0029b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import sys, os\n",
    "module_path = \"..\"\n",
    "sys.path.append(os.path.abspath(module_path))\n",
    "from utils.LineIterator import LineIterator\n",
    "\n",
    "def print_response_stream(response_stream):\n",
    "    event_stream = response_stream.get('Body')\n",
    "    for line in LineIterator(event_stream):\n",
    "        print(line, end='')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d6c5510-f55c-46f7-acb8-724a2d23a637",
   "metadata": {},
   "source": [
    "## Prepare Prompt and instructions"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9cc4ee14-b373-40d9-befe-db88f4506adf",
   "metadata": {},
   "source": [
    "To prompt Llama 2 Chat, you need to have following prompt format\n",
    "\n",
    "```\n",
    "<s>[INST] <<SYS>>\n",
    "{{ system_prompt }}\n",
    "<</SYS>>\n",
    "\n",
    "{{ user_message }} [/INST]\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8d9f9cc9-0e40-49d2-b0ae-7ce322aa169c",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def build_llama2_prompt(instructions):\n",
    "    stop_token = \"</s>\"\n",
    "    start_token = \"<s>\"\n",
    "    startPrompt = f\"{start_token}[INST] \"\n",
    "    endPrompt = \" [/INST]\"\n",
    "    conversation = []\n",
    "    for index, instruction in enumerate(instructions):\n",
    "        if instruction[\"role\"] == \"system\" and index == 0:\n",
    "            conversation.append(f\"<<SYS>>\\n{instruction['content']}\\n<</SYS>>\\n\\n\")\n",
    "        elif instruction[\"role\"] == \"user\":\n",
    "            conversation.append(instruction[\"content\"].strip())\n",
    "        else:\n",
    "            conversation.append(f\"{endPrompt} {instruction['content'].strip()} {stop_token}{startPrompt}\")\n",
    "\n",
    "    return startPrompt + \"\".join(conversation) + endPrompt\n",
    "\n",
    "def get_instructions(user_content):\n",
    "    \n",
    "    '''\n",
    "    Note: We are creating a fresh user content everytime by initializing instructions for every user_content.\n",
    "    This is to avoid past user_content when you are inferencing multiple times with new ask everytime.\n",
    "    ''' \n",
    "    \n",
    "    system_content = '''\n",
    "    You are a friendly and knowledgeable email marketing agent, Mr.MightyMark, working at AnyCompany. \n",
    "    Your goal is to send email to subscribers to help them understand the value of the new product and generate excitement for the launch.\n",
    "\n",
    "    Here are some tips on how to achieve your goal:\n",
    "\n",
    "    Be Professional. Address each subscriber by name and use a Professional tone.\n",
    "    Be informative. Explain the key features and benefits of the new product in a clear and concise way.\n",
    "    Be persuasive. Highlight how the new product can solve the subscriber's problems or improve their lives.\n",
    "\n",
    "    By following these tips, you can use email marketing to help your company launch a successful software product.\n",
    "    '''\n",
    "\n",
    "    instructions = [\n",
    "        { \"role\": \"system\",\"content\": f\"{system_content} \"},\n",
    "    ]\n",
    "    \n",
    "    instructions.append({\"role\": \"user\", \"content\": f\"{user_content}\"})\n",
    "    \n",
    "    return instructions\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "48f0cfda-6414-4623-b370-169799a2b934",
   "metadata": {},
   "source": [
    "## Inference the Llama 2 Chat LMI SageMaker endpoint for Streaming Response"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71b73f44-99f2-4aa5-a230-9b2f7dbf35a2",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Inference example 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0be1e0b7-bc14-446b-80f5-f55b36569d5f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "user_ask_1 = f'''\n",
    "AnyCompany recently announced new service launch named AnyCloud Internet Service.\n",
    "Write a short email about the product launch with Call to action to Alice Smith, whose email is alice.smith@example.com\n",
    "Mention the Coupon Code: EARLYB1RD to get 20% for 1st 3 months.\n",
    "'''\n",
    "\n",
    "instructions = get_instructions(user_ask_1)\n",
    "prompt = build_llama2_prompt(instructions)\n",
    "print(prompt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "316f38b8-b5c4-47de-9d8c-9d7df202977d",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "inference_params = {\n",
    "        \"do_sample\": True,\n",
    "        \"top_p\": 0.6,\n",
    "        \"temperature\": 1.0,\n",
    "        \"top_k\": 50,\n",
    "        \"max_new_tokens\": 512,\n",
    "        \"repetition_penalty\": 1.03,\n",
    "        \"stop\": [\"</s>\"],\n",
    "        \"return_full_text\": False\n",
    "    }\n",
    "payload = {\n",
    "    \"inputs\":  prompt,\n",
    "    \"parameters\": inference_params,\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f7a1b380-a4dc-4569-adde-9d20c0feb846",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "resp = get_realtime_response_stream(sagemaker_runtime, endpoint_name, payload)\n",
    "print_response_stream(resp)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "894ea4ff-c200-4344-8643-fb63a9da700e",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Inference example 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a9463df4-9acc-43d8-842d-ba5473f7b1f2",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "user_ask_2 = f'''\n",
    "AnyCompany recently announced new service launch named AnyCloud Streaming Service.\n",
    "Write a short email about the product launch with Call to action to Alice Smith, whose email is alice.smith@example.com\n",
    "Mention the Coupon Code: STREAM2DREAM to get 15% for 1st 6 months.\n",
    "'''\n",
    "\n",
    "instructions = get_instructions(user_ask_2)\n",
    "prompt = build_llama2_prompt(instructions)\n",
    "inference_params = {\n",
    "        \"do_sample\": True,\n",
    "        \"top_p\": 0.6,\n",
    "        \"temperature\": 0.9,\n",
    "        \"top_k\": 50,\n",
    "        \"max_new_tokens\": 512,\n",
    "        \"return_full_text\": False,\n",
    "    }\n",
    "payload = {\n",
    "    \"inputs\":  prompt,\n",
    "    \"parameters\": inference_params,\n",
    "}\n",
    "resp = get_realtime_response_stream(sagemaker_runtime, endpoint_name, payload)\n",
    "print_response_stream(resp)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de53188b-d0b9-45ca-bf4d-82d273374b0f",
   "metadata": {},
   "source": [
    "## Cleanup\n",
    "\n",
    "Change the value of the variable *cleanup* to `True` to perform clean up activity."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4fdda55a-19ec-41f2-8613-fd01c7bc7e67",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "cleanup = False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "21424a15-233a-4f1f-820b-1c10c3a5c84a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "sm_client = boto3.client('sagemaker')\n",
    "endpoint = sm_client.describe_endpoint(EndpointName=endpoint_name)\n",
    "endpoint_config_name = endpoint['EndpointConfigName']\n",
    "endpoint_config = sm_client.describe_endpoint_config(EndpointConfigName=endpoint_config_name)\n",
    "model_name = endpoint_config['ProductionVariants'][0]['ModelName']\n",
    "\n",
    "print(f\"\"\"\n",
    "About to delete the following sagemaker resources:\n",
    "Endpoint: {endpoint_name}\n",
    "Endpoint Config: {endpoint_config_name}\n",
    "Model: {model_name}\n",
    "\"\"\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2b177dd5-5353-4a93-85a8-4b860d49e6fd",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "if cleanup:\n",
    "    # delete endpoint\n",
    "    sm_client.delete_endpoint(EndpointName=endpoint_name)\n",
    "    # delete endpoint config\n",
    "    sm_client.delete_endpoint_config(EndpointConfigName=endpoint_config_name)\n",
    "    # delete model\n",
    "    sm_client.delete_model(ModelName=model_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2cf89f80-8145-4762-94f8-8d8e90db7d21",
   "metadata": {},
   "outputs": [],
   "source": [
    "if cleanup:\n",
    "    # delete model artifacts from s3\n",
    "    s3 = boto3.resource('s3')\n",
    "    s3_bucket = s3.Bucket(bucket)\n",
    "    s3_bucket.objects.filter(Prefix=s3_prefix).delete()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0f1060f8-52ac-4210-87c2-2aff275bc7a4",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "availableInstances": [
   {
    "_defaultOrder": 0,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.t3.medium",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 1,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.t3.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 2,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.t3.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 3,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.t3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 4,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 5,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 6,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 7,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 8,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 9,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 10,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 11,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 12,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5d.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 13,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5d.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 14,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5d.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 15,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5d.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 16,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5d.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 17,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5d.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 18,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5d.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 19,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 20,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": true,
    "memoryGiB": 0,
    "name": "ml.geospatial.interactive",
    "supportedImageNames": [
     "sagemaker-geospatial-v1-0"
    ],
    "vcpuNum": 0
   },
   {
    "_defaultOrder": 21,
    "_isFastLaunch": true,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.c5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 22,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.c5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 23,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.c5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 24,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.c5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 25,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 72,
    "name": "ml.c5.9xlarge",
    "vcpuNum": 36
   },
   {
    "_defaultOrder": 26,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 96,
    "name": "ml.c5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 27,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 144,
    "name": "ml.c5.18xlarge",
    "vcpuNum": 72
   },
   {
    "_defaultOrder": 28,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.c5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 29,
    "_isFastLaunch": true,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g4dn.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 30,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g4dn.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 31,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g4dn.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 32,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g4dn.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 33,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g4dn.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 34,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g4dn.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 35,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 61,
    "name": "ml.p3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 36,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 244,
    "name": "ml.p3.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 37,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 488,
    "name": "ml.p3.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 38,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.p3dn.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 39,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.r5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 40,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.r5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 41,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.r5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 42,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.r5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 43,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.r5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 44,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.r5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 45,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.r5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 46,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.r5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 47,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 48,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 49,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 50,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 51,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 52,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 53,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.g5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 54,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.g5.48xlarge",
    "vcpuNum": 192
   },
   {
    "_defaultOrder": 55,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 56,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4de.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 57,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.trn1.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 58,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.trn1.32xlarge",
    "vcpuNum": 128
   },
   {
    "_defaultOrder": 59,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.trn1n.32xlarge",
    "vcpuNum": 128
   }
  ],
  "instance_type": "ml.t3.medium",
  "kernelspec": {
   "display_name": "Python 3 (Data Science 3.0)",
   "language": "python",
   "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-west-2:236514542706:image/sagemaker-data-science-310-v1"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}