Last week I was working on a proof of concept solution which includes a service responsible to provide a simple front-facing search component for a hardware tools website. During the research, I stumbled upon various options and I wanted to try deploying Meilisearch on Azure Container Apps as it meets most of the requirements for the search service within the solution.
Overview of Meilisearch
Meilisearch is a RESTful search API that offers a fast and instant search experience (search as you type), it is designed for a vast majority of needs of small-to-medium businesses with little configuration needed during installation yet with high customization.
Meilisearch is an open-source project built using Rust with more than 29K stars on GitHub and support for various SDKs including dotnet.
Meilisearch on Azure Container Apps
Meilisearch can be deployed on Azure on different options, Meilisearch container image can be deployed on Azure App Services as per the documentation, on this post I will go over the steps needed to prepare the Bicep template needed to deploy a Meilisearch container image into Azure Container Apps and use storage mounts in Azure Container Apps to permanently host Meilisearch database into Azure Files.
The source code used for this post exists on GitHub.
Deploying Meilisearch on Azure Container Apps
In this post, I will go over the Bicep template needed to deploy Meilisearch into Azure Container Apps and any important notes needed for deployment.
You can click on the button below to deploy a Meilisearch instance into Azure Container Apps. This is the final result of the Bicep templates we are going to build:
If you are new to Bicep, you can check my previous post on how to deploy Container Apps using Bicep.
Step 1: Define Azure Storage Resource
Azure storage is needed to create a file share service under it, the file share service will allow us to mount a file share from Azure Files as a volume inside the Meilisearch container. This means that Meilisearch DB and its configuration files are written into the container volume location are persisted in the file share, this is durable storage so if the container is restarted, crashed or a new revision is deployed; the files on the share will not be impacted and the new provisioned container will find the files on the configured volume.
To create a file share service under Azure storage, create a directory named “deploy/modules”, add a new file named “storage.bicep” and use the code below:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 |
@description('The name of your application') param applicationName string @description('The Azure region where all resources in this example should be created') param location string = resourceGroup().location @description('A list of tags to apply to the resources') param resourceTags object @description('The name of the container to create. Defaults to applicationName value.') param containerName string = applicationName @description('The name of the Azure file share.') param shareName string @description('The name of storage account') param storageAccountName string resource storageAccount 'Microsoft.Storage/storageAccounts@2021-09-01' = { name: storageAccountName location: location tags: resourceTags sku: { name: 'Standard_LRS' } kind: 'StorageV2' } resource blobServices 'Microsoft.Storage/storageAccounts/blobServices@2021-09-01' = { name: 'default' parent: storageAccount } resource storageContainer 'Microsoft.Storage/storageAccounts/blobServices/containers@2021-09-01' = { name: containerName parent: blobServices } resource fileServices 'Microsoft.Storage/storageAccounts/fileServices@2021-09-01' = { name: 'default' parent: storageAccount } resource permanentFileShare 'Microsoft.Storage/storageAccounts/fileServices/shares@2022-05-01' = { name: shareName parent: fileServices properties: { accessTier: 'TransactionOptimized' enabledProtocols: 'SMB' shareQuota: 1024 } } var storageKeyValue = storageAccount.listKeys().keys[0].value output storageAccountName string = storageAccount.name output id string = storageAccount.id output apiVersion string = storageAccount.apiVersion output storageKey string = storageKeyValue |
Looking at the code above, notice that we’ve created a file share with the access tier “TransactionOptimized”, you can use “Premium” as it is backed by SSD drives and provides low latency. The size of the file share is set to 1024 gigabytes (1TB).
Step 2: Define Azure Log Analytics Workspace Resource
Add a new file named “logAnalyticsWorkspace.bicep” under the folder “modules”, and use the code below, the log analytics workspace is needed by the Container Apps Environment,
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 |
@description('The name of your Log Analytics Workspace') param logAnalyticsWorkspaceName string @description('The Azure region where all resources in this example should be created') param location string = resourceGroup().location @description('A list of tags to apply to the resources') param resourceTags object resource logAnalyticsWorkspace'Microsoft.OperationalInsights/workspaces@2021-12-01-preview' = { name: logAnalyticsWorkspaceName tags: resourceTags location: location properties: any({ retentionInDays: 30 features: { searchVersion: 1 } sku: { name: 'PerGB2018' } }) } var sharedKey = listKeys(logAnalyticsWorkspace.id, logAnalyticsWorkspace.apiVersion).primarySharedKey output workspaceResourceId string = logAnalyticsWorkspace.id output logAnalyticsWorkspaceCustomerId string = logAnalyticsWorkspace.properties.customerId output logAnalyticsWorkspacePrimarySharedKey string = sharedKey |
Step 3: Define an Azure Container Apps Environment Resource
Add a new file named “acaEnvironment.bicep” under the folder “modules” and use the code below:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 |
@description('The name of Azure Container Apps Environment') param acaEnvironmentName string @description('The Azure region where all resources in this example should be created') param location string = resourceGroup().location @description('A list of tags to apply to the resources') param resourceTags object param logAnalyticsWorkspaceCustomerId string @secure() param logAnalyticsWorkspacePrimarySharedKey string resource environment 'Microsoft.App/managedEnvironments@2022-03-01' = { name: acaEnvironmentName location: location tags: resourceTags properties: { appLogsConfiguration: { destination: 'log-analytics' logAnalyticsConfiguration: { customerId: logAnalyticsWorkspaceCustomerId sharedKey: logAnalyticsWorkspacePrimarySharedKey } } } } output acaEnvironmentId string = environment.id |
Step 4: Define an Azure Container Apps Environment Storages Resource
Add a new file named “acaEnvironmentStorages.bicep” under the folder “modules” and use the content below:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 |
@description('The name of Azure Container Apps Environment') param acaEnvironmentName string @description('The name of your storage account') param storageAccountResName string @description('The storage account key') @secure() param storageAccountResourceKey string @description('The ACA env storage name mount') param storageNameMount string @description('The name of the Azure file share. Defaults to applicationName value.') param shareName string resource environment 'Microsoft.App/managedEnvironments@2022-03-01' existing = { name: acaEnvironmentName } //Environment Storages resource permanentStorageMount 'Microsoft.App/managedEnvironments/storages@2022-03-01' = { name: storageNameMount parent: environment properties: { azureFile: { accountName: storageAccountResName accountKey: storageAccountResourceKey shareName: shareName accessMode: 'ReadWrite' } } } |
This is a key step to configure a storage definition of type AzureFile in the Container Apps Environment, within this file we have enabled the environment to use the Azure File share service for any Container App under this environment, we are setting the access mode to “ReadWrite” as we need to write and read files from the file share.
Step 5: Define an Azure Container Apps Resource
Add a new file named “containerApp.bicep” under the folder “modules” and use the content below:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 |
param containerAppName string param location string param environmentId string param containerImage string param targetPort int param containerRegistry string param containerRegistryUsername string param isPrivateRegistry bool param registryPassName string param minReplicas int = 0 param maxReplicas int = 1 @secure() param secListObj object param envList array = [] param revisionMode string = 'Single' param storageNameMount string param volumeName string param mountPath string param resourceTags object param resourceAllocationCPU string param resourceAllocationMemory string resource containerApp 'Microsoft.App/containerApps@2022-06-01-preview' = { name: containerAppName location: location tags: resourceTags properties: { managedEnvironmentId: environmentId configuration: { activeRevisionsMode: revisionMode secrets: secListObj.secArray registries: isPrivateRegistry ? [ { server: containerRegistry username: containerRegistryUsername passwordSecretRef: registryPassName } ] : null ingress: { external: true targetPort: targetPort transport: 'auto' traffic: [ { latestRevision: true weight: 100 } ] } dapr: null } template: { containers: [ { image: containerImage name: containerAppName env: envList volumeMounts: [ { mountPath:mountPath volumeName:volumeName } ] resources:{ cpu: json(resourceAllocationCPU) memory: resourceAllocationMemory } } ] volumes: [ { name: volumeName storageName: storageNameMount storageType: 'AzureFile' } ] scale: { minReplicas: minReplicas maxReplicas: maxReplicas } } } } output fqdn string = containerApp.properties.configuration.ingress.fqdn |
This module is responsible to deploy the actual Meilisearch container image to Container App, what we have done here is the following:
- Configuring the ingress of the container app to be external ingress (accepts HTTP requests from the public internet).
- Parameterizing the target port of the ingress controller, we will set the value in the next steps.
- Parameterizing the Meilisearch container image which will be deployed on this container app, the parameter will hold the Meilisearch image from docker hub.
- Parameterizing the compute resources CPU and memory of the container app.
- Parameterizing the secrets and environment variables arrays.
- Define one single storage volume of type AzureFile for the container app and parameterize the volume name and storage name mount.
- Define one single volume mount in the container app and parameterize the mount path and volume name.
- Lastly, outputting the FQDN of the provisioned container app as it will be the URL to access Meilisearch API.
Step 6: Define the Main module for the final deployment
Lastly, we need to define the Main Bicep module which will link the modules together, this will be the file that is referenced from AZ CLI command when creating the entire resources, to do so under the folder “deploy”, add a new file named “main.bicep” and use the code below:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 |
targetScope = 'subscription' //Azure Regions which Azure Container Apps available at can be found on this link: //https://azure.microsoft.com/en-us/explore/global-infrastructure/products-by-region/?products=container-apps®ions=all @description('The Azure region code for deployment resource group and resources such as westus, eastus, northeurope, etc...') param location string = 'westus' @description('The name of your search service. This value should be unique') param applicationName string = 'meilisearch' @description('The Container App CPU cores and Memory') @allowed([ { cpu: '0.25' memory: '0.5Gi' } { cpu: '0.5' memory: '1.0Gi' } { cpu: '0.75' memory: '1.5Gi' } { cpu: '1.0' memory: '2.0Gi' } { cpu: '1.25' memory: '2.50Gi' } { cpu: '1.5' memory: '3.0Gi' } { cpu: '1.75' memory: '3.5Gi' } { cpu: '2.0' memory: '4.0Gi' } ]) param containerResources object = { cpu: '1.0' memory: '2.0Gi' } @maxLength(4) @description('The environment of deployment such as dev, test, stg, prod, etc...') param deploymentEnvironment string = 'dev' @secure() @description('The Master API Key used to connect to Meilisearch instance') @minLength(32) param meilisearchMasterKey string = newGuid() var resourceGroupName = '${applicationName}-${deploymentEnvironment}-rg' var logAnalyticsWorkspaceResName = '${applicationName}-${deploymentEnvironment}-logs' var environmentName = '${applicationName}-${deploymentEnvironment}-env' var storageAccountName = '${take(applicationName,14)}${deploymentEnvironment}strg' var shareName = 'meilisearch-fileshare' var storageNameMount = 'permanent-storage-mount' var meilisearchImageName = 'getmeili/meilisearch:v0.29' var meilisearchAppPort = 7700 var dbMountPath = '/data/meili' var volumeName = 'azure-file-volume' var defaultTags = { environment: deploymentEnvironment application: applicationName } resource rg 'Microsoft.Resources/resourceGroups@2021-04-01' = { name: resourceGroupName location: location tags: defaultTags } module storageModule 'modules/storage.bicep' = { scope: resourceGroup(rg.name) name: '${deployment().name}--storage' params: { storageAccountName: storageAccountName location: rg.location applicationName: applicationName containerName: applicationName shareName: shareName resourceTags: defaultTags } } module logAnalyticsWorkspace 'modules/logAnalyticsWorkspace.bicep' = { scope: resourceGroup(rg.name) name: '${deployment().name}--logAnalyticsWorkspace' params: { logAnalyticsWorkspaceName: logAnalyticsWorkspaceResName location: rg.location resourceTags: defaultTags } } module environment 'modules/acaEnvironment.bicep' = { scope: resourceGroup(rg.name) name: '${deployment().name}--acaenvironment' params: { acaEnvironmentName: environmentName location: rg.location logAnalyticsWorkspaceCustomerId: logAnalyticsWorkspace.outputs.logAnalyticsWorkspaceCustomerId logAnalyticsWorkspacePrimarySharedKey: logAnalyticsWorkspace.outputs.logAnalyticsWorkspacePrimarySharedKey resourceTags: defaultTags } } module environmentStorages 'modules/acaEnvironmentStorages.bicep' = { scope: resourceGroup(rg.name) name: '${deployment().name}--acaenvironmentstorages' dependsOn:[ environment ] params: { acaEnvironmentName: environmentName storageAccountResName: storageModule.outputs.storageAccountName storageAccountResourceKey: storageModule.outputs.storageKey storageNameMount: storageNameMount shareName: shareName } } module containerApp 'modules/containerApp.bicep' = { scope: resourceGroup(rg.name) name: '${deployment().name}--${applicationName}' dependsOn: [ environment ] params: { containerAppName: applicationName location: rg.location environmentId: environment.outputs.acaEnvironmentId containerImage: meilisearchImageName targetPort: meilisearchAppPort minReplicas: 1 maxReplicas: 1 revisionMode: 'Single' storageNameMount: storageNameMount mountPath: dbMountPath volumeName: volumeName resourceTags: defaultTags resourceAllocationCPU: containerResources.cpu resourceAllocationMemory: containerResources.memory secListObj: { secArray: [ { name: 'meili-master-key-value' value: meilisearchMasterKey } ] } envList: [ { name: 'MEILI_MASTER_KEY' secretRef: 'meili-master-key-value' } { name: 'MEILI_DB_PATH' value: dbMountPath } ] } } output containerAppUrl string = containerApp.outputs.fqdn |
What we have done is the following:
- Defined a set of parameters so the end user can control the deployment of Meilisearch instance, parameters defined as the following:
- Location: The Azure region code (“westus”, “northeurope”, “australiacentral”, etc…). This should be a region where Azure container Apps and Azure Storage is available, you can check where Azure Container Apps are available on this link.
- Application Name: the name of the Meilisearch search service, this name will be part of the FQDN and will be used toĀ set resource group name, storage, container app environment, and log analytics workspace.
- Container Resources: Container App CPU and Memory. ReadĀ here to understand more about those CPU/Memory combinations. The limits are soft limits and you can request to increase the quota by submitting a support request.
- Deployment Environment: Used to identify deployment resources (“dev”, “stg”, “prod”, etc…) and tag them with the selected environment, this has nothing to do with the capacity or performance of the resources provisioned. This will be useful if you are deploying multipleĀ Meilisearch instances under the same subscription for dev/test scenarios.
- Meilisearch Master Key: This is the Master API Key used with the Meilisearch instance, minimum length is 32 characters. The recommendation is to generate a strong key, if not provided deployment template will generate a guide as the Master API key.
- Defined a set of variables to be passed to the child modules to provision needed resources, variables defined are:
- Share Name: the name of the file share which will be created under Azure storage files share, this name will be passed to the Container Apps environment too to configure the storage definition of type AzureFile in the Container Apps environment.
- Storage Name Mount: The name of the storage mount associated with the Meilisearch container.
- Meilisearch Image Name: The Meilisearch docker image name “getmeili/meilisearch:v0.29” hosted in docker hub using tag ‘v0.29’ this is the latest tag at the time of writing this post.
- Meilisearch App Port: This is the port that Container App listens to for incoming requests, Meilisearch uses port 7700, when ingress is enabled on Container App; the ingress endpoint is exposed on port 443.
- DB Mount Path: The path “/data/meili” is used, this path represents the volume inside the Container App mounted to Azure File Share. You can change the path if needed but the path needs to be the same in an environment variable named “MEILI_DB_PATH“
- Volume Name: Logical name used to define the volume used in the Container App.
- Notice how we are storing the “Meilisearch Master Key” securely in the Container Apps secrets, and using a “secretRef” in the environment variables to reference the secret. The name of the environment variable which stores the Master API Key must be “MEILI_MASTER_KEY“
- Lastly, we are returning the provisioned Container App FQDN as a deployment output.
Step 7: Deploy Meilisearch Resources using Azure CLI
Now we are ready to deploy the resources using Azure CLI, to do so, open a PowerShell console and use the script below, don’t forget to set actual values for the placeholders.
1 2 3 4 |
az deployment sub create ` --template-file ./main.bicep ` --location WestUS ` --parameters '{ \"meilisearchMasterKey\": {\"value\":\"YOUR_MASTER_KEY\"}, \"applicationName\": {\"value\":\"YOUR_APP_NAME\"}, \"deploymentEnvironment\": {\"value\":\"dev\"}, \"location\": {\"value\":\"westus\"} }' |
Note: You can use the “Deploy to Azure Button” highlighted above to deploy the resources.
If all is completed successfully you should see the resource group and the below 4 resources created under the subscription selected as the image below. To get the Container App FQDN, you can navigate to the Container App or you can get it from the deployment output tab.
Step 8: Test Deployed Meilisearch Instance
To test the deployed Meilisearch Instance, I’ve created a console application that uses the dotnet Meilisearch SDK and it creates an index named “movies” and indexes 40K documents using a JSON file (file is included in source code) or you can download it from this link.
The console application contains the below code:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 |
using System.Text.Json; namespace Meilisearch.Console { public class Movie { public int Id { get; set; } public string Title { get; set; } public string Poster { get; set; } public string Overview { get; set; } public IEnumerable<string> Genres { get; set; } } internal class Program { static async Task Main(string[] args) { MeilisearchClient client = new MeilisearchClient("https://<fqdn>.<location>.azurecontainerapps.io", "<MASTER API KEY>"); var options = new JsonSerializerOptions { PropertyNameCaseInsensitive = true }; string jsonString = await File.ReadAllTextAsync(@"movies.json"); var movies = JsonSerializer.Deserialize<IEnumerable<Movie>>(jsonString, options); var index = client.Index("movies"); var newSettings = new Settings { FilterableAttributes = new string[] { "genres" }, SortableAttributes = new string[] { "title" }, }; await index.UpdateSettingsAsync(newSettings); await index.AddDocumentsAsync<Movie>(movies,"id"); } } } |
What the console application does is the following:
- Instantiating MeilisearchClient by passing the URL of the Container App and the Master API Key.
- Reading the JSON content from the file.
- Updating the settings and attributes of an index named “movies”.
- Lastly, add the documents (movies) into the index named “movies”.
Meilisearch provides a PostMan collection that contains all the endpoints to configure your deployed Meilisearch instance and perform a search as well, Meilisearch PostMan collection can be downloaded from this link.
To verify that documents added successfully to the “movies” index, you can issue the below HTTP Get request and you see a list of movies returned.
1 2 3 |
GET /indexes/movies/documents Host: <FQDN>.<Location>.azurecontainerapps.io Authorization: Bearer <MASTER API KEY> |
Leave a Reply