calculate_builtin_embeddings#

calculate_builtin_embeddings(text: array, model: str = 'miniLM', file_path: Optional[str] = 'embeddings.npy', device: Optional[str] = None, long_sample_behaviour: str = 'average+warn', open_ai_batch_size: int = 500) array[source]#

Get the built-in embeddings for the dataset.

Parameters
textnp.array

The text to get embeddings for.

modelstr, default ‘miniLM’

The type of embeddings to return. Can be either ‘miniLM’ or ‘open_ai’. For ‘open_ai’ option, the model used is ‘text-embedding-ada-002’ and requires to first set an open ai api key by using the command openai.api_key = YOUR_API_KEY

file_pathOptional[str], default ‘embeddings.csv’

If given, the embeddings will be saved to the given file path.

devicestr, default None

The device to use for the embeddings. If None, the default device will be used.

long_sample_behaviourstr, default ‘average+warn’

How to handle long samples. Averaging is done as described in https://github.com/openai/openai-cookbook/blob/main/examples/Embedding_long_inputs.ipynb Currently, applies only to the ‘open_ai’ model, as the ‘miniLM’ model can handle long samples.

Options are:
  • ‘average+warn’ (default): average the embeddings of the chunks and warn if the sample is too long.

  • ‘average’: average the embeddings of the chunks.

  • ‘truncate’: truncate the sample to the maximum length.

  • ‘raise’: raise an error if the sample is too long.

  • ‘nan’: return an embedding vector of nans for each sample that is too long.

open_ai_batch_sizeint, default 500

The amount of samples to send to open ai in each batch. Reduce if getting errors from open ai.

Returns
np.array

The embeddings for the dataset.