multimodal_fin.processing.metadata package

Submodules

multimodal_fin.processing.metadata.coherence_analyzer module

class multimodal_fin.processing.metadata.coherence_analyzer.CoherenceAnalysis(**data)[source]

Bases: BaseModel

Represents the logical coherence analysis between a monologue and a response.

consistent: bool
contradictions: List[ContradictionDetail]
model_config: ClassVar[ConfigDict] = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

summary: str
topic_covered: bool
class multimodal_fin.processing.metadata.coherence_analyzer.CoherenceAnalyzer(model_name='llama3')[source]

Bases: object

Analyzes the logical coherence between a monologue and a response using an LLM model.

analyze_coherence(monologue, response)[source]

Evaluates whether a given response is topically and logically coherent with a preceding monologue.

The analysis determines:

  • If the response addresses a topic mentioned in the monologue.

  • If the response is logically consistent with the content of the monologue.

  • If contradictions exist, it identifies and explains them.

Parameters:

  • monologue (str) – The original monologue text (e.g., CEO/CFO statements).

  • response (str) – The follow-up response text (e.g., Q&A answers).

Returns:

A dictionary following the CoherenceAnalysis schema.

Return type:

dict

model_name: str = 'llama3'

Name of the model used for inference (default is “llama3”).

class multimodal_fin.processing.metadata.coherence_analyzer.ContradictionDetail(**data)[source]

Bases: BaseModel

Represents a contradiction found between a monologue and a response.

explanation: str
model_config: ClassVar[ConfigDict] = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

monologue_excerpt: str
response_excerpt: str

multimodal_fin.processing.metadata.metadata_enricher module

class multimodal_fin.processing.metadata.metadata_enricher.MetadataEnricher(sec10k_model_names, qa_analyzer_models, num_evaluations=5, device='cpu', verbose=1)[source]

Bases: object

Enriches a DataFrame with metadata about topic, QA, and coherence.

device: str = 'cpu'

Device for model inference (e.g., ‘cpu’ or ‘cuda’).

enrich(df)[source]

Orchestrates enrichment of the dataframe into structured JSON.

Parameters:

df (DataFrame) – DataFrame containing sentences and their embeddings.

Returns:

Structured metadata dictionary with monologues and Q&A pairs.

Return type:

Dict[str, Any]

num_evaluations: int = 5

Number of repetitions per model to compute uncertainty.

qa_analyzer_models: List[str]

List of models used for QA answerability analysis.

sec10k_model_names: List[str]

List of model names used for topic classification.

verbose: int = 1

silent, 1: info).

Type:

Verbosity level (0

multimodal_fin.processing.metadata.prompt_builder module

class multimodal_fin.processing.metadata.prompt_builder.PromptBuilder[source]

Bases: object

Static class to build prompts for various LLM-based analysis tasks.

static analize_qa(intervention, response)[source]

Builds a prompt to analyze whether a multi-question intervention was answered fully, partially, or not at all.

Parameters:

  • intervention (str) – The original question or set of questions.

  • response (str) – The corresponding response from the speaker.

Returns:

A list of messages formatted for the LLM to assess QA coverage.

Return type:

List[dict[str, str]]

static check_coherence(monologue, response)[source]

Builds a prompt to evaluate the coherence between a monologue and a response.

Parameters:

  • monologue (str) – The earlier speech or monologue from the executive.

  • response (str) – The follow-up response to be checked for logical consistency.

Returns:

A list of messages formatted for the LLM to assess coherence.

Return type:

List[dict[str, str]]

static explain_why_other(text)[source]

Builds a prompt asking the LLM to explain why a given text was classified as ‘Other’.

Parameters:

text (str) – The text originally classified as Other.

Returns:

A list of formatted chat messages for the LLM.

Return type:

List[dict[str, str]]

static prompt_10k(text)[source]

Builds a prompt to classify a text into 10-K sections: Business, Risk Factors, MD&A, or Other.

Parameters:

text (str) – The financial or business-related intervention.

Returns:

A list of formatted chat messages for the LLM.

Return type:

List[dict[str, str]]

static prompt_monologue(text)[source]

Builds a prompt to classify an intervention as a Monologue or Procedure.

Parameters:

text (str) – The intervention text.

Returns:

A list of formatted chat messages.

Return type:

List[dict[str, str]]

static prompt_qa(text)[source]

Builds a prompt to classify a meeting intervention as Question, Answer, or Procedure.

Parameters:

text (str) – The intervention text to classify.

Returns:

A list of messages formatted for an LLM chat input.

Return type:

List[dict[str, str]]

multimodal_fin.processing.metadata.qa_analyzer module

class multimodal_fin.processing.metadata.qa_analyzer.EvaluatedQA(**data)[source]

Bases: BaseModel

Represents a single question from an intervention and its evaluation.

answer_quote: str | None
answer_summary: str | None
answered: Literal['yes', 'partially', 'no']
model_config: ClassVar[ConfigDict] = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

question: str
class multimodal_fin.processing.metadata.qa_analyzer.InterventionAnalysis(**data)[source]

Bases: BaseModel

Represents the full QA analysis for an intervention.

evaluations: List[EvaluatedQA]
intervention: str
model_config: ClassVar[ConfigDict] = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

response: str
class multimodal_fin.processing.metadata.qa_analyzer.QAAnalyzer(model_name='llama3', NUM_EVALUATIONS=5)[source]

Bases: UncertaintyMixin

Analyzes Q&A interactions by evaluating whether questions were answered in responses.

NUM_EVALUATIONS: int = 5

Number of LLM passes to estimate uncertainty.

analize_qa(intervention, response)[source]

Runs the LLM to analyze whether each question in the intervention is answered in the response.

Parameters:

  • intervention (str) – The text containing one or more questions.

  • response (str) – The response from the speaker.

Returns:

The parsed LLM response following InterventionAnalysis schema.

Return type:

dict

evaluate_qa_model(data)[source]

Evaluates a QA model on a dataset of interventions with ground truth.

Parameters:

data (list) – A list of dicts, each containing a ‘response’ and list of ‘label’ dicts with true Q&A labels.

Returns:

DataFrame with predicted and true labels per question.

Return type:

DataFrame

get_pred(question, response)[source]

Performs multiple evaluations to determine the QA label and uncertainty.

Parameters:

  • question (str) – A single question to evaluate.

  • response (str) – The response to evaluate against.

Returns:

(predicted_label, confidence, extra_info_dict)

Return type:

tuple

get_pred_question(question, response)[source]

Returns the answer status for a given question-response pair using a single pass.

Parameters:

  • question (str) – The question to check.

  • response (str) – The response to evaluate.

Returns:

One of [‘yes’, ‘partially’, ‘no’] or None if failed.

Return type:

Optional[str]

model_name: str = 'llama3'

Name of the LLM to use.

multimodal_fin.processing.metadata.sec10k_analyzer module

class multimodal_fin.processing.metadata.sec10k_analyzer.Category10K(**data)[source]

Bases: BaseModel

Pydantic schema used for validating LLM output when classifying SEC 10-K topics.

category: Literal['Business', 'Risk Factors', 'MD&A', 'Other']
model_config: ClassVar[ConfigDict] = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class multimodal_fin.processing.metadata.sec10k_analyzer.SEC10KAnalyzer(model='llama3', NUM_EVALUATIONS=10)[source]

Bases: UncertaintyMixin

Class responsible for classifying intervention text into 10-K categories using LLMs.

NUM_EVALUATIONS: int = 10

Number of times the classification is repeated to estimate uncertainty.

classify_dataframe(df)[source]

Classifies an entire DataFrame of interventions by applying get_pred on each row.

Parameters:

df (DataFrame) – DataFrame with a ‘text’ column.

Returns:

The original DataFrame with an added ‘classification’ column.

Return type:

DataFrame

classify_text(text)[source]

Classifies a given text into one of the 10-K categories.

Parameters:

text (str) – Text to classify.

Returns:

One of [‘Business’, ‘Risk Factors’, ‘MD&A’, ‘Other’].

Return type:

str

explain_other_category(text)[source]

Provides a natural language explanation for why a text was classified as ‘Other’.

Parameters:

text (str) – The text classified as ‘Other’.

Returns:

Explanation generated by the LLM.

Return type:

str

get_pred(text)[source]

Predicts the category for a text using repeated sampling for uncertainty estimation.

Parameters:

text (str) – Text to classify.

Returns:

Most likely category, confidence score, and list of predictions.

Return type:

Tuple[str, float, List[str]]

model: str = 'llama3'

The name of the LLM model to use.

Module contents