With the move towards agentic AI we need to think in terms of Workflows. What chain of events has to happen to get from point A to point B or what path takes us from input to the desired output. In today’s post we’re going to build a workflow that takes an Image file provided by the user, processes it to extract numeric data, then prints the results to the screen. In today’s post we’ll just focus on building the core functionality. In the next post in this series we’ll expand this into an agentic solution which runs automatically and stores its results in a spreadsheet.
To keep this example manageable I’ve created several images that contain some text and a number. As you can see below I’ve used different colors and fonts and even wording so that we can’t easily write code to process these images. In other words a non-AI approach to this problem would require us encoding rules for every possible case. That also means once we get an image that we’ve not encountered before, we’d have to create a new rule to handle this. This is what separates traditional robotics process automation (RPA) from Agentic AI. RPA relies on rules and consistency in the data input whereas the LLM may be able to apply what it knows to a new situation and gives us the desired result.




Above: The original image set
In this workflow we’ll use traditional (or non-AI) tools to extract all the text data from the images. It will then send that data along with a prompt to my local LLM to request the number be pulled out of the data.
Why not just put it all in AI?
Could we do this entire project in AI? Well possibly yes, however, I suspect a pure AI solution would lack the auditability of the workflow I’m proposing. Maybe in this simple project AI would get it right 99.9% of the time, however, as you’ll see we could scale this workflow to analyze something more than just pictures of a number. Perhaps we send in invoices, or tax forms. Using separate tools allows us to test each step in the chain and ensure that right output, at that step, is feeding into the next tool. To reduce the risk of AI hallucination we only introduce the LLM at the point where its unique capabilities are needed.
We’re going to use a few different Python modules and programs, all of which are freely available and are various degrees of open source. I did coauthor this program this with AI (vibecoded), however, my usual disclaimer applies – do not run any AI generated code unless you understand exactly what it is doing. Always review the output and ensure it is not harmful to you, your computer or your organization.
For this project our goal is to read an image that contains a label and a number. We will process that image to extract the label and the number from them and convert these to machine readable text. Our output is going to be a JSON file which contains the data.
We’re going to use the following modules and programs:
- Tesseract – Tesseract is a free and open source optical character recognition engine. Its job here is to extract the text (and numeric) data from the images.
- Pillow – Python Image Library (aka PIL) is a free and open source library for manipulating images
- Ollama – is a free and open source python library for interacting with an Ollama server. Ollama is the name of AI software running on my AI Server (link to article)
- Pydantic – a free and open-source data validation library. Here, we use it to define a strict schema that forces the AI to return predictable JavaScript Object Notation (JSON) data. JSON files are text files that are easy for both humans to read and machines to parse which makes them a great interchange format for communication between systems and programs
The block diagram below will give you a sense of the work flow we’re building today.

Code
Here’s the code for this project, we’ll walkthrough it below:
import os
import pytesseract
from PIL import Image
from ollama import chat, Client
from pydantic import BaseModel, Field
# 1. Target Schema, This is our desired JSON output
class NumberExtraction(BaseModel):
value: float = Field(description="The primary numeric value extracted from the text")
label: str = Field(description="Context or label associated with the number")
def extract_from_text_llm(image_path: str):
# Step A: Local OCR using Tesseract
raw_text = pytesseract.image_to_string(Image.open(image_path))
print(f"Raw OCR Output:\n{raw_text.strip()}\n")
# Step B: Standard Text Model (e.g., mistral, llama3, qwen2.5)
client = Client(host='http://YOUR_OLLAMA_HOST:11434')
response = client.chat(
model='qwen2.5-coder:14b',
messages=[
{
'role': 'user',
'content': f"Extract the key numeric metric and its label from this raw OCR text: {raw_text}"
}
],
format=NumberExtraction.model_json_schema(),
options={'temperature': 0}
)
# Step C: Parse Structured JSON
result = NumberExtraction.model_validate_json(response.message.content)
print(f"Parsed Data: {result.value} | Label: {result.label}")
return result
if __name__ == "__main__": extract_from_text_llm(r"path/to/your/image.png")
Note: This code will not run unless you have an Ollama server running locally or on your network and the required modules installed in your python instance.
Code Walkthrough
In the first group we’ll import the modules we’re going to use. These are our tools that’ll be calling throughout the program.
The next section labeled #1 Target Schema, this is our desired output or what we want our JSON file to look like. It will have two fields, or categories of data:
- Value, which holds the number
- Label, which holds any text label in the image
Next, we create a function called extract_from_text_llm which accepts the file path of an image as its parameter. This is where tesseract and Pillow come into play. We define a variable called raw_text which is going to hold all of the extracted text from the image. Tesseract has a built in function called image_to_string that will do the heavy lifting for us. Recall string is another word for text in computer programming. To perform this function it needs an image so we use the Image.open function of Pillow to essentially open the image for Tesseract.
For a debugging or audit check we use a Print statement to show us the text that was extracted.
Now it’s time to call the LLM, in step B. Ollama requires a few pieces to function so we define a client variable which is given the location of my local LLM server on my network in its host parameter. This will be used to interact with Ollama. Now, we do the heavy lifting. We define a response variable which calls Ollama and provides it the prompt and the model to use. For this run we’re using the Qwen 2.5 14 billion parameter coder model. We’re passing a data structure into Ollama, the first piece is the model to use. The messages bit is our prompt, role tells Ollama who is providing this prompt, in this case the user of application, however, this could be Assistant, or System depending on our use case. Content is our prompt and you can see that we feed in the raw_text variable from Step A at the end of the prompt. Lastly we tell Ollama that we want a particular JSON output using the format parameter to the NumberExtraction schema we defined earlier in the program. We provide one more parameter temperature:0. This tells Ollama to use “greedy decoding” which is basically to give us the next word without thinking too much. It promotes consistent output but sacrifices creativity. In our use we don’t want too much creativity and we will use this setting. Once this command is sent the LLM goes to work and fills our response variable with its answer!
We now reach Step C which is our user output – this defines a variable called result which is first validated in our JSON schema in other words it’s put into a format we can work with in the following steps. Keep in mind the variable response.message.content is the response from the LLM. We then print out the data from the string using the print command and close out our function with the return command sending the result variable, our JSON data, back to the main program where we could do more processing if we wanted.
The final lines of code, the __name__ == “__main__” bit, this is the main entry point for the program. In other words you can see the command is to call the extract_from_text_llm function we’ve discussed providing it a file path of an image to work with.
For my run I’d modified my entry point to run all 4 test files and you can see from my initial run I determined that this indeed is working pretty well but there is one oddity.
Debugging Sample 2

Sample #2 the label comes up blank , you can see there’s nothing after Label: in the bottom of the image. It should be “This Is”.
I was curious as to why this might be happening, my first thought is that it has a two word label and I created another sample with a two word label Max Load and sample 5 works as expected.
I asked Gemini what might be happening and it suggested that the issue was the words chosen. The LLM understands that words before the colon are the label however the words “This Is” aren’t normally a category so it could be excluding them on purpose. It suggested I strengthen the prompt by adding “Capture all text preceding the number as the label, excluding the colon, even if it is nonsensical filler text.” After I added this it worked as intended.

Tool Chain Observability
This issue was easier to spot because of the tool chain approach. I could see that Tesseract was indeed extracting the “This is” label and we knew that what was flowing into the LLM was the data we wanted. Trouble shooting then was much easier than if we had just tried to do this with a single tool. The tool chain approach is key to building consistent and reliable workflows.
Next steps
I hope you enjoyed this first step in AI workflows. In the next article we’ll turn this into an agentic AI solution and build an automated workflow around this structure.
Do you want to learn how to automate your financial workpapers? Do you want to build MicroApps like this one to bolster your own automation toolkit? Do you want to get started with VBA?
My book, Beginning Microsoft Excel VBA Programming for Accountants has many examples like this to teach you to use Excel to maximize your productivity! It’s available on Amazon,Apple iBooks and other eBook retailers!



















