How I built an ingestion pipeline for an AI property assistant

The client was missing out on leads at night and on weekends, when the estate agencies were closed. Visitors could browse properties, but booking a viewing required someone to take the next step.

We built an AI sales assistant for UK real estate agencies. Visitors could search for properties and start booking a viewing through chat, even outside office hours. The product went on to serve more than 30 agencies.

I participated from the beginning and built the ingestion pipeline that supplied the assistant with property data. The main problem was: the agencies could not give us API access, and every website had a different structure.

I proposed using each agency's website as the data source. That gave us one onboarding flow: provide the website URL, process its listings in the background, and make those properties available to the assistant.

This article follows that flow through the implementation I reviewed. I worked on this with a team; below, I distinguish my original proposal from the behavior of the inspected code. The diagrams describe that checkout, and the agency count describes the product's adoption.

Start with a property catalog

The assistant needed more than page text. It needed properties with identities, prices, locations, availability, and enough descriptive information to answer a visitor's request.

A search for a two-bedroom home under a particular budget contains both exact constraints and a question about relevance. Those requirements shaped the ingestion output: a structured property catalog, followed by a semantic search index.

Each step has a different responsibility. Firecrawl fetches website content. Model calls help classify pages and turn their contents into records. Application code checks those outputs and controls how they affect stored data.

That separation let us reuse the flow across agencies while keeping the assistant's input consistent.

Choosing a service I could validate quickly

I knew Firecrawl from using its MCP server in my development workflow. For this project, I compared providers and read their API documentation before deciding how to integrate one.

The prices of the options I considered were close enough that integration effort mattered. Some alternatives required more local setup before I could test them. Firecrawl gave me a hosted API, documentation I could follow, and credits to try it against real sites.

I remember completing the first tests and extractions in roughly a week. That was the initial evaluation, not the time it took to build the complete pipeline.

I was looking for a practical answer: how quickly could I find out whether the service produced useful inputs for our application? The effort between choosing and validating a tool became part of the architecture decision.

Discovering the right pages

An agency website contains listings, search results, staff pages, location guides, and navigation. Sending every discovered page through property extraction would spend work on content the catalog did not need.

The current implementation starts a Firecrawl crawl and combines its discovered URLs with URLs collected from the site's sitemap. It also loads known property URLs from the catalog so later runs can revisit existing listings.

Discovery has a fallback path. If the result looks thin relative to known stock, the pipeline can inspect listing indexes. If the filtered set is still thin, it tries Firecrawl's map endpoint and another harvesting pass. The final set is deduplicated, merged with known URLs, prioritized, and capped before extraction tasks are scheduled.

This is more involved than the shorthand “map the site and scrape its properties.” The repository contains heuristics for particular URL and listing-page patterns. The reusable onboarding flow needs those discovery rules to handle variation between websites.

Recognizable listing URL patterns are retained by deterministic rules. A model helps select likely property URLs from the remaining candidates. The application then checks the returned URLs against the supplied set. A URL invented by the model is discarded instead of becoming a new scraping target.

This boundary is useful: the model can classify candidates, while ordinary code controls which candidates exist.

Prompt engineering with runtime checks

Each selected URL becomes an extraction task. Firecrawl collects the page content, and a model turns it into a record that follows the application's property schema.

The record includes fields such as the property page URL, title, address, property type, bedrooms, bathrooms, price, and description. The response must include the schema's required keys. Fields such as floor area, images, and EPC URL can be null.

The prompt asks for the expected structure. A runtime decoder checks the returned JSON against that structure. TypeScript types describe the contract inside the application, but an external model response still needs validation when it arrives.

The checks go beyond valid JSON. A structurally valid response can still lack usable property data. In the inspected implementation, the address and price must both be non-empty before the record proceeds.

There are also separate outcomes for a page with no usable listing, an inconclusive extraction, and a failure. That distinction becomes important when processing a property the system has seen before.

A schema cannot prove that the model read the correct price. It can reject the wrong shape and help enforce application rules. Extraction quality still depends on the source content, the model's interpretation, and checks appropriate to the field.

This is where prompt engineering and conventional code fit together. The model handles variable page content, and the application decides whether the result is acceptable.

Background work and observable progress

The agency should be able to continue signing up while its listings are being processed. Adding its website emits a domain event, and a reaction schedules ingestion work.

The reaction checks whether ingestion has already been scheduled for that agency. If another crawl is active, the request goes into a queue for the dispatcher. Those checks help avoid duplicate scheduling and overlapping work; they are not a claim of exactly-once execution across the system.

Repeated work also needs stable identities. A property is identified from its agency and source URL, while a crawl result is identified from the property and crawl session. A retried event-emission step can therefore check whether that result has already been recorded. The model's generated ID does not decide the property's identity.

Within a crawl, extraction tasks have staggered start times. This spreads out requests instead of starting every page extraction together. Staggering does not impose a strict ceiling on concurrency: a slow task can still overlap with later tasks.

The application projects ingestion events into a status view with scheduled, running, completed, and failed states, along with extraction counts and errors. That gives the product a way to show progress and gives the team a place to investigate failures.

One subtlety is that finishing extraction and finishing search indexing are separate milestones.

The diagram shows that separation, rather than a fixed order for every event. The crawl workflow concludes from its extraction results. It does not explicitly wait for every Qdrant write.

The property projection writes the catalog inside a MongoDB transaction and updates Qdrant before that transaction commits. The two stores do not share a transaction. Projection failures can require replay, and a completed Qdrant write cannot be rolled back with MongoDB. Stable point IDs make repeated indexing update the same search point.

For the interface, a completed crawl should not automatically be treated as proof that every property is already searchable. If I were adding a precise “ready to search” indicator, I would give it its own indexing acknowledgment.

Embedding property records

The semantic index is built from the structured catalog. The implementation assembles text from the property's title, address, expanded location, type, room counts, price information, and description.

An embedding model converts that text into a numeric vector. Search can compare the vector for a visitor's request with the vectors for properties to rank semantic similarity.

For example, a visitor might ask for a home with space to work remotely, while a listing describes a study. This is an illustrative reason to use semantic search, not a claim that every such query will return the right listing.

The inspected code uses a local feature-extraction pipeline through Transformers.js. Its default model is Xenova/bge-base-en-v1.5, with mean pooling and normalization. The model loads once per process and is reused; the deployment expects its files in the image.

Qdrant stores the vectors with cosine distance. Each agency has its own collection, and a stable point ID derived from the property identity allows updates to upsert the same property.

This is a property-level representation. The pipeline does not index arbitrary raw HTML fragments as independently searchable chunks. Normalizing the listing first gives the index a consistent input and a link back to a catalog record.

Separate collections provide a clear partition for each agency's search data. Correct authorization and agency selection still need to happen in the application.

Combining exact constraints with semantic relevance

Embeddings are useful for relevance, but a vector score is not how I would enforce a budget or a bedroom count. Those values already exist as structured fields.

The search path combines catalog filtering with vector search. It obtains candidates from stored property fields, embeds the query, and searches within the agency's vector collection. A later model step filters the returned candidates for the request.

That final step selects among existing candidates. If its response cannot be used, or it selects none, the implementation keeps the original candidates. It is a best-effort relevance filter.

This is the division of responsibilities in the search path, not a guarantee of search accuracy. Candidate filtering, vector matching, and the final model response each need verification. A useful test set would include strict budgets, unavailable listings, empty results, and agencies with similar properties.

Ingestion prepares data for both kinds of work. The catalog keeps fields the application can inspect directly. The semantic index lets it compare descriptions with the visitor's language.

Keeping the catalog current

Listings change after onboarding. A property can become unavailable, its price can change, or someone can correct its description in the application.

The implementation distinguishes evidence that a listing is gone from a temporary inability to read it. A confirmed missing page can contribute to an unavailable result. A timeout or incomplete extraction should not automatically remove a previously valid listing.

The code also contains availability checks that combine page signals with model-assisted interpretation. Those are decisions based on evidence and confidence, rather than proof that the source website always exposes its status correctly.

When a property becomes unavailable, its catalog status is updated and its vector is removed. For manually edited properties, the refresh path preserves selected human-edited fields. Certain changing fields, such as price and images, can still update.

These rules make repeated ingestion different from a one-time import. The system needs a policy for what to update, what to preserve, and what an unsuccessful refresh means.

What made the flow reusable

Firecrawl and AI made this approach practical for us. They helped us work with websites whose structure we did not control. My work was connecting extraction, structured data, background processing, and search into a flow that the product could reuse.

The implementation has limits: discovery rules, caps on work, provider failures, uncertain extraction results, and asynchronous indexing. I would measure coverage, extraction quality, indexing lag, and cost per usable property before making throughput claims.

For the agency, the flow stayed understandable: provide the website, connect the calendar, and install the assistant's script. Behind that interface, ingestion and indexing prepared the listings for chat.

I enjoyed turning that repeated onboarding work into something the product could handle.