# CSV Bulk Loader CLI for LanceDB ## Background A data team needs a small, reusable command-line tool that ingests CSV files into a [LanceDB](https://docs.lancedb.com/) table, embedding a designated text column via the OpenAI `text-embedding-3-small` endpoint, and then exposes a semantic search subcommand for retrieving top-k matches. The environment already contains a deterministic, pre-generated dataset at `/app/data/articles.csv` with around 5000 rows and the columns `id,title,body,category,published`. Your job is to build the CLI that loads that data into LanceDB and searches it. ## Requirements - Implement a single Python script at `/home/user/loader_project/loader.py` exposing two subcommands: - `ingest`: reads a CSV file, splits it into batches of size N, embeds the chosen text column using the OpenAI `text-embedding-3-small` model, and writes all rows (with their embeddings) to a LanceDB table. - `search`: embeds a query string with the same OpenAI model, runs a vector search against an existing LanceDB table, and prints the top-k results as a single JSON object to standard output. - All embeddings MUST be produced by real OpenAI API calls; no local models, no mock vectors. - Run-id isolation: ingest/search MUST work against whatever table name the verifier passes via `--table`. The verifier will pass a name suffixed with the current `ZEALT_RUN_ID` so that concurrent test runs do not collide. - The LanceDB database must live at `/home/user/loader_project/lance_db`. All tables are created inside that directory. - Ingestion MUST preserve every CSV row exactly once (no de-duplication, no row drops). Row count in the LanceDB table after ingest MUST equal the row count of the CSV (excluding the header). ## Implementation Hints - Read the OpenAI key from the `OPENAI_API_KEY` environment variable. - Use `lancedb.connect(...)` for the database connection and `db.create_table(...)` (with overwrite-on-rerun semantics) to create the embedded table. - Batch your OpenAI embedding calls; sending one row per HTTP call against 5000 rows will be both slow and rate-limited. - Decide how you want to store the CSV columns alongside the embedding vector (Arrow / Pydantic / list-of-dicts are all fine). The verifier only requires that the schema preserves `id`, `title`, `body`, `category`, `published`, and that the embedding column can be searched with `table.search(query_vector)`. - For `search`, run a vector search via `table.search(query_vector).limit(k)` and serialize the resulting rows (without the raw embedding vector) into JSON. ## Acceptance Criteria - Project path: /home/user/loader_project - Command (ingest): `python3 loader.py ingest --csv --table --text-col --batch-size ` - Command (search): `python3 loader.py search --table --query --k ` - Both commands MUST exit with status 0 on success. - LanceDB database directory: `/home/user/loader_project/lance_db` - After `ingest`, the named LanceDB table MUST contain exactly the same number of rows as the source CSV (excluding the header). - After `ingest`, the table MUST be openable via `lancedb.connect('/home/user/loader_project/lance_db').open_table()` and contain at minimum the columns `id`, `title`, `body`, `category`, `published`, plus one vector column suitable for `text-embedding-3-small` (1536 dimensions). - `search` MUST print a single JSON object to stdout with the following shape and NOTHING else (logs may go to stderr): ```json { "query": "", "k": , "results": [ { "id": , "title": "", "category": "", "published": "", "score": } ] } ``` The `results` array MUST have length `min(k, table_row_count)` and be ordered from best to worst match. The `score` field may be either a distance or a similarity (any monotonic ordering) but must be a finite number. - The verifier will pass the table name `articles_${ZEALT_RUN_ID}` to both subcommands. Read `ZEALT_RUN_ID` is NOT required from your code (the verifier provides the suffix as part of `--table`).

claude-4-6-sonnet	gemini-3-flash	gemini-3.1-pro	gemini-3.5-flash	glm-5.1
135.5s	71.3s	95.3s	160.2s	328.4s
117.5s	116.5s	169.4s	268.6s	252.0s
169.4s	59.3s	98.3s	142.2s	204.7s
142.6s	59.2s	93.8s	128.7s	600.2s
119.9s	122.0s	129.6s	284.8s	195.6s
157.7s	103.0s	133.0s	244.8s	212.4s
104.3s	133.6s	64.3s	189.1s	187.2s
138.6s	88.4s	197.6s	399.5s	225.3s
95.0s	62.0s	78.0s	147.8s	130.7s
60.8s	49.2s	38.8s	72.3s	111.5s

Task

Task