Research-toolset - AI research tool

What problem were you trying to solve when you created this tool?

I wanted to get as much determinism as possible into AI-based research, instead of relying on a single answer from a language model. That was the main motivation of all the projects I will publish now under open-source. That is why also this tool consists of many agents, several layers of checks and confidence assessments.

Here, It started with a narrow task: checking large numbers of citations, with a clear statement for each entry of what is correct and what is not. We refined that over many rounds with real reference lists.

Then further questions came up. How do you research publications on a topic and filter them, for example by the number of citations? How do you connect web indexes, so that the tool can use what an institution already knows?

At the same time, agentic approaches developed quickly. Since 2023 and 2024 we went from simple exchanges between a few specialised agents to more complex setups, and the more complex ones were clearly more capable. In the end the work became a directed graph of tasks, a DAG. It works like a project schedule: each step starts only when the results it depends on are available, and independent steps run in parallel. Planning agents build the graph, other agents work through it, and quality checks sit between and after the steps. That was a large step forward in quality and traceability.

After that came connectors, to bring in more data. And techniques from retrieval-augmented generation: embedders and rerankers, which really do improve precision when sources are matched and ranked. So the tool combines language models with deterministic steps, and it uses different models for different tasks. That assumes the models run reliably in the background, and that you can afford the token volume of repeated runs.

How could this be useful to an archive—for example, when researching collections, checking references or investigating an institution or person?

I see three main uses.

First, researching collections: where related holdings are kept, what has been published about the provenance of a collection, including sources in other languages.

Second, checking references in publications, exhibition catalogues or submitted manuscripts. The tool checks every entry against the large scholarly databases, CrossRef, OpenAlex, Semantic Scholar, arXiv and DBLP. It corrects details, adds missing DOIs and flags entries that cannot be found anywhere. That has become more important since AI-written texts started to contain invented citations.

Third, researching an institution or a person. For historical persons and institutions that is web research. For one's own organisation there is an institution mode that focuses on the organisation's website, its search index and its staff directory.

In addition there are analysis modes: an in-depth explanation of a topic, a decision analysis, a research design, a literature review, or a first draft of a grant proposal, for example for a digitisation project.

What are its limitations? What should an archivist always check rather than simply accepting the report?

The tool only sees what its data sources give it. Without further connectors that is the open web and the scholarly databases. Most archival holdings are not online. The same applies to the reference check. Archival references and grey literature are not in those databases.

A link shows where a statement comes from. It does not prove that the source was read correctly. Dates, attributions, shelfmarks and quotations should always be checked in the source itself.

What the tool can see depends on the installation, and that is where it can grow. The tool is built around connectors. A connector that reads an archive's own portal more precisely is easy to add and works immediately for every page of that portal. A completely new search source takes some development work in the pipeline. Both are documented.

So I would treat a report as a well-documented starting point.

If an archive wanted to try the tool, how difficult would it be to get started, and where can people find it?

The tool is open source under the MIT licence. It is on GitHub at github.com/maltedreyer-5/research-toolset .

On a laptop with Python 3.12, for example in PyCharm, the installation takes a few minutes: install the requirements, copy the example configuration, enter the address of the language model, and start.

For a server there is a Dockerfile and a Compose example. You configure the language model and, if available, an embedder and a reranker, and then start the container. All of this is described in the documentation. The main decision is the language model. A local model keeps everything in-house. A hosted model is quicker to start with. Everything else is configuration.