MINDCRAWL: AI-Powered Crawling Backed by CDTI and FEDER

The MINDCRAWL project consisted of developing a new “CRAWLING” solution using artificial intelligence models, capable of intelligently and autonomously crawling information from countless websites; studying, analyzing, and assessing all their content to offer organized data and added value to the solution's users in a reliable and simple manner.

The project has been co-financed by CDTI and the European Regional Development Fund (ERDF 2021-2027).

• Budget: €1,242,062 

• EU Aid: €299,441 

The proposed project has had the following specific technical objectives:

• Analysis and definition of an innovative event bus with pub/sub technology, to detect problems and generate alerts, generate AI models, dump information into specialized databases, and obtain the information to self-repair the system scripts.
• Research, design, and development of new AI models to solve information categorization problems.
• Research, design, and development of columnar databases specialized in analytical queries using databases with more complex separate infrastructure.
• Integration of the solution with external user authentication systems through Single Sign On technologies and an automatic configuration workflow for new users.
• Research, design, and development of a new price change suggestion simulation system.

The results obtained after the project was developed are:

• Development of a comprehensive intelligent crawling platform with self-repair modules and automatic classification using AI.
• Development of real-time analytical query functionalities and interactive dashboards.
• Substantial improvement in performance and reliability, reducing the average time for script self-repair by 40%, increasing automatic coverage of analyzed sites by over 50%, and decreasing the index of websites with crawling errors from 8.90% to 4.75%.

Project execution location: Murcia, Spain.

Project execution period: 01/2024-12/2025.