Challenge

As part of the workshop, we propose a task-based LLM evaluation challenge. It involves information extraction in French in a few-shot context, where only the annotation guide annotation guide describing the entity classes and their annotation conventions and a few annotated annotated documents. Participants are invited to evaluate LLM implementations, but participation with other with other approaches is welcome and will help to put the results of LLM-based systems into perspective. LLM-based systems.

Data from this challenge, developed within DGA, will be provided to participants under a signature of a data use agreement. These are newsletters and blogs in French, annotated annotated with entities of interest to the intelligence field (name, location, organization, but also function, equipment...) that can be nested and trigger events. events. Expected output formats and evaluation metrics will also be communicated. communicated. At the end of the training phase, test data will be made available to participants for participants for 3 days (to be chosen within a one-week interval) and the results will be communicated to them. Participants will be invited to present their work in a short paper and on the on the day of the workshop.