{"id":6803,"date":"2025-12-16T18:09:33","date_gmt":"2025-12-16T16:09:33","guid":{"rendered":"https:\/\/eurocc-bulgaria.bg\/?p=6803"},"modified":"2025-12-16T18:27:10","modified_gmt":"2025-12-16T16:27:10","slug":"system-for-merging-and-iterative-quality-assurance-of-the-results-of-high-performance-generative-ai-systems-collaboration-with-industrial-end-users","status":"publish","type":"post","link":"https:\/\/eurocc-bulgaria.bg\/?p=6803","title":{"rendered":"System for merging and iterative quality assurance of the results of high-performance Generative AI systems (Collaboration with industrial (end) users)"},"content":{"rendered":"\t\t<div data-elementor-type=\"wp-post\" data-elementor-id=\"6803\" class=\"elementor elementor-6803\" data-elementor-settings=\"[]\">\n\t\t\t\t\t\t<div class=\"elementor-inner\">\n\t\t\t\t\t\t\t<div class=\"elementor-section-wrap\">\n\t\t\t\t\t\t\t<section class=\"has_eae_slider elementor-section elementor-top-section elementor-element elementor-element-42416b6 elementor-section-boxed elementor-section-height-default elementor-section-height-default\" data-id=\"42416b6\" data-element_type=\"section\">\n\t\t\t\t\t\t<div class=\"elementor-container elementor-column-gap-default\">\n\t\t\t\t\t\t\t<div class=\"elementor-row\">\n\t\t\t\t\t<div class=\"has_eae_slider elementor-column elementor-col-50 elementor-top-column elementor-element elementor-element-2db1ab9\" data-id=\"2db1ab9\" data-element_type=\"column\">\n\t\t\t<div class=\"elementor-column-wrap elementor-element-populated\">\n\t\t\t\t\t\t\t<div class=\"elementor-widget-wrap\">\n\t\t\t\t\t\t<div class=\"elementor-element elementor-element-5dcd686 elementor-widget elementor-widget-image\" data-id=\"5dcd686\" data-element_type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"elementor-image\">\n\t\t\t\t\t\t\t\t\t\t<img loading=\"lazy\" decoding=\"async\" width=\"150\" height=\"150\" src=\"https:\/\/eurocc-bulgaria.bg\/wp-content\/uploads\/2022\/11\/EuroCC-Logo_BulgariaV1.jpg\" class=\"attachment-thumbnail size-thumbnail\" alt=\"\" \/>\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t<div class=\"has_eae_slider elementor-column elementor-col-50 elementor-top-column elementor-element elementor-element-395c949\" data-id=\"395c949\" data-element_type=\"column\">\n\t\t\t<div class=\"elementor-column-wrap elementor-element-populated\">\n\t\t\t\t\t\t\t<div class=\"elementor-widget-wrap\">\n\t\t\t\t\t\t<div class=\"elementor-element elementor-element-723a402 elementor-widget elementor-widget-text-editor\" data-id=\"723a402\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"elementor-text-editor elementor-clearfix\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-32945cf elementor-widget elementor-widget-image\" data-id=\"32945cf\" data-element_type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"elementor-image\">\n\t\t\t\t\t\t\t\t\t\t<img loading=\"lazy\" decoding=\"async\" width=\"150\" height=\"146\" src=\"https:\/\/eurocc-bulgaria.bg\/wp-content\/uploads\/2021\/03\/law2019-06-117044295.jpg\" class=\"attachment-thumbnail size-thumbnail\" alt=\"\" \/>\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<section class=\"has_eae_slider elementor-section elementor-top-section elementor-element elementor-element-f288d6c elementor-section-boxed elementor-section-height-default elementor-section-height-default\" data-id=\"f288d6c\" data-element_type=\"section\">\n\t\t\t\t\t\t<div class=\"elementor-container elementor-column-gap-default\">\n\t\t\t\t\t\t\t<div class=\"elementor-row\">\n\t\t\t\t\t<div class=\"has_eae_slider elementor-column elementor-col-50 elementor-top-column elementor-element elementor-element-dba6949\" data-id=\"dba6949\" data-element_type=\"column\">\n\t\t\t<div class=\"elementor-column-wrap elementor-element-populated\">\n\t\t\t\t\t\t\t<div class=\"elementor-widget-wrap\">\n\t\t\t\t\t\t<div class=\"elementor-element elementor-element-f483c1b elementor-widget elementor-widget-text-editor\" data-id=\"f483c1b\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"elementor-text-editor elementor-clearfix\"><p><strong>NCC presenting the success story<\/strong><\/p><p>NCC-Bulgaria has been founded by the Institute of Information and Communication Technologies at<br \/>the Bulgarian Academy of Sciences, Sofia University \u201cSt. Kliment Ohridski\u201d and the University of<br \/>National and World Economy.NCC-Bulgaria is focused on:<\/p><ul><li>Creating a roadmap for successful work in the field of HPC, big data analysis and AI,<\/li><li>Analyzing the existing competencies and facilitating the use of HPC\/HPDA\/AI in Bulgaria<\/li><li>Raising awareness and promoting HPC\/HPDA\/AI use in companies and the public sector.<\/li><\/ul><div><p style=\"mso-add-space: auto; text-align: justify; text-indent: -14.2pt; line-height: 107%; mso-list: l0 level1 lfo1; margin: 0cm 0cm 6.0pt 14.2pt;\"><b><span style=\"font-size: 12.0pt; line-height: 107%; mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin; mso-ansi-language: EN-GB;\">Scientific partners involved<\/span><\/b><b><span style=\"font-size: 12.0pt; line-height: 107%; mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin; mso-ansi-language: EN-GB;\">:<\/span><\/b><b><\/b><\/p><p>TBI Bank<\/p><\/div><div><span lang=\"EN\" style=\"font-size: 11.0pt; line-height: 107%; font-family: 'Calibri',sans-serif; mso-ascii-theme-font: minor-latin; mso-fareast-font-family: Calibri; mso-fareast-theme-font: minor-latin; mso-hansi-theme-font: minor-latin; mso-bidi-font-family: 'Times New Roman'; mso-bidi-theme-font: minor-bidi; mso-ansi-language: EN; mso-fareast-language: EN-US; mso-bidi-language: AR-SA;\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-6643 aligncenter\" src=\"https:\/\/eurocc-bulgaria.bg\/wp-content\/uploads\/2025\/12\/euro2Picture2.png\" alt=\"\" width=\"150\" height=\"150\" \/><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-934 aligncenter\" src=\"https:\/\/eurocc-bulgaria.bg\/wp-content\/uploads\/2021\/03\/law2019-06-117044295.jpg\" alt=\"\" width=\"150\" height=\"150\" \/><\/span><\/div><div><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-6644 aligncenter\" src=\"https:\/\/eurocc-bulgaria.bg\/wp-content\/uploads\/2025\/12\/TBIbank.jpg alt=\" width=\"117\" height=\"117\" \/><\/div><div><span lang=\"EN\" style=\"font-size: 11.0pt; line-height: 107%; font-family: 'Calibri',sans-serif; mso-ascii-theme-font: minor-latin; mso-fareast-font-family: Calibri; mso-fareast-theme-font: minor-latin; mso-hansi-theme-font: minor-latin; mso-bidi-font-family: 'Times New Roman'; mso-bidi-theme-font: minor-bidi; mso-ansi-language: EN; mso-fareast-language: EN-US; mso-bidi-language: AR-SA;\">\u00a0<\/span><\/div><p><strong>Technical\/scientific Challenge:<\/strong><\/p><p>Effectively integrating multiple high-performance Generative AI systems (such as Large Language Models) into business workflows presents multiple key technical and scientific challenges. Most state-of-the-art LLMs are built on autoregressive frameworks, meaning they generate text sequentially, predicting each token based solely on preceding tokens. Models have a finite context window, which restricts the amount of background information and detail that can be incorporated into prompts and responses. Generating lengthy or highly detailed responses often leads to issues with maintaining consistent logic, style, and accuracy across the entire output.<\/p><p>Currently, there is no standardized, streamlined workflow to manage the lifecycle of generative AI interactions effectively. Manually consolidating outputs from multiple LLMs requires human judgment or additional software tools, adding complexity and inefficiency to workflows. Ensuring the final output meets high standards for quality, accuracy, readability, and originality poses additional scientific and technical problems. Combining multiple outputs intelligently requires advanced natural language processing techniques and specialized tools.<\/p><p>Adjusting the final merged text for consistent tone, style, and grammatical correctness is critical for professionalism but introduces additional layers of complexity. Reliably detecting and mitigating potential plagiarism or originality issues is essential yet challenging, particularly when balancing automated processes and manual oversight.<\/p><\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t<div class=\"has_eae_slider elementor-column elementor-col-50 elementor-top-column elementor-element elementor-element-e26fb19\" data-id=\"e26fb19\" data-element_type=\"column\">\n\t\t\t<div class=\"elementor-column-wrap elementor-element-populated\">\n\t\t\t\t\t\t\t<div class=\"elementor-widget-wrap\">\n\t\t\t\t\t\t<div class=\"elementor-element elementor-element-7d76f2d elementor-widget elementor-widget-text-editor\" data-id=\"7d76f2d\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"elementor-text-editor elementor-clearfix\"><p><strong>Solutions: <\/strong><\/p><p>To ensure the highest quality and consistency in the final results provided to the end user, the outputs from the selected Large Language Models (OpenAI\u2019s GPT-4o, Anthropic\u2019s Claude 3.5 Sonnet, and Google\u2019s Gemini 1.5) undergo a structured merging and refinement process. The merging procedure involves pairwise combinations of responses leveraging external tools such as EasyPeasy.ai and the OpenAI o1 API <a href=\"https:\/\/eurocc.unwe.bg\/system-for-merging-and-iterative-quality-assurance-of-the-results-of-high-performance-generative-ai-systems\/#_ftn1\">[1]<\/a> (the current state-of-the-art reasoning model, see <strong>Figure 1<\/strong>).<\/p><ul><li><strong><u>Pairwise Merging:<\/u><\/strong> Outputs from two models are merged using EasyPeasy.ai\u2019s text-merging capability [2]. This tool intelligently consolidates distinct strengths, clarity, and coherence into one unified text, ensuring comprehensive coverage of key points and reducing redundancy. The merging is performed in pairs of two \u2013 the first two outputs \u2013 <em>Text_Output_1 and Text_Output_2<\/em> are merged into <em>Result_1,<\/em> and then <em>Result_1<\/em> is merged with <em>Text_Output_3<\/em> to get to the final merged result. An alternative approach is using a reasoning model (like OpenAI\u2019s o1 or DeepSeek R1) to perform the merging, since these models can use their \u201cthinking\u201d to reason about the similarities and differences among the three different outputs.<\/li><li><strong><u>Quality Assurance of the Merged Output: <\/u><\/strong>Once the merged output is finalized into a single coherent piece, the text undergoes additional rigorous quality assurance steps.<\/li><li><strong><u>Spelling, Grammar, Style, and Tone Check (Trinka Grammar Checker API [3]):<\/u><\/strong> The integrated text is validated for language accuracy, grammatical correctness, consistency in style, and appropriateness of tone using the robust capabilities of the Trinka Grammar Checker API. Any identified linguistic issues &#8211; such as typographical errors, stylistic inconsistencies, awkward phrases, or inappropriate tone &#8211; are automatically corrected, ensuring professional-quality language suitable for the specified target audience.<\/li><li><strong><u>Plagiarism Check (Copyleaks API [4]):<\/u><\/strong> After ensuring linguistic quality, the output undergoes an originality assessment via Copyleaks API to verify that the merged text maintains high originality standards. Copyleaks returns a plagiarism confidence score. A threshold of at least 80% originality is required to pass. Outputs below this threshold are flagged and either reprocessed or reviewed manually, maintaining content authenticity and integrity.<\/li><li><strong><u>Final Adjustments and Enhancements:<\/u><\/strong> Once the text passes both linguistic quality and originality checks, it undergoes one final round of enhancements using the OpenAI o1 API. The objective of this stage is to incorporate subtle refinements based on the outcomes of previous validation steps:<\/li><li>Style refinement: Ensure a smooth narrative flow;<\/li><li>Clarification improvements: Eliminate ambiguous or unclear wording;<\/li><li>Precision enhancements: Reinforce exactness and factual reliability;<\/li><li>Tone adjustments: Ensure alignment with the specified nuance and intent.<\/li><\/ul><p>Upon completion of the comprehensive quality assurance and refinement phases, the system presents the final output through an intuitive and user-friendly web interface. Users are provided with options to either approve the final result or submit feedback for additional fine-tuning and reprocessing.<\/p><p><strong>Scientific impact: <\/strong><\/p><ul><li>Proposes a new method (as a part of the whole system) for systematically merging diverse LLM outputs to leverage complementary model strengths.<\/li><li>Establishes robust post-processing validation protocols in order to enhance output reliability and user trust.<\/li><li>Provides empirical data on the efficacy of AI-driven text merging and enhancement techniques.<\/li><\/ul><p><strong>Benefits:<\/strong><\/p><ul><li>Provides users with singular, authoritative outputs that represent the collective intelligence of multiple generative AI models.<\/li><li>Removes manual overhead for users, delivering a polished, publication-quality result ready for immediate use.<\/li><\/ul><\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<section class=\"has_eae_slider elementor-section elementor-top-section elementor-element elementor-element-7bf9fe8 elementor-section-boxed elementor-section-height-default elementor-section-height-default\" data-id=\"7bf9fe8\" data-element_type=\"section\">\n\t\t\t\t\t\t<div class=\"elementor-container elementor-column-gap-default\">\n\t\t\t\t\t\t\t<div class=\"elementor-row\">\n\t\t\t\t\t<div class=\"has_eae_slider elementor-column elementor-col-50 elementor-top-column elementor-element elementor-element-9c858c6\" data-id=\"9c858c6\" data-element_type=\"column\">\n\t\t\t<div class=\"elementor-column-wrap elementor-element-populated\">\n\t\t\t\t\t\t\t<div class=\"elementor-widget-wrap\">\n\t\t\t\t\t\t<div class=\"elementor-element elementor-element-c31a884 elementor-widget elementor-widget-text-editor\" data-id=\"c31a884\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"elementor-text-editor elementor-clearfix\"><p><strong>Success story # Highlights:<\/strong><\/p><ul><li>An iterative workflow for merging and quality assurance of large language models&#8217; outputs is presented and implemented.<\/li><li>User-friendly web interface for reviewing and managing the process step-by-step<strong>.<\/strong><\/li><\/ul><\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-8fdd3da elementor-widget elementor-widget-text-editor\" data-id=\"8fdd3da\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"elementor-text-editor elementor-clearfix\"><p><strong>Figure 1: Architecture of the second stage (after the prompt has been distributed among different models and we\u2019ve received their outputs)<\/strong><\/p><\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-ee1e414 elementor-widget elementor-widget-image\" data-id=\"ee1e414\" data-element_type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"elementor-image\">\n\t\t\t\t\t\t\t\t\t\t<img loading=\"lazy\" decoding=\"async\" width=\"888\" height=\"384\" src=\"https:\/\/eurocc-bulgaria.bg\/wp-content\/uploads\/2025\/12\/ArchSecondStage.png\" class=\"attachment-large size-large\" alt=\"\" \/>\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t<div class=\"has_eae_slider elementor-column elementor-col-50 elementor-top-column elementor-element elementor-element-0664904\" data-id=\"0664904\" data-element_type=\"column\">\n\t\t\t<div class=\"elementor-column-wrap elementor-element-populated\">\n\t\t\t\t\t\t\t<div class=\"elementor-widget-wrap\">\n\t\t\t\t\t\t<div class=\"elementor-element elementor-element-b515d6a elementor-widget elementor-widget-text-editor\" data-id=\"b515d6a\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"elementor-text-editor elementor-clearfix\"><p><strong>Reference:<\/strong><\/p><p>[1] Introducing OpenAI o1, <a href=\"https:\/\/openai.com\/o1\/\">https:\/\/openai.com\/o1\/<\/a><\/p><p>[2] Easy-peasy, <a href=\"https:\/\/easy-peasy.ai\">https:\/\/easy-peasy.ai<\/a><\/p><p>[3] Trinka, <a href=\"https:\/\/trinka.ai\">https:\/\/trinka.ai<\/a><\/p><p>[4] Copyleaks, <a href=\"https:\/\/copyleaks.ai\">https:\/\/copyleaks.ai<\/a><\/p><p><strong>\u00a0<\/strong><\/p><p><strong>Contact:<\/strong><\/p><ul><li><strong>Venko Andonov, <\/strong><em>University of National and World Economy, Sofia, vandonov[at]unwe.bg<\/em><\/li><li><strong> Valentin Kisimov<\/strong><strong>, <\/strong><em>University of National and World Economy, Sofia, <\/em><em>vkisimov[at]unwe.bg<\/em><\/li><\/ul><p><strong>\u00a0<\/strong><\/p><\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t","protected":false},"excerpt":{"rendered":"<p>NCC presenting the success story NCC-Bulgaria has been founded by the Institute of Information and Communication Technologies atthe Bulgarian Academy of Sciences, Sofia University \u201cSt. Kliment Ohridski\u201d and the University [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":6804,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[129],"tags":[],"_links":{"self":[{"href":"https:\/\/eurocc-bulgaria.bg\/index.php?rest_route=\/wp\/v2\/posts\/6803"}],"collection":[{"href":"https:\/\/eurocc-bulgaria.bg\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/eurocc-bulgaria.bg\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/eurocc-bulgaria.bg\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/eurocc-bulgaria.bg\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=6803"}],"version-history":[{"count":4,"href":"https:\/\/eurocc-bulgaria.bg\/index.php?rest_route=\/wp\/v2\/posts\/6803\/revisions"}],"predecessor-version":[{"id":6810,"href":"https:\/\/eurocc-bulgaria.bg\/index.php?rest_route=\/wp\/v2\/posts\/6803\/revisions\/6810"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/eurocc-bulgaria.bg\/index.php?rest_route=\/wp\/v2\/media\/6804"}],"wp:attachment":[{"href":"https:\/\/eurocc-bulgaria.bg\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=6803"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/eurocc-bulgaria.bg\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=6803"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/eurocc-bulgaria.bg\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=6803"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}