TripleGrader is presented, an open-source web platform for grading subject-predicate-object triples on a 0 to 10 ordinal scale, and each triple is scored independently, and peer scores stay hidden until the grader has rated it.
Abstract
Knowledge graphs need validated relational triples, and gold benchmarks of such triples are often built by several annotators. Most text annotation platforms focus on span tagging and compute agreement on categorical labels. They are not designed for ordinal quality grading of triples that have already been extracted. We present TripleGrader, an open-source web platform for grading subject-predicate-object triples on a 0 to 10 ordinal scale. In TripleGrader, each triple is scored independently, and peer scores stay hidden until the grader has rated it. Graders then discuss entries in threaded comments and propose revisions, which the session owner approves and publishes as new versions. For agreement, the platform computes quadratic-weighted Cohen's kappa for two raters and a quadratic-weighted Fleiss-type kappa for three or more. Graded sessions and the final benchmark are exported in common formats. We evaluated the platform in three ways. We ran 72 automated tests, including 18 workflow scenarios through the REST API. We also injected 10 error types to check whether prescribed corrections could be applied while preserving unaffected triples. On two small corpora with existing gold benchmarks, we then used the platform to curate the triples extracted by four large language models into each benchmark. Only 10% to 32% of the candidate triples could be kept unchanged, and 53% to 71% needed corrections, partly to match the benchmarks'naming conventions. Per model, 11 to 29 missing triples had to be added. On these corpora, the extracted triples therefore needed substantial human review. The source code is available at https://github.com/razz0n/triple-grader-tool.
It is found that Llama 3 delivers comparable quality to leading language models such as GPT-4 on a plethora of tasks, and performs competitively with the state-of-the-art on image, video, and speech recognition tasks.
This paper investigates developing real-world perception for active agents, proposes Gibson Environment for this purpose, and showcases a set of perceptual tasks learned therein.
F. Xia, Amir Zamir, Zhi-Yang He et al.· 2018 IEEE/CVF Conference on...· 1,069 citations· ⚡112
GAOKAO-Bench is introduced, an intuitive benchmark that employs questions from the Chinese GAOKAO examination as test samples, including both subjective and objective questions that contribute a robust evaluation benchmark for future large language models and offers valuable insights into the advantages and limitations...
Xiaotian Zhang, Chun-yan Li, Yi Zong et al.· arXiv.org· 216 citations· ⚡17
An algorithm is proposed which leverages disentangled semantic factors to generate adversarial perturbation by altering controlled semantic attributes to fool the learner towards various "adversarial" targets.
Haonan Qiu, Chaowei Xiao, Lei Yang et al.· arXiv.org· 208 citations· ⚡17
This work proposes a HALO that directly maximizes the utility of generations instead of maximizing the log-likelihood of preferences, and suggests that the best loss depends on the inductive biases most appropriate for a given setting, an oft-overlooked consideration.
This work investigates the possibilities of using LLMs in a resume screening setting via a document retrieval framework that simulates job candidate selection and finds that the MTEs are biased, significantly favoring White-associated names in 85% of cases and female-associated names in only 11.1% of cases.