========================== QUESTION DIALOGS DATASET ========================== For more details see the paper: "Data Collection for Interactive Learning through the Dialog", 2016, Vodolán Miroslav, Filip Jurčíček, http://arxiv.org/abs/1603.09631 The dataset consists of standard data split into training, development and test files: 1) question_dialogs-train.json 2) question_dialogs-dev.json 3) question_dialogs-test.json Dataset files contain one dialog per line. The dialogs are stored in json format. Three python scripts are released with the dataset: a) interactive_learning_evaluator.py - Evaluates given model in interactive manner on dialogs simmulated from conversations in dataset. b) interactive_model_base.py - Base class which simplifies developement of interactive models by providing standard routines for communication with interactive_learning_evaluator.py simulator. c) simple_interactive_model.py - Simple implementation of interactive model that can be used for testing of interactive_learning_evaluator.py environment. ==============SCRIPTS USAGE============== The interactive_learning_evaluator.py script can be run with simple_interactive_model.py in this way (for more info about its parameters use --help): > python interactive_learning_evaluator.py "python simple_interactive_model.py" --train_dialogs dataset/question_dialogs-train.json which results in output similar to: Training dialogs from files ['question_dialogs-train.json'] dialog_count: 950 # Count of dialogs in whole dataset turn_count: 4237 # Count of turns in all dialogs simulation_wall_time: 1s # How long the simulation took avg_dialog_wall_time: 1ms # How long the simulation took per dialog in average answer_correct_count: 3 # How many of answers provided by the model were correct answer_incorrect_count: 2 # How many of answers provided by the model were incorrect requested_explanations: 947 # How many explanations were requested during simulation requested_answers: 945 # How many answer hints were requested during simulation requested_answers_with_correct_answer: 175 # How many answer hints with correct answer were requested during simulation extracted_correct_answers: 1 # How many of the answers extracted from answer hints were correct extracted_incorrect_answers: 1 # How many of the answers extracted from answer hints were incorrect extracted_correct_no-answers: 10 # How many answer hints without correct answer were recognized extracted_incorrect_no-answers: 1 # How many answer hints without correct answer were not recognized efficiency_score: -1.2014737 # Efficiency score presented in the paper "Data Collection for Interactive Learning through the Dialog" answer_l2: 0.9966132 # L2 distance for answer in all dialogs INCLUDING dialogs answered by dontknow answer_precision: 0.6000000 # Precision among answers EXCLUDING dialogs answered by dontknow answer_recall: 0.0031579 # Recall among answers EXCLUDING dialogs answered by dontknow extracted_answer_l2: 0.9883598 # L2 distance for extracted answer in all dialogs where answer hint was requested extracted_answer_precision: 0.5000000 # Precision of answer extraction in dialogs with answer hint containing correct answer extracted_answer_recall: 0.0057143 # Recall of answer extraction in dialogs with answer hint containing correct answer ==============COMUNICATION PROTOCOL============== Model communicates with the evaluation simmulator by sending json messages over standard input and output. The messages contain information about actuall dialog phase. We distinguish three dialog phases indicated by following action values: 1) new_dialog - question that should be answered during the dialog is posed in this first phase, {"action": "new_dialog", "question": "what is a recording from the physician"} 2) explanation_turns - simulator answers on request for broader explanation of the posed question, {"action": "explanation_turns", "turns": [...]} 3) answer_turns - simulation of rerouting the answer to another user which tries to answer the question. {"action": "answer_turns", "turns": [...]} The model can provide answer distribution (and stop simulation of current dialog) in every phase of the dialog by following message. {"answer": {"www.freebase.com/m/08jr8x": 0.3, "www.freebase.com/m/07s9rl0": 0.4, "www.freebase.com/m/03k9fj": 0.3}} The distribution for third phase of the dialog can contain a special value no_correct_answer, meaning that the answer hint does not contain a correct answer for the posed question. {"answer": null} If the model needs more information for the question, it can request it by following actions: 1) request_explanation - request for explanation_turns can be send in first phase of the dialog, {"action": "request_explanation"} 2) request_answer - request for answer_turns can be send in first or second phase of the dialog. {"action": "request_answer"} COMMUNICATION EXAMPLE: First dialog answered immediatly after the question is posed IN: {"action": "new_dialog", "question": "what is a recording from the physician"} OUT: {"answer": {"www.freebase.com/m/0drcrdd": 1.0}} Second dialog answered after the explanation IN: {"action": "new_dialog", "question": "which are the symptoms of tonsillolith?"} OUT: {"action": "request_explanation"} IN: { "action": "explanation_turns", "turns": [ { "output": { "dialog_acts": { "question": "which are the symptoms of tonsillolith?", "act": "WelcomeWithRephraseRequest" }, "transcript": "Hello, I need help with this question: 'which are the symptoms of tonsillolith?'. Can you put this question in a different way?", "time": "2016-03-17T12:41:02.98421+01:00" }, "turn_index": 1, "input": { "chat_slu": { "text": "What are the symptoms of tonsillolith?", "act": "Inform" }, "chat": "What are the symptoms of tonsillolith?", "time": "2016-03-17T12:42:01.92121+01:00" } }, { "output": { "dialog_acts": { "at_least": false, "act": "RequestExplanation" }, "transcript": "This seems like an interesting question. Could you explain in detail what it is really asking for?", "time": "2016-03-17T12:42:01.921758+01:00" }, "turn_index": 2, "input": { "chat_slu": { "text": "how does your body feel if you have contracted tonsillolith?", "act": "Inform" }, "chat": "how does your body feel if you have contracted tonsillolith?", "time": "2016-03-17T12:42:35.527993+01:00" } } ] } OUT: {"answer": {"www.freebase.com/m/0b76bty": 1.0}} Third dialog where all phases are present is not answered (dontknow message) IN: {"action": "new_dialog", "question": "what company developed the game version of lego star wars ii: the original trilogy?"} OUT: {"action": "request_explanation"} IN: { "action": "explanation_turns", "turns": [ { "output": { "dialog_acts": { "question": "what company developed the game version of lego star wars ii: the original trilogy?", "at_least": true, "act": "RephraseQuestionPropose" }, "transcript": "Ok, could you please rephrase this next question: 'what company developed the game version of lego star wars ii: the original trilogy?'?", "time": "2016-03-17T16:06:47.970723+01:00" }, "turn_index": 1, "input": { "chat_slu": { "text": "We are wondering which enterprise is behind the developpement of this trilogy.", "act": "Inform" }, "chat": "We are wondering which enterprise is behind the developpement of this trilogy.", "time": "2016-03-17T16:07:21.862559+01:00" } }, { "output": { "dialog_acts": { "act": "TooBriefRephrase" }, "transcript": "I am sorry, but your rephrasing seems too brief. Could you provide more details please?", "time": "2016-03-17T16:07:21.863993+01:00" }, "turn_index": 2, "input": { "chat_slu": { "text": "We are wondering which enterprise is behind the developpement of this game trilogy.", "act": "Inform" }, "chat": "We are wondering which enterprise is behind the developpement of this game trilogy.", "time": "2016-03-17T16:07:29.341545+01:00" } }, { "output": { "dialog_acts": { "at_least": false, "act": "RequestExplanation" }, "transcript": "This seems like an interesting question. Could you explain in detail what it is really asking for?", "time": "2016-03-17T16:07:29.343085+01:00" }, "turn_index": 3, "input": { "chat_slu": { "text": "We want to know who made it possible to play this game, who developped it.", "act": "Inform" }, "chat": "We want to know who made it possible to play this game, who developped it.", "time": "2016-03-17T16:07:55.415331+01:00" } } ] } OUT: {"action": "request_answer"} IN: { "action": "answer_turns", "turns": [ { "output": { "dialog_acts": { "at_least": false, "act": "RequestAnswer" }, "transcript": "It sounds reasonable, however, I still cannot find the answer. Could you give me the correct answer for the question please?", "time": "2016-03-17T16:07:55.420181+01:00" }, "turn_index": 4, "input": { "chat_slu": { "act": "DontKnow" }, "chat": "I'm sorry but I don't know the answer, again.", "time": "2016-03-17T16:08:05.97022+01:00" } } ] } OUT: {"answer": null}