DeFi SG Improvement proposal

DeFi Sub-Governance
#1

DeFi SG Improvement proposal

English version below

  1. DeFi SG - работает с одними из наиболее сложных ИТ и бизнес-задач. DeFi SG создал свой framework описания контеста , что позволяет формулировать достаточно детализированные и профессиональные требования к работам, предоставляемым на конкурсы.

  2. На недавнем созвоне DeFi SG состоялось обсуждение возможного развития системы оценивания работ. Было выдвинуто предложение использовать детализированные требования по конкурсам, как шаблон для оценивания результатов (конкурсных заявок). По мнению ряда участников такой подход позволит сделать процедуру оценивания более обоснованной, детализированной и прозрачной как для конкурсантов, так и для остальных участников DeFi SG.

  3. Связывание детализированных требований с системой оценивания является логичным развитием, потому что позволяет отслеживать и проверять требования сформулированные в конкурсе (traceability best-practice of requirements engineering discipline).

  4. На практике в простейшем виде, когда мы имеем два списка требований (Hard and Soft criterias), то оценивание по каждому критерию будет происходить в таблице - либо в бинарном виде (выполнено/не выполнено), либо по шкале 0-10. Соответственно общая оценка каждого судьи будет выводиться, как среднее из указанных им оценок.

  5. Более того, если конкурсные критерии имеют различный вес, то веса могут быть непосредственно указаны в условиях конкурса. В этом случае расчет общей оценки должен быть проведен с учетом веса каждого критерия.

Например, для DEX contest stage 3, возможен такой вариант:

Hard criteria:

Критерий Вес Оценка
Ability to create new pairs, exchange, view information about rates, composition of pools, existing pools, exchange statistics through the user interface 15%
Support for de-bot interface and / or web and mobile interface for real use 15%
Description and use of mechanisms for maintaining liquidity / order book / users in the system 15%
The product must be deployed on the FreeTON mainnet 15%

Soft criteria:

Критерий Вес Оценка
Using one of the solutions implemented in phase 1 and / or 2 10%
Mainly everyday English to make technical documentation easier to understand. Additional languages will be a plus. 10%
Completeness and readiness of the product for real use in the main network 10%
Additional interfaces, de-bots, mobile apps, SDK and public APIs will be a plus 10%
  1. В случае наличия неформальных критериев оценивания, например по экономической модели предложенного на конкурс решения, такие критерии с определенным и прописанным в условиях конкурса весом включаются в общий список.

Пожалуйста, поддержите и прокомментируйте!

English version:

  1. DeFi SG hosts some of the most complex contests both from the technical and business points of view. DeFi SG created its own comprehensive contest framework that allows it to launch contests with well-defined and clear requirements.

  2. During the recent DeFi SG weekly call there was a discussion about the contest submission evaluation process. It was proposed to start using detailed requirements and criteria for contests as a template for evaluation of contest submissions. Some of the DeFi SG participants are convinced that using this approach will make the evaluation process more solid, comprehensive and transparent both for the contestants and other DeFi SG participants.

  3. Linking the detailed requirements to the evaluation process is a logical evolution of the scoring system since it allows trace and validate requirements defined in a contest (according to best-practice of requirements engineering discipline).

  4. In practice when we have 2 sets of criteria (Hard and Soft), it is proposed to score the submissions by each criteria using a spreadsheet either in a binary manner (yes/no) or using a 0-10 scale. Consequently a unified score from each jury will be calculated as an average from all the scores.

  5. Moreover if the contest criteria have different weights these parameters can be defined in the contest terms. In this case the calculation of the total score must be done using these weights.

For example, for DEX Stage 3 contest a system like this is possible:

Hard criteria:

Criteria Weight Score
Ability to create new pairs, exchange, view information about rates, composition of pools, existing pools, exchange statistics through the user interface 15%
Support for de-bot interface and / or web and mobile interface for real use 15%
Description and use of mechanisms for maintaining liquidity / order book / users in the system 15%
The product must be deployed on the FreeTON mainnet 15%

Soft criteria:

Criteria Weight Score
Using one of the solutions implemented in phase 1 and / or 2 10%
Mainly everyday English to make technical documentation easier to understand. Additional languages will be a plus. 10%
Completeness and readiness of the product for real use in the main network 10%
Additional interfaces, de-bots, mobile apps, SDK and public APIs will be a plus 10%
  1. In case there are non-formal criteria like “good economy of the solution” these criteria along with a defined weight can be included into the list.

Please support and comment!

7 Likes
#2

Thank you for your proposal. In general, the preparation of weighted assessment criteria is a good approach. However, selecting weights and choosing criteria is not a simple task.

In your example above, you devoted 40% of the score weight to soft criteria, which ruins the idea of splitting between hard and soft ones. Soft criteria are nice to have but not obligatory. A decent work that complies with all hard criteria shall have a chance to get a score of 10 without implementing soft criteria.

That is precisely what I’ve told on one of DeFi calls: the main reason for creating an assessment framework is to give participants enough confidence that jurors score their work fair. But such a framework may differ between jurors depending on their skills and knowledge: techy guys will judge more the source code part, whereas economists will pay attention to how balanced the product’s financial side is.

Additionally, it is always good to remember that all jurors are publicly available, and it’s up to you to present your work and explain its pros and cons through an AMA session.