When Arbitrum protocols look for audits, they look to Sherlock.
In the past year, Sherlock has conducted 47 audits for protocol teams that have either deployed on Arbitrum or publicly plan to deploy on Arbitrum including GMX, KyberSwap, and Lyra. Sherlock has prevented 183 Critical severity vulnerabilities from reaching Arbitrum mainnet across protocols representing $470M of Arbitrum’s TVL.
Sherlock is requesting $4M — not for Sherlock — but for Arbitrum protocols, so they can access reduced-cost audits from Sherlock through the Ecosystem Security Fund. This initiative will enhance the security of the Arbitrum ecosystem for the long term as well as entice up-and-coming protocol teams to build on Arbitrum.
Arbitrum has one of the most vibrant developer communities and dedicated user bases in the entire Web3 space. The recent STIP is proof that the Arbitrum Foundation always takes the long-term well-being of their projects into account. However, there is one aspect that can pose a great threat to any ecosystem and requires constant vigilance: security.
Arbitrum projects’ success depends on their commitment to security. Exploits can tarnish user trust and attract significant negative media coverage. Moreover, due to smart contracts' composability, vulnerabilities can spread easily.
As Pew Research Center highlights, distrust in the security and reliability of Web3 projects is a major barrier to crypto adoption.
Without adequate security, scaling and long-lasting adoption is not possible.
Sherlock’s primary mission since Day 1 has been protecting end-users, who usually are the most affected by hacks and exploits. Sherlock aims to fortify the broader crypto industry, where Arbitrum plays a significant role.
The Ecosystem Security Fund proposal will enhance the security of the Arbitrum ecosystem, one project at a time.
The Ecosystem Security Fund as well as Sherlock’s own values directly align with the Arbitrum Community Values described in Section 6 of the Arbitrum DAO Constitution.
Encouraging and subsidizing audits would not only attract far more protocols to Arbitrum (audits are expensive), but it would also be a very commendable move by the Arbitrum DAO which will enhance the security of crypto as a whole, no matter which chains a protocol ends up on.
From Sherlock’s perspective, Arbitrum has become a very popular chain for builders. Sherlock is very grateful to have been able to help many protocols that have either deployed on Arbitrum or declared to Sherlock’s auditors that they plan to do so in the future. Here’s a list of Arbitrum-aligned projects that Sherlock has audited in the past year:
References
A high percentage of these protocols have come back for multiple audits from Sherlock. Here are a few things they’ve said about Sherlock:
Notional (V3 is committed to Arbitrum): “Notional has gotten 14 audits from 6 different firms, and ever since we first used Sherlock in October of 2022 they have been, and will continue to be for the foreseeable future, our exclusive audit provider. Sherlock is the best audit experience we've ever had, hands down.”
Perennial (live on Arbitrum and the next version is in a Sherlock audit): “Perennial has done multiple audits with Sherlock and has been continually impressed with the process, from scheduling to onboarding to operation the Sherlock team makes it extremely easy to get audits done. Having used both audit firms and other contest audit platforms, we have found the auditors in Sherlock's contests to be exceptional - finding numerous complex and subtle bugs that had otherwise gone unnoticed. We are excited to continue to use Sherlock for future protocol upgrades.”
Index Coop (latest version is planned for Arbitrum): “Index Coop has had many audits performed by top-tier firms and independent security researchers, but none of them compare to Sherlock in terms of coverage, comprehension, and collaboration. The contest model paired with a dedicated security researcher has led to the highest impact outcomes for our protocol and we consistently recommend Sherlock to other protocols regardless of what stage they are in.”
See here for a supplementary list of Arbitrum-aligned teams that have spoken about Sherlock.
One of the main reasons why so many Arbitrum-aligned projects elect to trust Sherlock with their codebase security is Sherlock’s “best of both worlds” approach. The section below will focus on briefly exploring why it's a superior model, supported by raw data.
Currently, there are 2 main approaches to audits: Traditional and Contest.
Traditional: 2-4 security experts pore over the codebase for a few weeks and flag any vulnerabilities they see.
Contest: A protocol team creates a prize pool ($30k-$200k usually), and anyone in the world can submit vulnerabilities. The higher the severity, the more of the prize pool the security expert is awarded.
Sherlock: Sherlock’s approach provides a protocol with the focus, collaboration, and assurance of a traditional audit, alongside the breadth of security expert participation from an audit contest.
Benefits of Sherlock’s approach:
How does Sherlock know the model is effective?
Sherlock’s claims are supported by data gathered from 94 contests, where over 1000 vulnerabilities were found. The image below showcases how many bugs are missed if an audit is done without a reserved auditor (such as a Senior Watson) or without a community contest.
Can you afford to NOT include both?
Sherlock appoints a lead auditor from a roster of elite senior security experts, known as Senior Watsons who are identified by a pink crown. Notable experts on this list among many others include Xiaoming90, hyh, WatchPug, and 0x52. They earn attractive fixed pay for leading the contest along with the ability to compete for the entire prize pool.
The Lead Senior Watson's fixed role ensures that each contest will have at least one expert auditor thoroughly reviewing the codebase, consulting the protocol team, and conducting a complimentary half-day fix review to ensure any fixes have been implemented securely.
The talents of this one individual will be enhanced by competing against an unlimited number of independent security experts / teams striving to win a greater portion of the prize pool. The Senior Watson only keeps their “senior” status as long as they outperform the other auditors in the field, pushing them to give maximum effort.
At the core of this model is Sherlock’s continuous commitment to incentive alignment.
As advised by the delegates that Sherlock worked with to craft this proposal, Sherlock will create an Audit Council. The role of the Audit Council is to be the final gatekeeper/decision-maker to determine which projects building on Arbitrum (or planning to build on Arbitrum) are eligible to get discounted audits as part of the program. Sherlock will form this council and the associated on-chain Gnosis multisig within 1 week of the proposal passing to allow projects to take advantage of the program as soon as possible.
5 people will be elected to the multisig (3 representatives from Arbitrum, and 2 representatives from Sherlock). The Arbitrum members will be 2 representatives from the Arbitrum DAO and 1 member of the Arbitrum Security Council. Both Sherlock and Arbitrum will finalize their list of members for the Gnosis multisig within 7 days of proposal approval.
The council will have the ability to add/remove any extra eligibility criteria as deemed appropriate to make this initiative as successful as possible for Arbitrum and protocol teams before the requests from protocol teams start coming in.
The main responsibilities of this council will be:
Protocol team requests will begin evaluation within two weeks of this proposal’s approval. The council will conduct continuous reviews for audit requests, prioritizing rapid access to quality audits. Sherlock pledges to provide a quote within 72 hours of receiving the relevant codebase, considering its size and complexity. Further procedures for request management will be determined by council consensus.
Additionally, Sherlock commits to community involvement in refining this partnership. Gaining consensus from protocols on Arbitrum is essential to ensure the council's procedures are effective.
Sherlock will consistently seek feedback from Arbitrum's protocol teams to confirm the request process is manageable and that the provided timelines and results meet protocol teams' needs, aiming to make Arbitrum an attractive platform for developers.
The Ecosystem Security Fund aims to provide premium security audits for protocols on Arbitrum with substantial TVL/users, or those planning to join that are expected to attract substantial TVL/users. The Audit Council will require data on TVL, user statistics, and deployment strategy to make their assessment.
Prioritization is given to protocols that substantially benefit the Arbitrum ecosystem and there is no favouritism for certain categories (DeFi, Gaming, SocialFi, etc.) over others.
Protocols must meet one of the following criteria for consideration:
Note: Compliance with any or all of the 3 criteria does not guarantee acceptance into the Ecosystem Security Fund.
Protocols with characteristics disallowed by the Arbitrum Foundation, detailed in the Ineligible Application section of their Submission Guidelines, are ineligible for inclusion. The Audit Council reserves the right to modify eligibility criteria for protocol requests as necessary.
100% of funds dedicated to this program will be used to audit Arbitrum projects.
Here’s the expected breakdown for purchasing an audit through the program:
Note: For protocols that have (or are expected to have) an outsized impact on the Arbitrum ecosystem, the Audit Council will have the ability to increase the % of the audit paid by the Ecosystem Security Fund up to a maximum of 80%.
Requiring the protocol to pay for up to 40% of the audit costs ensures that audits are 2.5x more affordable for eligible protocol teams while still making sure that there is enough cost (”skin in the game”) associated with every audit so that protocol teams don’t abuse the program.
Sherlock’s “final exam” audit approach is best suited to the Ecosystem Security Fund because it offers the greatest capital efficiency while minimizing the possibility of Critical severity vulnerabilities.
Sherlock’s audit pricing is substantively based on nSLOC. But Sherlock also has an in-house scoping team that has the authority to add a qualitative aspect to pricing to take into account extra complexity (assembly code, etc.).
Sherlock’s pricing is based on years of experience attracting top talent to audits and 80% of the cost of each audit goes directly to security experts (the highest ratio in the audit industry). Sherlock has engineered its pricing to attract a top-ranked Senior Watson to lead each audit. For example, Sherlock has experimented with weekly fixed pay for Senior Watsons from $7.5k to $22.5k and has landed on the current schema as being optimal. Sherlock also ensures the audit will attract hundreds of independent auditors and teams, optimizing for the highest quality audit possible.
The nSLOC (adjusted for qualitative input) drives both the price and length of the audit. Here are some example audit quotes:
Sherlock works closely with top Senior Watsons to ensure the cost and length of each audit are correct. Sherlock’s average audit is <2,500 nSLOC and Sherlock finds 13 important issues and 4 Critical severity issues on average for each audit.
A detailed overview of Sherlock’s pricing system can be viewed here: https://docs.google.com/spreadsheets/d/1wDGFivAYyHYBJCY5WQY6CReJ4E2jiDz7_9Vheh5Tqs8/edit?usp=sharing (use the provided comments to make understanding the table easier)
At the end of every quarter, Sherlock will return any unspent amount (based on a $1M quarterly allocation) to the Arbitrum DAO.
Within three months of proposal acceptance, Sherlock will:
$4M will be allocated to the Ecosystem Security Fund, but any unused funds will be returned every quarter if $1M is not spent on audits in that quarter.
These funds will be 100% reserved for Sherlock audits done on qualifying protocols building on Arbitrum or planning to build on Arbitrum.
The Audit Council’s multisig will be the steward of the funds. For ease of implementation, the entire $4M amount can be sent to the Audit Council’s multisig on acceptance of this proposal. And the Audit Council will remit any unused portion of $1M every quarter.
The $4M should not be sent to the Audit Council multisig until all 5 members of the multisig have been successfully added and have completed a test transaction using the multisig. This is to reduce any chance for error and ensure that Arbitrum’s 3 council members have a voting majority with respect to the $4M at all times.
Sherlock completed 47 audits for Arbitrum-aligned protocols in the past year. This can serve as a baseline for audit demand prior to the Ecosystem Security Fund's 60% subsidy. With this incentive, the annual demand for Sherlock audits on Arbitrum protocols is expected to be significantly higher.
How many audits can this initiative fund?
To determine this, audits are categorized into three complexity levels, which primarily influence cost. Below are the criteria for each category with a short description.
Size: Size of the codebase; Cost: Average cost of the audit
Why $4M for the Ecosystem Security Fund?
In the past 12 months, Sherlock has done 47 audits for Arbitrum-aligned projects, many of whom could fit the eligibility criteria for the Ecosystem Security Fund. According to historic data Sherlock did 5 High, 15 Medium and 27 Low Complexity audits which amount to:
This $2M serves as an estimate of the past year's audit demand from eligible projects. Sherlock reached this volume without any significant discounts. A 60% discount could potentially triple the number of interested projects. If audit volume reaches $6M (a 3x increase) and the fund subsidizes an average of 67% (since some projects can be eligible for an 80% discount) of the audit costs, a $4M fund would be fully utilized.
The $4M fund could provide an estimated 141 audits (47 last year x 3) for approved protocols over the course of a year. Given that last year's 47 audits secured 27 projects (57%), Sherlock projects that 141 audits could secure approximately 80 Arbitrum projects in total.
What KPIs can be used to determine the Ecosystem Security Fund’s success?
| Audits Approved | Audits Scheduled | Audits Completed | Protocols Deployed on Arbitrum | On-Chain Exploits | |
|---|---|---|---|---|---|
| KPI | 25 audits approved by the Audit Council | 20 audits scheduled with Sherlock | 20 audit reports delivered | 20 protocols deployed to Arbitrum | 0 on-chain exploits for audited protocols |
| Deadline | 3 months after the proposal passes | 3 months after the proposal passes | 4 months after the proposal passes | 4 months after the proposal passes | 6 months after the proposal passes |
| Source of Truth | Quarterly report linking on-chain payments with projects | https://audits.sherlock.xyz/contests | Sherlock Audit Archives | Arbitrum deployment addresses | TVL drops (on-chain) & media (off-chain) |
Starting 13 weeks after this proposal is approved, Sherlock will issue quarterly financial reports to the Arbitrum DAO forum. These reports will contain key metrics enabling DAO members to evaluate the program's effectiveness. A sample report with dummy data is available here.
Operational Updates
For complete transparency, Sherlock has set up a Telegram channel and bot specifically dedicated to the Ecosystem Security Fund. To stay updated, you can join the channel. The bot will disseminate real-time information about protocol teams navigating the audit process.
Click here for a preview. The table below outlines the types of updates the bot provides. Additionally, Sherlock will accommodate any extra update requests from the Audit Council.
| Trigger | Announcement |
|---|---|
| New Audit Request | {Project Name} Requested an Audit. Link: {Website/Twitter}; Size: {nSLOC}; Cost: {Total Price} |
| Audit Approved/Rejected | The request made by {Project Name} has been approved/rejected by the Audit Council. Confirmation Link: {ArbiScan Link} |
| Senior Selected | Sherlock has selected {name} to be the Lead Senior Watson for {Project Name}! Here’s more info about their background: {Senior Watson profile link} |
| Contest Announcement | {Project Name} audit contest has been announced! {link to twitter} |
| Contest Started | {Project Name} audit contest is officially live! |
| Contest Finalized | {Project Name} final audit and judging contest rewards for Watsons have been announced! |
On-chain Address verification
The Audit Council multisig should ONLY ever send funds to Sherlock’s multisig address on Arbitrum and back to the address from the Arbitrum DAO that initially funds it.
Sherlock’s multisig address on Arbitrum: 0xBe427a7fA085B22fF4928815DF3c2948509d36D3
| Risk | Mitigation |
|---|---|
| The Arbitrum DAO faces the potential of investing millions in a product that protocol teams may not desire or deem valuable. | Sherlock is one of the most in-demand auditors in the industry (measured by $ volume of audits) and among Arbitrum-aligned projects. Sherlock's audits with a Security Fund is a powerful combination, and Sherlock's exceptional BD team will ensure widespread awareness among relevant projects. |
| Sherlock audits may not be effective enough at preventing vulnerabilities. | To gauge Sherlock's effectiveness in identifying critical vulnerabilities compared to other auditors, consider consulting with protocol teams like Ajna, Index Coop, and Tokemak who have undergone multiple concurrent audits by Sherlock. |
| The $4M budget for this proposal may be excessive with limited interest from protocol teams. | If the $4M budget proves excessive, Sherlock will refund any unutilized funds within the quarter ($1M allocated per quarter). Further returns of funds may be considered if future quarters indicate a low probability of reaching $1M. |
| The $4M budget may prove insufficient, limiting participation from protocol teams. | In a positive light, this suggests strong builder engagement with the program. Arbitrum DAO can opt to maintain the current budget or increase quarterly allocations after hitting the $1M cap for two or more quarters. |
| Sherlock may struggle to uphold top-tier audit quality as demand from protocols increases. | Sherlock boasts a highly scalable audit model, consistently drawing over 300 experts per audit. Nevertheless, Sherlock maintains a designated list of Senior Watsons, and no audits occur without their involvement, which has not occurred thus far despite sustained increased demand. |
| A potential conflict of interest arises as Sherlock both conducts audits and participates in audit approval decisions through the Audit Council. | The Audit Council will consist of 2 signers from Sherlock and 3 signers selected by Arbitrum. This arrangement ensures Sherlock will never be able to “force” an approval decision with its 2 signers. |
In the past year, Sherlock has conducted 47 audits and surfaced 183 Critical severity vulnerabilities for protocols building on Arbitrum, including $470M of the current TVL on Arbitrum. There is likely no other auditor that Arbitrum protocols trust more than Sherlock.
The advent of the Ecosystem Security Fund signals a step-change commitment to security by Arbitrum, and Sherlock is excited to be at the forefront of such an initiative.
When Arbitrum protocols look for audits, they look to Sherlock.
In the past year, Sherlock has conducted 47 audits for protocol teams that have either deployed on Arbitrum or publicly plan to deploy on Arbitrum including GMX, KyberSwap, and Lyra. Sherlock has prevented 183 Critical severity vulnerabilities from reaching Arbitrum mainnet across protocols representing $470M of Arbitrum’s TVL.
Sherlock is requesting $4M — not for Sherlock — but for Arbitrum protocols, so they can access reduced-cost audits from Sherlock through the Ecosystem Security Fund. This initiative will enhance the security of the Arbitrum ecosystem for the long term as well as entice up-and-coming protocol teams to build on Arbitrum.
Arbitrum has one of the most vibrant developer communities and dedicated user bases in the entire Web3 space. The recent STIP is proof that the Arbitrum Foundation always takes the long-term well-being of their projects into account. However, there is one aspect that can pose a great threat to any ecosystem and requires constant vigilance: security.
Arbitrum projects’ success depends on their commitment to security. Exploits can tarnish user trust and attract significant negative media coverage. Moreover, due to smart contracts' composability, vulnerabilities can spread easily.
As Pew Research Center highlights, distrust in the security and reliability of Web3 projects is a major barrier to crypto adoption.
Without adequate security, scaling and long-lasting adoption is not possible.
Sherlock’s primary mission since Day 1 has been protecting end-users, who usually are the most affected by hacks and exploits. Sherlock aims to fortify the broader crypto industry, where Arbitrum plays a significant role.
The Ecosystem Security Fund proposal will enhance the security of the Arbitrum ecosystem, one project at a time.
The Ecosystem Security Fund as well as Sherlock’s own values directly align with the Arbitrum Community Values described in Section 6 of the Arbitrum DAO Constitution.
Encouraging and subsidizing audits would not only attract far more protocols to Arbitrum (audits are expensive), but it would also be a very commendable move by the Arbitrum DAO which will enhance the security of crypto as a whole, no matter which chains a protocol ends up on.
From Sherlock’s perspective, Arbitrum has become a very popular chain for builders. Sherlock is very grateful to have been able to help many protocols that have either deployed on Arbitrum or declared to Sherlock’s auditors that they plan to do so in the future. Here’s a list of Arbitrum-aligned projects that Sherlock has audited in the past year:
References
A high percentage of these protocols have come back for multiple audits from Sherlock. Here are a few things they’ve said about Sherlock:
Notional (V3 is committed to Arbitrum): “Notional has gotten 14 audits from 6 different firms, and ever since we first used Sherlock in October of 2022 they have been, and will continue to be for the foreseeable future, our exclusive audit provider. Sherlock is the best audit experience we've ever had, hands down.”
Perennial (live on Arbitrum and the next version is in a Sherlock audit): “Perennial has done multiple audits with Sherlock and has been continually impressed with the process, from scheduling to onboarding to operation the Sherlock team makes it extremely easy to get audits done. Having used both audit firms and other contest audit platforms, we have found the auditors in Sherlock's contests to be exceptional - finding numerous complex and subtle bugs that had otherwise gone unnoticed. We are excited to continue to use Sherlock for future protocol upgrades.”
Index Coop (latest version is planned for Arbitrum): “Index Coop has had many audits performed by top-tier firms and independent security researchers, but none of them compare to Sherlock in terms of coverage, comprehension, and collaboration. The contest model paired with a dedicated security researcher has led to the highest impact outcomes for our protocol and we consistently recommend Sherlock to other protocols regardless of what stage they are in.”
See here for a supplementary list of Arbitrum-aligned teams that have spoken about Sherlock.
One of the main reasons why so many Arbitrum-aligned projects elect to trust Sherlock with their codebase security is Sherlock’s “best of both worlds” approach. The section below will focus on briefly exploring why it's a superior model, supported by raw data.
Currently, there are 2 main approaches to audits: Traditional and Contest.
Traditional: 2-4 security experts pore over the codebase for a few weeks and flag any vulnerabilities they see.
Contest: A protocol team creates a prize pool ($30k-$200k usually), and anyone in the world can submit vulnerabilities. The higher the severity, the more of the prize pool the security expert is awarded.
Sherlock: Sherlock’s approach provides a protocol with the focus, collaboration, and assurance of a traditional audit, alongside the breadth of security expert participation from an audit contest.
Benefits of Sherlock’s approach:
How does Sherlock know the model is effective?
Sherlock’s claims are supported by data gathered from 94 contests, where over 1000 vulnerabilities were found. The image below showcases how many bugs are missed if an audit is done without a reserved auditor (such as a Senior Watson) or without a community contest.
Can you afford to NOT include both?
Sherlock appoints a lead auditor from a roster of elite senior security experts, known as Senior Watsons who are identified by a pink crown. Notable experts on this list among many others include Xiaoming90, hyh, WatchPug, and 0x52. They earn attractive fixed pay for leading the contest along with the ability to compete for the entire prize pool.
The Lead Senior Watson's fixed role ensures that each contest will have at least one expert auditor thoroughly reviewing the codebase, consulting the protocol team, and conducting a complimentary half-day fix review to ensure any fixes have been implemented securely.
The talents of this one individual will be enhanced by competing against an unlimited number of independent security experts / teams striving to win a greater portion of the prize pool. The Senior Watson only keeps their “senior” status as long as they outperform the other auditors in the field, pushing them to give maximum effort.
At the core of this model is Sherlock’s continuous commitment to incentive alignment.
As advised by the delegates that Sherlock worked with to craft this proposal, Sherlock will create an Audit Council. The role of the Audit Council is to be the final gatekeeper/decision-maker to determine which projects building on Arbitrum (or planning to build on Arbitrum) are eligible to get discounted audits as part of the program. Sherlock will form this council and the associated on-chain Gnosis multisig within 1 week of the proposal passing to allow projects to take advantage of the program as soon as possible.
5 people will be elected to the multisig (3 representatives from Arbitrum, and 2 representatives from Sherlock). The Arbitrum members will be 2 representatives from the Arbitrum DAO and 1 member of the Arbitrum Security Council. Both Sherlock and Arbitrum will finalize their list of members for the Gnosis multisig within 7 days of proposal approval.
The council will have the ability to add/remove any extra eligibility criteria as deemed appropriate to make this initiative as successful as possible for Arbitrum and protocol teams before the requests from protocol teams start coming in.
The main responsibilities of this council will be:
Protocol team requests will begin evaluation within two weeks of this proposal’s approval. The council will conduct continuous reviews for audit requests, prioritizing rapid access to quality audits. Sherlock pledges to provide a quote within 72 hours of receiving the relevant codebase, considering its size and complexity. Further procedures for request management will be determined by council consensus.
Additionally, Sherlock commits to community involvement in refining this partnership. Gaining consensus from protocols on Arbitrum is essential to ensure the council's procedures are effective.
Sherlock will consistently seek feedback from Arbitrum's protocol teams to confirm the request process is manageable and that the provided timelines and results meet protocol teams' needs, aiming to make Arbitrum an attractive platform for developers.
The Ecosystem Security Fund aims to provide premium security audits for protocols on Arbitrum with substantial TVL/users, or those planning to join that are expected to attract substantial TVL/users. The Audit Council will require data on TVL, user statistics, and deployment strategy to make their assessment.
Prioritization is given to protocols that substantially benefit the Arbitrum ecosystem and there is no favouritism for certain categories (DeFi, Gaming, SocialFi, etc.) over others.
Protocols must meet one of the following criteria for consideration:
Note: Compliance with any or all of the 3 criteria does not guarantee acceptance into the Ecosystem Security Fund.
Protocols with characteristics disallowed by the Arbitrum Foundation, detailed in the Ineligible Application section of their Submission Guidelines, are ineligible for inclusion. The Audit Council reserves the right to modify eligibility criteria for protocol requests as necessary.
100% of funds dedicated to this program will be used to audit Arbitrum projects.
Here’s the expected breakdown for purchasing an audit through the program:
Note: For protocols that have (or are expected to have) an outsized impact on the Arbitrum ecosystem, the Audit Council will have the ability to increase the % of the audit paid by the Ecosystem Security Fund up to a maximum of 80%.
Requiring the protocol to pay for up to 40% of the audit costs ensures that audits are 2.5x more affordable for eligible protocol teams while still making sure that there is enough cost (”skin in the game”) associated with every audit so that protocol teams don’t abuse the program.
Sherlock’s “final exam” audit approach is best suited to the Ecosystem Security Fund because it offers the greatest capital efficiency while minimizing the possibility of Critical severity vulnerabilities.
Sherlock’s audit pricing is substantively based on nSLOC. But Sherlock also has an in-house scoping team that has the authority to add a qualitative aspect to pricing to take into account extra complexity (assembly code, etc.).
Sherlock’s pricing is based on years of experience attracting top talent to audits and 80% of the cost of each audit goes directly to security experts (the highest ratio in the audit industry). Sherlock has engineered its pricing to attract a top-ranked Senior Watson to lead each audit. For example, Sherlock has experimented with weekly fixed pay for Senior Watsons from $7.5k to $22.5k and has landed on the current schema as being optimal. Sherlock also ensures the audit will attract hundreds of independent auditors and teams, optimizing for the highest quality audit possible.
The nSLOC (adjusted for qualitative input) drives both the price and length of the audit. Here are some example audit quotes:
Sherlock works closely with top Senior Watsons to ensure the cost and length of each audit are correct. Sherlock’s average audit is <2,500 nSLOC and Sherlock finds 13 important issues and 4 Critical severity issues on average for each audit.
A detailed overview of Sherlock’s pricing system can be viewed here: https://docs.google.com/spreadsheets/d/1wDGFivAYyHYBJCY5WQY6CReJ4E2jiDz7_9Vheh5Tqs8/edit?usp=sharing (use the provided comments to make understanding the table easier)
At the end of every quarter, Sherlock will return any unspent amount (based on a $1M quarterly allocation) to the Arbitrum DAO.
Within three months of proposal acceptance, Sherlock will:
$4M will be allocated to the Ecosystem Security Fund, but any unused funds will be returned every quarter if $1M is not spent on audits in that quarter.
These funds will be 100% reserved for Sherlock audits done on qualifying protocols building on Arbitrum or planning to build on Arbitrum.
The Audit Council’s multisig will be the steward of the funds. For ease of implementation, the entire $4M amount can be sent to the Audit Council’s multisig on acceptance of this proposal. And the Audit Council will remit any unused portion of $1M every quarter.
The $4M should not be sent to the Audit Council multisig until all 5 members of the multisig have been successfully added and have completed a test transaction using the multisig. This is to reduce any chance for error and ensure that Arbitrum’s 3 council members have a voting majority with respect to the $4M at all times.
Sherlock completed 47 audits for Arbitrum-aligned protocols in the past year. This can serve as a baseline for audit demand prior to the Ecosystem Security Fund's 60% subsidy. With this incentive, the annual demand for Sherlock audits on Arbitrum protocols is expected to be significantly higher.
How many audits can this initiative fund?
To determine this, audits are categorized into three complexity levels, which primarily influence cost. Below are the criteria for each category with a short description.
Size: Size of the codebase; Cost: Average cost of the audit
Why $4M for the Ecosystem Security Fund?
In the past 12 months, Sherlock has done 47 audits for Arbitrum-aligned projects, many of whom could fit the eligibility criteria for the Ecosystem Security Fund. According to historic data Sherlock did 5 High, 15 Medium and 27 Low Complexity audits which amount to:
This $2M serves as an estimate of the past year's audit demand from eligible projects. Sherlock reached this volume without any significant discounts. A 60% discount could potentially triple the number of interested projects. If audit volume reaches $6M (a 3x increase) and the fund subsidizes an average of 67% (since some projects can be eligible for an 80% discount) of the audit costs, a $4M fund would be fully utilized.
The $4M fund could provide an estimated 141 audits (47 last year x 3) for approved protocols over the course of a year. Given that last year's 47 audits secured 27 projects (57%), Sherlock projects that 141 audits could secure approximately 80 Arbitrum projects in total.
What KPIs can be used to determine the Ecosystem Security Fund’s success?
| Audits Approved | Audits Scheduled | Audits Completed | Protocols Deployed on Arbitrum | On-Chain Exploits | |
|---|---|---|---|---|---|
| KPI | 25 audits approved by the Audit Council | 20 audits scheduled with Sherlock | 20 audit reports delivered | 20 protocols deployed to Arbitrum | 0 on-chain exploits for audited protocols |
| Deadline | 3 months after the proposal passes | 3 months after the proposal passes | 4 months after the proposal passes | 4 months after the proposal passes | 6 months after the proposal passes |
| Source of Truth | Quarterly report linking on-chain payments with projects | https://audits.sherlock.xyz/contests | Sherlock Audit Archives | Arbitrum deployment addresses | TVL drops (on-chain) & media (off-chain) |
Starting 13 weeks after this proposal is approved, Sherlock will issue quarterly financial reports to the Arbitrum DAO forum. These reports will contain key metrics enabling DAO members to evaluate the program's effectiveness. A sample report with dummy data is available here.
Operational Updates
For complete transparency, Sherlock has set up a Telegram channel and bot specifically dedicated to the Ecosystem Security Fund. To stay updated, you can join the channel. The bot will disseminate real-time information about protocol teams navigating the audit process.
Click here for a preview. The table below outlines the types of updates the bot provides. Additionally, Sherlock will accommodate any extra update requests from the Audit Council.
| Trigger | Announcement |
|---|---|
| New Audit Request | {Project Name} Requested an Audit. Link: {Website/Twitter}; Size: {nSLOC}; Cost: {Total Price} |
| Audit Approved/Rejected | The request made by {Project Name} has been approved/rejected by the Audit Council. Confirmation Link: {ArbiScan Link} |
| Senior Selected | Sherlock has selected {name} to be the Lead Senior Watson for {Project Name}! Here’s more info about their background: {Senior Watson profile link} |
| Contest Announcement | {Project Name} audit contest has been announced! {link to twitter} |
| Contest Started | {Project Name} audit contest is officially live! |
| Contest Finalized | {Project Name} final audit and judging contest rewards for Watsons have been announced! |
On-chain Address verification
The Audit Council multisig should ONLY ever send funds to Sherlock’s multisig address on Arbitrum and back to the address from the Arbitrum DAO that initially funds it.
Sherlock’s multisig address on Arbitrum: 0xBe427a7fA085B22fF4928815DF3c2948509d36D3
| Risk | Mitigation |
|---|---|
| The Arbitrum DAO faces the potential of investing millions in a product that protocol teams may not desire or deem valuable. | Sherlock is one of the most in-demand auditors in the industry (measured by $ volume of audits) and among Arbitrum-aligned projects. Sherlock's audits with a Security Fund is a powerful combination, and Sherlock's exceptional BD team will ensure widespread awareness among relevant projects. |
| Sherlock audits may not be effective enough at preventing vulnerabilities. | To gauge Sherlock's effectiveness in identifying critical vulnerabilities compared to other auditors, consider consulting with protocol teams like Ajna, Index Coop, and Tokemak who have undergone multiple concurrent audits by Sherlock. |
| The $4M budget for this proposal may be excessive with limited interest from protocol teams. | If the $4M budget proves excessive, Sherlock will refund any unutilized funds within the quarter ($1M allocated per quarter). Further returns of funds may be considered if future quarters indicate a low probability of reaching $1M. |
| The $4M budget may prove insufficient, limiting participation from protocol teams. | In a positive light, this suggests strong builder engagement with the program. Arbitrum DAO can opt to maintain the current budget or increase quarterly allocations after hitting the $1M cap for two or more quarters. |
| Sherlock may struggle to uphold top-tier audit quality as demand from protocols increases. | Sherlock boasts a highly scalable audit model, consistently drawing over 300 experts per audit. Nevertheless, Sherlock maintains a designated list of Senior Watsons, and no audits occur without their involvement, which has not occurred thus far despite sustained increased demand. |
| A potential conflict of interest arises as Sherlock both conducts audits and participates in audit approval decisions through the Audit Council. | The Audit Council will consist of 2 signers from Sherlock and 3 signers selected by Arbitrum. This arrangement ensures Sherlock will never be able to “force” an approval decision with its 2 signers. |
In the past year, Sherlock has conducted 47 audits and surfaced 183 Critical severity vulnerabilities for protocols building on Arbitrum, including $470M of the current TVL on Arbitrum. There is likely no other auditor that Arbitrum protocols trust more than Sherlock.
The advent of the Ecosystem Security Fund signals a step-change commitment to security by Arbitrum, and Sherlock is excited to be at the forefront of such an initiative.
Sherlock's proposal and supporting evidence for exploring Arbitrum in the secure dApp ecology are impressive, and JOJO Exchange fully supports it!
Sherlock's proposal and supporting evidence for exploring Arbitrum in the secure dApp ecology are impressive, and JOJO Exchange fully supports it!
Might take a look at @dk3 proposal on the RFP.
Good added metrics if fn. reports are out...
Any improvement in security is a good expansion in my book.
Great @dk3 , appreciate the best effort.
After discussing the new RFC proposal with DK to understand their vision, Sherlock believes that an RFP-style process for auditing services is in the best interest of the DAO (if executed well). Because of this, Sherlock has decided not to post this proposal to Snapshot, and Sherlock will instead direct its coalition of delegates to vote for DK's RFC and to vote against all other auditor-specific proposals.
Of course, a lot of effort is needed to turn DK's RFC into a fully fleshed-out version that everyone can rally behind. Sherlock has successfully been through similar RFP processes with other DAOs and Sherlock is happy to provide feedback to DK and others as to what works well and what causes issues.
After discussing the new RFC proposal with DK to understand their vision, Sherlock believes that an RFP-style process for auditing services is in the best interest of the DAO (if executed well). Because of this, Sherlock has decided not to post this proposal to Snapshot, and Sherlock will instead direct its coalition of delegates to vote for DK's RFC and to vote against all other auditor-specific proposals.
Of course, a lot of effort is needed to turn DK's RFC into a fully fleshed-out version that everyone can rally behind. Sherlock has successfully been through similar RFP processes with other DAOs and Sherlock is happy to provide feedback to DK and others as to what works well and what causes issues.
To confirm, Sherlock is withdrawing its own proposal and putting its endorsement (and delegates) behind DK's RFC because Sherlock believes the RFP process approach is in the best interest of the Arbitrum DAO.
Hey @jengajojo,
Thanks for the thoughtful questions.
Questions
Hey @jengajojo,
Thanks for the thoughtful questions.
Questions
Yes, this proposal is not related in any way to the Cyfrin proposal (Arbitrum Security Enhancement Fund). This proposal was inspired by conversations with people at Arbitrum a few weeks back who invited Sherlock to develop an idea for an ecosystem pool to fund audits for Arbitrum projects.
Sherlock has found 183 Critical vulnerabilities and 419 Medium-severity vulnerabilities in Arbitrum-aligned projects over 47 audits in the past 12 months. That's just shy of 4 Critical vulnerabilities per audit. Here's that data.
Here's the current implementation of the quarterly report. All metrics related to the current KPIs can be found in the quarterly report, so the report should give a good sense of the program's effectiveness every quarter. As for transparency, some of the columns link to on-chain data or Sherlock contest data which can help verify the report's accuracy. If any other data can be added to increase transparency, Sherlock is happy to consider it.
The five KPIs outlined in the Impact Estimation section are designed with specific goals and deadlines, which should serve as the benchmark for evaluating the success of the program and whether it should be renewed or not.
General Feedback
Agreed. Sherlock has also received this feedback from other sources and the proposal will be updated with exact specifications for the committee, including the methodology for appointment, role, and compensation.
Agreed. In the Risks and Mitigations section, a new row has been added which discusses this conflict of interest.
Agreed. Please see the Timeline section. Each major milestone is outlined and tied to a specific deadline. If you think any milestones are missing, please don't hesitate to let us know.
Agreed. Please see the Risks and Mitigation section and let us know what you think.
Thanks for the great questions and feedback.
The proposal is pretty comprehensive! I like the concept of an Audit Council.
A couple of suggestions:
The proposal is pretty comprehensive! I like the concept of an Audit Council.
A couple of suggestions:
Apart from providing quarterly financial reports, you should also provide a quarterly performance report e.g. number of audits, number of audit contest participants, number of medium-level/high-level bugs reported, number of fixes.
Sherlock can consider contributing a small sum of its own money to this Ecosystem Security Fund (assuming the $4M is approved). This could be in the form of a grant or bonus discounts for early-stage protocols or protocols deemed crucial to Arbitrum's growth, to be selected by the Audit Council. Doing this would be a strong signal of Sherlock's commitment to its own proposal.
Really well written ser. Perfectly lines up with Arbitrum's constitution for the ecosystem security fund. Hope this passes and will definitely be voting for the approval of this :fire:
Fixed!
Thanks for letting us know.
KPIs for Victory: The success metrics are laid out—approved audits, scheduled timelines, completed audits, deployed protocols, and a clean record of on-chain exploits
Sherlock promises quarterly financial reports, a testament to transparency. Unspent funds? They go back to the Arbitrum treasury, ensuring fiscal prudence.
Hi @shaneMkt,
Thanks for the positive remarks and very much appreciate the suggestions!
Hi @shaneMkt,
Thanks for the positive remarks and very much appreciate the suggestions!
This is a great idea. The financial report does not include performance metrics. The financial report will have a link to each audit report, and each audit report includes ~80% of the suggested performance metrics. But it could be even more impactful to create a performance report that aggregates these metrics and adds the last couple that are missing from the audit report.
Sherlock can consider the best way to approach this. Right now, 80% of each audit is passed directly to independent security experts / teams, and the remaining 20% is used for operational purposes. There is a widespread practice of heavily discounting audits in the audit industry (50%+ discounts). Sherlock does not partake in this type of discounting and instead offers everyday low prices to each customer (treating each customer equally). Because of this, Sherlock may not be able to offer discounts as large as other auditors. But there may be some ability to contribute in this manner.
Thanks again for the kind remarks and feedback!
Have to appreciate such a comprehensive proposal.
Truly in line with Arbitrum's values and what we are trying to build here. This should be a great incentive for new teams to build and expand to Arbitrum.
Might take a look at @dk3 proposal on the RFP.
Good added metrics if fn. reports are out...
Any improvement in security is a good expansion in my book.
Great @dk3 , appreciate the best effort.
After discussing the new RFC proposal with DK to understand their vision, Sherlock believes that an RFP-style process for auditing services is in the best interest of the DAO (if executed well). Because of this, Sherlock has decided not to post this proposal to Snapshot, and Sherlock will instead direct its coalition of delegates to vote for DK's RFC and to vote against all other auditor-specific proposals.
Of course, a lot of effort is needed to turn DK's RFC into a fully fleshed-out version that everyone can rally behind. Sherlock has successfully been through similar RFP processes with other DAOs and Sherlock is happy to provide feedback to DK and others as to what works well and what causes issues.
After discussing the new RFC proposal with DK to understand their vision, Sherlock believes that an RFP-style process for auditing services is in the best interest of the DAO (if executed well). Because of this, Sherlock has decided not to post this proposal to Snapshot, and Sherlock will instead direct its coalition of delegates to vote for DK's RFC and to vote against all other auditor-specific proposals.
Of course, a lot of effort is needed to turn DK's RFC into a fully fleshed-out version that everyone can rally behind. Sherlock has successfully been through similar RFP processes with other DAOs and Sherlock is happy to provide feedback to DK and others as to what works well and what causes issues.
To confirm, Sherlock is withdrawing its own proposal and putting its endorsement (and delegates) behind DK's RFC because Sherlock believes the RFP process approach is in the best interest of the Arbitrum DAO.
Hey @jengajojo,
Thanks for the thoughtful questions.
Questions
Hey @jengajojo,
Thanks for the thoughtful questions.
Questions
Yes, this proposal is not related in any way to the Cyfrin proposal (Arbitrum Security Enhancement Fund). This proposal was inspired by conversations with people at Arbitrum a few weeks back who invited Sherlock to develop an idea for an ecosystem pool to fund audits for Arbitrum projects.
Sherlock has found 183 Critical vulnerabilities and 419 Medium-severity vulnerabilities in Arbitrum-aligned projects over 47 audits in the past 12 months. That's just shy of 4 Critical vulnerabilities per audit. Here's that data.
Here's the current implementation of the quarterly report. All metrics related to the current KPIs can be found in the quarterly report, so the report should give a good sense of the program's effectiveness every quarter. As for transparency, some of the columns link to on-chain data or Sherlock contest data which can help verify the report's accuracy. If any other data can be added to increase transparency, Sherlock is happy to consider it.
The five KPIs outlined in the Impact Estimation section are designed with specific goals and deadlines, which should serve as the benchmark for evaluating the success of the program and whether it should be renewed or not.
General Feedback
Agreed. Sherlock has also received this feedback from other sources and the proposal will be updated with exact specifications for the committee, including the methodology for appointment, role, and compensation.
Agreed. In the Risks and Mitigations section, a new row has been added which discusses this conflict of interest.
Agreed. Please see the Timeline section. Each major milestone is outlined and tied to a specific deadline. If you think any milestones are missing, please don't hesitate to let us know.
Agreed. Please see the Risks and Mitigation section and let us know what you think.
Thanks for the great questions and feedback.
The proposal is pretty comprehensive! I like the concept of an Audit Council.
A couple of suggestions:
The proposal is pretty comprehensive! I like the concept of an Audit Council.
A couple of suggestions:
Apart from providing quarterly financial reports, you should also provide a quarterly performance report e.g. number of audits, number of audit contest participants, number of medium-level/high-level bugs reported, number of fixes.
Sherlock can consider contributing a small sum of its own money to this Ecosystem Security Fund (assuming the $4M is approved). This could be in the form of a grant or bonus discounts for early-stage protocols or protocols deemed crucial to Arbitrum's growth, to be selected by the Audit Council. Doing this would be a strong signal of Sherlock's commitment to its own proposal.
Really well written ser. Perfectly lines up with Arbitrum's constitution for the ecosystem security fund. Hope this passes and will definitely be voting for the approval of this :fire:
Fixed!
Thanks for letting us know.
KPIs for Victory: The success metrics are laid out—approved audits, scheduled timelines, completed audits, deployed protocols, and a clean record of on-chain exploits
Sherlock promises quarterly financial reports, a testament to transparency. Unspent funds? They go back to the Arbitrum treasury, ensuring fiscal prudence.
Hi @shaneMkt,
Thanks for the positive remarks and very much appreciate the suggestions!
Hi @shaneMkt,
Thanks for the positive remarks and very much appreciate the suggestions!
This is a great idea. The financial report does not include performance metrics. The financial report will have a link to each audit report, and each audit report includes ~80% of the suggested performance metrics. But it could be even more impactful to create a performance report that aggregates these metrics and adds the last couple that are missing from the audit report.
Sherlock can consider the best way to approach this. Right now, 80% of each audit is passed directly to independent security experts / teams, and the remaining 20% is used for operational purposes. There is a widespread practice of heavily discounting audits in the audit industry (50%+ discounts). Sherlock does not partake in this type of discounting and instead offers everyday low prices to each customer (treating each customer equally). Because of this, Sherlock may not be able to offer discounts as large as other auditors. But there may be some ability to contribute in this manner.
Thanks again for the kind remarks and feedback!
Have to appreciate such a comprehensive proposal.
Truly in line with Arbitrum's values and what we are trying to build here. This should be a great incentive for new teams to build and expand to Arbitrum.
The below response reflects the views of L2BEAT’s governance team, composed of @krst and @Sinkas, and it’s based on the combined research, fact-checking and ideation of the two.
The below response reflects the views of L2BEAT’s governance team, composed of @krst and @Sinkas, and it’s based on the combined research, fact-checking and ideation of the two.
Sherlock has put forward a very comprehensive proposal and we’re happy to see them participate in @dk3 proposal to consolidate all security-related proposals into an RFP process. We just wanted to drop a comment and express our support. We look forward to working with the Sherlock team, as well as other stakeholders into forming a process to establish a process through which the DAO can fund security audits for Arbitrum projects.
Chatted with a few project delegates, and I have raised a proposal and snapshot to take an alternative approach which targets inclusivity for security orgs to collaborate on a RFP process.
https://forum.arbitrum.foundation/t/rfc-consolidate-security-proposals-into-a-rfp-process/19328
The below response reflects the views of L2BEAT’s governance team, composed of @krst and @Sinkas, and it’s based on the combined research, fact-checking and ideation of the two.
The below response reflects the views of L2BEAT’s governance team, composed of @krst and @Sinkas, and it’s based on the combined research, fact-checking and ideation of the two.
Sherlock has put forward a very comprehensive proposal and we’re happy to see them participate in @dk3 proposal to consolidate all security-related proposals into an RFP process. We just wanted to drop a comment and express our support. We look forward to working with the Sherlock team, as well as other stakeholders into forming a process to establish a process through which the DAO can fund security audits for Arbitrum projects.
Chatted with a few project delegates, and I have raised a proposal and snapshot to take an alternative approach which targets inclusivity for security orgs to collaborate on a RFP process.
https://forum.arbitrum.foundation/t/rfc-consolidate-security-proposals-into-a-rfp-process/19328
Can you enable viewing of comments, believe they are disabled - thanks!
Thank you for the elaborate proposal @Sherlock
Questions
Thank you for the elaborate proposal @Sherlock
Questions
Am I correct in understanding that the council proposed here will superseed other applications such as https://forum.arbitrum.foundation/t/non-constitutional-aip-arbitrum-security-enhancement-fund/19161 ?
Could you throw some light on how successful were these audits conducted by Sherlock in preventing vulnerabilities, especially high-severity ones?
What specific metrics and data points will be included in the quarterly reports to evaluate the program's effectiveness? How will transparency be ensured in reporting?
The proposal mentions that after 12 months, a decision will be made on whether to continue or adjust the program. What criteria will be used to make this decision?
General Feedback
Can you enable viewing of comments, believe they are disabled - thanks!
Thank you for the elaborate proposal @Sherlock
Questions
Thank you for the elaborate proposal @Sherlock
Questions
Am I correct in understanding that the council proposed here will superseed other applications such as https://forum.arbitrum.foundation/t/non-constitutional-aip-arbitrum-security-enhancement-fund/19161 ?
Could you throw some light on how successful were these audits conducted by Sherlock in preventing vulnerabilities, especially high-severity ones?
What specific metrics and data points will be included in the quarterly reports to evaluate the program's effectiveness? How will transparency be ensured in reporting?
The proposal mentions that after 12 months, a decision will be made on whether to continue or adjust the program. What criteria will be used to make this decision?
General Feedback
