feat: leveldb migrations tool

feat: migration tools for sqlite

docs/MIGRATION_GUIDE.md

@@ -1,162 +0,0 @@
-# Migration Guide from Arweave's SmartWeave SDK to Warp SDK
-
-This guide describes <strong>the simplest</strong> way to switch to the new version of SmartWeave. It uses `WarpFactory` for Web to quickly obtain fully configured, mem-cacheable SmartWeave instance. To see a more detailed explanation of all the core modules check out the [source code.](https://github.com/redstone-finance/warp)
-
-### You can watch this tutorial on YouTube 🎬
-
-- [Youtube link](https://www.youtube.com/watch?v=fNjUV7mHFqw)
-
-[![warp migration guide](https://img.youtube.com/vi/fNjUV7mHFqw/0.jpg)](https://www.youtube.com/watch?v=fNjUV7mHFqw)
-
-### Need help? 🙋‍♂️
-
-Please feel free to contact us [on Discord](https://redstone.finance/discord) if you face any problems.
-
-## 1. Update dependencies 📦
-
-#### 1.1 Install Warp
-
-```bash
-# Yarn
-yarn add warp-contracts
-
-# or NPM
-npm install warp-contracts
-```
-
-#### 1.2 Remove smartweave v1
-
-If smartweave was installed globally, add `-g` flag to npm or use `yarn global`
-
-```bash
-# Yarn
-yarn remove smartweave
-
-# or NPM
-npm uninstall smartweave
-```
-
-#### 1.3 Replace imports
-
-You can import the full API or individual modules.
-
-```typescript
-import * as WarpSdk from 'warp-contracts';
-import { Warp, Contract, ... } from 'warp-contracts';
-```
-
-## 2. Update your implementation 🧑‍💻
-
-### 2.1 Initialize a Warp client
-
-```typescript
-import Arweave from 'arweave';
-import { WarpNodeFactory } from 'warp-contracts';
-
-// Create an Arweave instance
-const arweave = Arweave.init({
-  host: 'dh48zl0solow5.cloudfront.net',
-  port: 443,
-  protocol: 'https',
-  timeout: 20000,
-  logging: false
-});
-
-// Create a Warp client
-const smartweave = WarpNodeFactory.memCached(arweave);
-```
-
-In this example we've used the `memCached` method. You can see other available methods in documentation:
-
-- [For Web](https://smartweave.docs.redstone.finance/classes/SmartWeaveWebFactory.html)
-- [For Node.js](https://smartweave.docs.redstone.finance/classes/SmartWeaveNodeFactory.html)
-
-#### [Optional] Custom modules 🛠
-
-Warp has a modular architecture, which allows you to connect custom modules to any part of the Warp client implementation. See [custom-client-example.ts](https://github.com/redstone-finance/redstone-smartweave-examples/blob/main/src/custom-client-example.ts) to learn more.
-
-### 2.2 Initialize contract object
-
-```typescript
-// Simple connection (allows to read state)
-const contract = warp.contract('YOUR_CONTRACT_TX_ID');
-```
-
-💡 Note! For being able to write interactions to blockchain you need to connect wallet to contract object.
-
-```typescript
-const contract = warp
-  .contract('YOUR_CONTRACT_TX_ID')
-  .connect(jwk) // jwk should be a valid private key (in JSON Web Key format)
-  .setEvaluationOptions({
-    // with this flag set to true, the write will wait for the transaction to be confirmed
-    waitForConfirmation: true
-  });
-```
-
-### 2.3 Interact with your contract
-
-#### Read state (readContract in V1)
-
-```typescript
-// Read state (similar to the "readContract" from SmartWeave V1)
-const { state, validity } = await contract.readState();
-
-// state is an object with the latest state
-
-// validity is an object with valid and invalid transaction IDs
-// E.g. { "TX_ID1": true, "TX_ID2": false, ...}
-```
-
-#### View state (interactRead in V1)
-
-```typescript
-// View state (similar to the "interactRead" from SmartWeave V1)
-const { result } = await contract.viewState<Input, View>({
-  function: "NAME_OF_YOUR_FUNCTION",
-  data: { ... }
-});
-```
-
-#### Write interaction (interactWrite in V1)
-
-```typescript
-// Write interaction (similar to the "interactWrite" from SmartWeave V1)
-const result = await contract.writeInteraction({
-  function: "NAME_OF_YOUR_FUNCTION",
-  data: { ... }
-});
-```
-
-💡 You can read detailed explanation of each contract method [here.](https://github.com/warp-contracts/warp#contract-methods)
-
-### [Optional] 2.4 Confgure logging
-
-Warp uses `tslog` library for logging. By default logger is set to "debug" level, which means that all messages with level "debug" or higher are logged.
-
-#### Update logger options
-
-```typescript
-LoggerFactory.INST.setOptions({
-  type: 'json',
-  displayFilePath: 'hidden',
-  displayInstanceName: false,
-  minLevel: 'info'
-});
-```
-
-Learn more about available logging options in [tslog documentation.](https://tslog.js.org/tsdoc/interfaces/isettingsparam.html)
-
-#### Update logger level
-
-Instead of updaitng all logger options you can simply set the new minimum logger level
-
-```typescript
-LoggerFactory.INST.logLevel('info');
-```
-
-Available log levels are listed [here.](https://github.com/redstone-finance/warp/blob/main/src/logging/RedStoneLogger.ts#L1)
-
-## 3. Test everything 🔥
-
-Before deploying your changes test it carefully. If you face any problems please contact us [on our discord](https://redstone.finance/discord). We'll be happy to help 😊

package.json

@@ -119,6 +119,7 @@
     "vm2": false,
     "archiver": false,
     "stream-buffers": false,
-    "constants": false
+    "constants": false,
+    "knex": false
   }
 }

src/__tests__/integration/basic/arweave-transactions-loading.ts

@@ -55,7 +55,7 @@ describe('Testing the Arweave interactions loader', () => {
     await addFunds(arweave, wallet);
 
     contractSrc = fs.readFileSync(path.join(__dirname, '../data/inf-loop-contract.js'), 'utf8');
-    const contractTxId = await warp.createContract.deploy({
+    const { contractTxId } = await warp.createContract.deploy({
       wallet,
       initState: JSON.stringify({
         counter: 10

src/__tests__/integration/basic/vrf.test.ts

@@ -7,7 +7,7 @@ import {
   ArweaveGatewayInteractionsLoader,
   defaultCacheOptions,
   EvaluationOptions,
-  GQLEdgeInterface,
+  GQLEdgeInterface, GQLNodeInterface,
   InteractionsLoader,
   LexicographicalInteractionsSorter,
   LoggerFactory,
@@ -57,7 +57,7 @@ describe('Testing the Profit Sharing Token', () => {
     loader = new VrfDecorator(arweave);
     LoggerFactory.INST.logLevel('error');
 
-    warp = WarpFactory.levelDbCached(arweave, {
+    warp = WarpFactory.custom(arweave, {
       ...defaultCacheOptions,
       inMemory: true
     })
@@ -150,24 +150,24 @@ class VrfDecorator extends ArweaveGatewayInteractionsLoader {
 
   async load(
     contractTxId: string,
-    fromBlockHeight: number,
+    fromSortKey?: string,
-    toBlockHeight: number,
+    toSortKey?: string,
-    evaluationOptions: EvaluationOptions
+    evaluationOptions?: EvaluationOptions
-  ): Promise<GQLEdgeInterface[]> {
+  ): Promise<GQLNodeInterface[]> {
-    const result = await super.load(contractTxId, fromBlockHeight, toBlockHeight, evaluationOptions);
+    const result = await super.load(contractTxId, fromSortKey, toSortKey, evaluationOptions);
     const arUtils = this.arweave.utils;
 
     const sorter = new LexicographicalInteractionsSorter(this.arweave);
 
     for (const r of result) {
-      r.node.sortKey = await sorter.createSortKey(r.node.block.id, r.node.id, r.node.block.height);
+      r.sortKey = await sorter.createSortKey(r.block.id, r.id, r.block.height);
-      const data = arUtils.stringToBuffer(r.node.sortKey);
+      const data = arUtils.stringToBuffer(r.sortKey);
       const [index, proof] = Evaluate(key.getPrivate().toArray(), data);
-      r.node.vrf = {
+      r.vrf = {
-        index: useWrongIndex.includes(r.node.id)
+        index: useWrongIndex.includes(r.id)
           ? arUtils.bufferTob64Url(Uint8Array.of(1, 2, 3))
           : arUtils.bufferTob64Url(index),
-        proof: useWrongProof.includes(r.node.id)
+        proof: useWrongProof.includes(r.id)
           ? 'pK5HGnXo_rJkZPJorIX7TBCAEikcemL2DgJaPB3Pfm2D6tZUdK9mDuBSRUkcHUDNnrO02O0-ogq1e32JVEuVvgR4i5YFa-UV9MEoHgHg4yv0e318WNfzNWPc9rlte7P7RoO57idHu5SSkm7Qj0f4pBjUR7lWODVKBYp9fEJ-PObZ'
           : arUtils.bufferTob64Url(proof),
         bigint: bufToBn(index).toString(),

src/__tests__/regression/read-state.test.ts

@@ -50,7 +50,7 @@ describe.each(chunked)('v1 compare.suite %#', (contracts: string[]) => {
       console.log('readState', contractTxId);
       try {
         console.log = function () {}; // to hide any logs from contracts...
-        const result2 = await WarpFactory.levelDbCached(arweave, {
+        const result2 = await WarpFactory.custom(arweave, {
           ...defaultCacheOptions,
           inMemory: true
         })
@@ -82,7 +82,7 @@ describe.each(chunkedVm)('v1 compare.suite (VM2) %#', (contracts: string[]) => {
         .readFileSync(path.join(__dirname, 'test-cases', 'contracts', `${contractTxId}.json`), 'utf-8')
         .trim();
       console.log('readState', contractTxId);
-      const result2 = await WarpFactory.levelDbCached(arweave, {
+      const result2 = await WarpFactory.custom(arweave, {
         ...defaultCacheOptions,
         inMemory: true
       })
@@ -110,7 +110,7 @@ describe.each(chunkedGw)('gateways compare.suite %#', (contracts: string[]) => {
     async (contractTxId: string) => {
       const blockHeight = 855134;
       console.log('readState Warp Gateway', contractTxId);
-      const warpR = await WarpFactory.levelDbCached(arweave, {
+      const warpR = await WarpFactory.custom(arweave, {
         ...defaultCacheOptions,
         inMemory: true
       })
@@ -120,7 +120,7 @@ describe.each(chunkedGw)('gateways compare.suite %#', (contracts: string[]) => {
       const resultString = stringify(result.state).trim();
 
       console.log('readState Arweave Gateway', contractTxId);
-      const result2 = await WarpFactory.levelDbCached(arweave, {
+      const result2 = await WarpFactory.custom(arweave, {
         ...defaultCacheOptions,
         inMemory: true
       })
@@ -142,7 +142,7 @@ describe('readState', () => {
     const result = await readContract(arweave, contractTxId, blockHeight);
     const resultString = stringify(result).trim();
 
-    const result2 = await WarpFactory.levelDbCached(arweave, {
+    const result2 = await WarpFactory.custom(arweave, {
       ...defaultCacheOptions,
       inMemory: true
     })
@@ -166,7 +166,7 @@ describe('readState', () => {
       target: '6Z-ifqgVi1jOwMvSNwKWs6ewUEQ0gU9eo4aHYC3rN1M'
     });
 
-    const v2Result = await WarpFactory.levelDbCached(arweave, {
+    const v2Result = await WarpFactory.custom(arweave, {
       ...defaultCacheOptions,
       inMemory: true
     })

src/contract/migration/MigrationTool.ts

@@ -0,0 +1,60 @@
+import { defaultArweaveMs, EvalStateResult, sortingLast } from '@warp/core';
+import Arweave from 'arweave';
+import { LoggerFactory } from '@warp/logging';
+import { LevelDbCache } from '@warp';
+import knex from 'knex';
+
+export type MigrationResult = Array<{ contractTxId: string; height: number; sortKey: string }>;
+
+export class MigrationTool {
+  private readonly logger = LoggerFactory.INST.create('MigrationTool');
+
+  constructor(private readonly arweave: Arweave, private readonly levelDb: LevelDbCache<EvalStateResult<unknown>>) {}
+
+  async migrateSqlite(sqlitePath: string): Promise<MigrationResult> {
+    this.logger.info(`Migrating from sqlite ${sqlitePath} to leveldb.`);
+
+    const knexDb = knex({
+      client: 'sqlite3',
+      connection: {
+        filename: sqlitePath
+      },
+      useNullAsDefault: true
+    });
+
+    const cache = await knexDb
+      .select(['contract_id', 'height', 'state'])
+      .from('states')
+      .max('height')
+      .groupBy(['contract_id']);
+
+    this.logger.info(`Migrating ${cache?.length} contracts' state`);
+
+    const result = [];
+
+    for (const entry of cache) {
+      const contractTxId = entry['contract_id'];
+      const height = entry['height'];
+      const state = JSON.parse(entry['state']);
+
+      const blockHeightString = `${height}`.padStart(12, '0');
+      const sortKey = `${blockHeightString},${defaultArweaveMs},${sortingLast}`;
+
+      this.logger.debug(`Migrating ${contractTxId} at height ${height}: ${sortKey}`);
+
+      await this.levelDb.put(
+        {
+          contractTxId,
+          sortKey: `${blockHeightString},${defaultArweaveMs},${sortingLast}`
+        },
+        new EvalStateResult(state.state, state.validity, {})
+      );
+
+      result.push({ contractTxId, height, sortKey });
+    }
+
+    this.logger.info(`Migration done.`);
+
+    return result;
+  }
+}

src/core/Warp.ts

@@ -4,7 +4,8 @@ import {
   HandlerApi,
   InteractionsLoader,
   WarpBuilder,
-  StateEvaluator
+  StateEvaluator,
+  EvalStateResult
 } from '@warp/core';
 import Arweave from 'arweave';
 import {
@@ -16,6 +17,8 @@ import {
   PstContractImpl
 } from '@warp/contract';
 import { GQLNodeInterface } from '@warp/legacy';
+import { MigrationTool } from '../contract/migration/MigrationTool';
+import { LevelDbCache } from '@warp/cache';
 
 /**
  * The Warp "motherboard" ;-).
@@ -27,19 +30,22 @@ import { GQLNodeInterface } from '@warp/legacy';
  */
 export class Warp {
   readonly createContract: CreateContract;
+  readonly migrationTool: MigrationTool;
 
   constructor(
     readonly arweave: Arweave,
+    readonly levelDb: LevelDbCache<EvalStateResult<unknown>>,
     readonly definitionLoader: DefinitionLoader,
     readonly interactionsLoader: InteractionsLoader,
     readonly executorFactory: ExecutorFactory<HandlerApi<unknown>>,
     readonly stateEvaluator: StateEvaluator
   ) {
     this.createContract = new DefaultCreateContract(arweave);
+    this.migrationTool = new MigrationTool(arweave, levelDb);
   }
 
-  static builder(arweave: Arweave): WarpBuilder {
+  static builder(arweave: Arweave, cache: LevelDbCache<EvalStateResult<unknown>>): WarpBuilder {
-    return new WarpBuilder(arweave);
+    return new WarpBuilder(arweave, cache);
   }
 
   /**

src/core/WarpBuilder.ts

@@ -5,15 +5,17 @@ import {
   ContractDefinitionLoader,
   DebuggableExecutorFactory,
   DefinitionLoader,
+  EvalStateResult,
   ExecutorFactory,
   HandlerApi,
   InteractionsLoader,
+  LevelDbCache,
   MemCache,
-  WarpGatewayContractDefinitionLoader,
-  WarpGatewayInteractionsLoader,
-  Warp,
   SourceType,
-  StateEvaluator
+  StateEvaluator,
+  Warp,
+  WarpGatewayContractDefinitionLoader,
+  WarpGatewayInteractionsLoader
 } from '@warp';
 
 export const WARP_GW_URL = 'https://d1o5nlqr4okus2.cloudfront.net';
@@ -24,7 +26,7 @@ export class WarpBuilder {
   private _executorFactory?: ExecutorFactory<HandlerApi<unknown>>;
   private _stateEvaluator?: StateEvaluator;
 
-  constructor(private readonly _arweave: Arweave) {}
+  constructor(private readonly _arweave: Arweave, private readonly _cache: LevelDbCache<EvalStateResult<unknown>>) {}
 
   public setDefinitionLoader(value: DefinitionLoader): WarpBuilder {
     this._definitionLoader = value;
@@ -73,6 +75,7 @@ export class WarpBuilder {
   build(): Warp {
     return new Warp(
       this._arweave,
+      this._cache,
       this._definitionLoader,
       this._interactionsLoader,
       this._executorFactory,

src/core/WarpFactory.ts

@@ -55,14 +55,8 @@ export class WarpFactory {
    * Returns a fully configured {@link Warp} that is using arweave.net compatible gateway
    * (with a GQL endpoint) for loading the interactions.
    */
-  static arweaveGw(
+  static arweaveGw(arweave: Arweave, cacheOptions: CacheOptions = defaultCacheOptions): Warp {
-    arweave: Arweave,
+    return this.custom(arweave, cacheOptions).useArweaveGateway().build();
-    cacheOptions: CacheOptions = {
-      maxStoredTransactions: 20,
-      inMemory: false
-    }
-  ): Warp {
-    return this.levelDbCached(arweave, cacheOptions).useArweaveGateway().build();
   }
 
   /**
@@ -71,24 +65,19 @@ export class WarpFactory {
   static warpGw(
     arweave: Arweave,
     gatewayOptions: GatewayOptions = defaultWarpGwOptions,
-    cacheOptions: CacheOptions = {
+    cacheOptions: CacheOptions = defaultCacheOptions
-      maxStoredTransactions: 20,
-      inMemory: false
-    }
   ): Warp {
-    return this.levelDbCached(arweave, cacheOptions)
+    return this.custom(arweave, cacheOptions)
       .useWarpGateway(gatewayOptions.confirmationStatus, gatewayOptions.source, gatewayOptions.address)
       .build();
   }
 
-  static levelDbCached(arweave: Arweave, cacheOptions: CacheOptions): WarpBuilder {
+  static custom(arweave: Arweave, cacheOptions: CacheOptions): WarpBuilder {
-    const executorFactory = new CacheableExecutorFactory(arweave, new HandlerExecutorFactory(arweave), new MemCache());
+    const cache = new LevelDbCache<EvalStateResult<unknown>>(cacheOptions);
-    const stateEvaluator = new CacheableStateEvaluator(
-      arweave,
-      new LevelDbCache<EvalStateResult<unknown>>(cacheOptions),
-      [new Evolve()]
-    );
 
-    return Warp.builder(arweave).setExecutorFactory(executorFactory).setStateEvaluator(stateEvaluator);
+    const executorFactory = new CacheableExecutorFactory(arweave, new HandlerExecutorFactory(arweave), new MemCache());
+    const stateEvaluator = new CacheableStateEvaluator(arweave, cache, [new Evolve()]);
+
+    return Warp.builder(arweave, cache).setExecutorFactory(executorFactory).setStateEvaluator(stateEvaluator);
   }
 }

src/core/modules/impl/LexicographicalInteractionsSorter.ts

@@ -6,7 +6,7 @@ const defaultArweaveMs = ''.padEnd(13, '9');
 const defaultArweaveMs_After_Block_973730 = ''.padEnd(13, '0');
 export const block_973730 = 973730;
 
-const sortingLast = ''.padEnd(64, 'z');
+export const sortingLast = ''.padEnd(64, 'z');
 
 /**
  * implementation that is based on current's SDK sorting alg.

tools/migrate.ts

@@ -0,0 +1,31 @@
+/* eslint-disable */
+import Arweave from 'arweave';
+import {defaultCacheOptions, defaultWarpGwOptions, LoggerFactory, WarpFactory} from '../src';
+
+LoggerFactory.INST.logLevel('debug');
+
+async function main() {
+  const arweave = Arweave.init({
+    host: 'arweave.net', // Hostname or IP address for a Arweave host
+    port: 443, // Port
+    protocol: 'https', // Network protocol http or https
+    timeout: 60000, // Network request timeouts in milliseconds
+    logging: false // Enable network request logging
+  });
+
+  const warp = WarpFactory.warpGw(arweave, defaultWarpGwOptions, {
+    ...defaultCacheOptions,
+    dbLocation: './tools/.leveldb'
+  });
+
+  const result = await warp.migrationTool.migrateSqlite("./tools/sqlite/contracts-3008.sqlite");
+
+  console.log(result);
+
+  const dump = await warp.stateEvaluator.dumpCache();
+
+  console.log(dump);
+}
+
+
+main().catch((e) => console.error(e));

yarn.lock

@@ -7200,7 +7200,7 @@ sprintf-js@~1.0.2:
   resolved "https://registry.yarnpkg.com/sprintf-js/-/sprintf-js-1.0.3.tgz#04e6926f662895354f3dd015203633b857297e2c"
   integrity sha512-D9cPgkvLlV3t3IzL0D0YLvGA9Ahk4PcvVwUbN0dSGr1aP0Nrt4AEnTUbuGvquEC0mA64Gqt1fzirlRs5ibXx8g==
 
-sqlite3@^5.0.2, sqlite3@^5.0.3:
+sqlite3@^5.0.3:
   version "5.0.8"
   resolved "https://registry.yarnpkg.com/sqlite3/-/sqlite3-5.0.8.tgz#b4b7eab7156debec80866ef492e01165b4688272"
   integrity sha512-f2ACsbSyb2D1qFFcqIXPfFscLtPVOWJr5GmUzYxf4W+0qelu5MWrR+FAQE1d5IUArEltBrzSDxDORG8P/IkqyQ==